Skip to content

refactor(upgrade): move indexDataPlatformsStep and migrateHomePageLinksStep to non-blocking system update#17708

Merged
Dutt23 merged 4 commits into
masterfrom
worktree-sd+move-remaining-boot-steps
Jun 5, 2026
Merged

refactor(upgrade): move indexDataPlatformsStep and migrateHomePageLinksStep to non-blocking system update#17708
Dutt23 merged 4 commits into
masterfrom
worktree-sd+move-remaining-boot-steps

Conversation

@Dutt23
Copy link
Copy Markdown
Contributor

@Dutt23 Dutt23 commented Jun 3, 2026

## Why

`IndexDataPlatformsStep` and `MigrateHomePageLinksStep` were blocking GMS startup
unnecessarily — running sequentially before GMS could serve traffic, despite neither
being critical to request serving.

`IndexDataPlatformsStep` also had a correctness bug: the original boot step fetched
only the first 1000 data platform URNs and stopped, silently skipping any beyond that
page. Any deployment with >1000 data platforms would have an incomplete search index
after running this migration.

## What

1. **Moves both steps** from `BootstrapManagerFactory` (blocking, sequential) to the
   `datahub-upgrade` `NonBlockingSystemUpgrade` pattern so GMS serves traffic
   immediately while these steps run out-of-band.

2. **Fixes pagination bug** in `IndexDataPlatformsStep`: the new implementation
   iterates pages of 1000 URNs until `start >= listResult.getTotal()`, ensuring all
   data platforms are indexed regardless of count.

### Changes

- Added `IndexDataPlatforms` / `IndexDataPlatformsStep` under `system/dataplatforms/`
  with paginated `listUrns` loop (`BATCH_SIZE=1000`, iterates all pages)
- Added `MigrateHomePageLinks` / `MigrateHomePageLinksStep` under `system/homepagelinks/`
- `showHomePageRedesign` flag gate preserved at wrapper construction time — step is a
  no-op when flag is off
- `STEP_ID` / `VERSION` match original boot step constants for idempotency (existing
  deployments that already ran the old step will skip)
- Added config flags (default `true`):
  ```
  systemUpdate.indexDataPlatforms.enabled
  systemUpdate.migrateHomePageLinks.enabled
  ```
- Removed dead source files from `metadata-service/factories/boot/steps/`
- Added unit tests for both steps

## Bug Fix

| | Old (boot step) | New (system-update step) |
|---|---|---|
| Fetch strategy | `listUrns(start=0, count=1000)` once | Paginated loop until `start >= total` |
| Data platforms >1000 | ❌ Silently truncated | ✅ Fully indexed |

## Impact

GMS startup no longer blocks on these steps. Data platforms >1000 are now fully indexed
after migration. Follows the same pattern established in #17691.

> **Note:** To disable either step, set the corresponding `systemUpdate.*.enabled`
> flag to `false`.

@github-actions github-actions Bot added the devops PR or Issue related to DataHub backend & deployment label Jun 3, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

Bundle Report

Changes will increase total bundle size by 19.97kB (0.09%) ⬆️. This is within the configured threshold ✅

Detailed changes
Bundle name Size Change
datahub-react-web-esm 23.4MB 19.97kB (0.09%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name Size Change Total Size Change (%)
assets/index-*.js 6.72kB 8.82MB 0.08%
assets/en-*.js 13.24kB 126.46kB 11.7% ⚠️

@Dutt23 Dutt23 marked this pull request as ready for review June 3, 2026 10:53
@maggiehays maggiehays added the needs-review Label for PRs that need review from a maintainer. label Jun 3, 2026
@maggiehays maggiehays added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Jun 4, 2026
@Dutt23 Dutt23 merged commit 9e50e65 into master Jun 5, 2026
58 checks passed
@Dutt23 Dutt23 deleted the worktree-sd+move-remaining-boot-steps branch June 5, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops PR or Issue related to DataHub backend & deployment pending-submitter-merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants