[iris] Split cluster/backends into backends/platforms/setup

[rjpower]

Regroup cluster/backends/ by concern into sibling trees, with no behavior change:

• backends/ — the pure TaskBackend implementations (rpc/, k8s/tasks.py) plus the shared protocol seam (protocols.py, types.py).
• platforms/ (new) — runtime substrate drivers: WorkerInfraProvider impls (gcp/manual workers), cloud-API wrappers (GcpService, CloudK8sService), worker bootstrap, remote-exec health. admin bring-up/ops: ControllerProvider impls, vm_lifecycle.py, controller bootstrap, the provider factory.
  Part of the multi-backend design ([Design] Iris multi-backend: Model D (federation peers + local backend contract) #6718).

[rjpower]

@claude any reason the "setup/controller" modules can't live in platform as well? then we'd cleanly have platform: GCP & K8s setup tooling & "backends": the task dispatch backends. No setup package needed.

If this makes sense, please adjust the layout accordingly.

[claude]

Claude finished @rjpower's task in 1m 39s —— View job

---

Can setup/controller modules live under platforms/ instead?

• Mapped setup/ vs platforms/ contents and import directions
• Assessed the merge against the PR's stated layering invariant
• Reached a recommendation (below) — holding on implementation pending your call

Short answer: there's no technical blocker — but merging setup/ into platforms/ deletes the exact invariant this PR exists to establish, so I'd rather confirm before doing it.

Why it's technically fine

The import graph is already strictly one-directional:

• Nothing in platforms/ or backends/ imports setup/ (grep: zero matches).
• setup/ imports freely from both (factory.py, the three */controller*.py, vm_lifecycle.py).

So there are no cycles to fear. You could git mv setup/gcp/controller*.py → platforms/gcp/, setup/k8s/controller.py → platforms/k8s/, setup/manual/controller.py → platforms/manual/, and vm_lifecycle.py + factory.py to platforms/, and it would import and run unchanged.

Why I'd pause first

The PR's own rationale is:

nothing in backends/ or platforms/ imports from setup/ … That direction is what lets a future per-cluster agent depend on backends/ + platforms/ without dragging in admin/controller-bringup code — the structural prerequisite for the multi-backend work in #6718.

The 2-tree layout you're describing is conceptually cleaner ("platform = GCP/K8s tooling, backend = dispatch"), but it collapses that boundary. After the merge, the per-cluster agent that imports platforms/ gets vm_lifecycle.py, the controller-bootstrap renderers, and factory.py (which pulls all ControllerProvider impls) in its dependency surface. The thing the PR makes a checkable package-level fact goes back to being a convention.

So the real question isn't "can it?" but "is the worker-vs-controller-bringup split worth a package boundary?" — and #6718 is the thing that answers that. If the per-cluster-agent dependency-slimming still matters, keep setup/. If that's no longer a goal, merging is reasonable.

A middle option

If you want no setup/ package and keep the boundary checkable, group by cloud but keep worker/controller as distinct modules with a lint/grep rule:

platforms/gcp/workers.py      # WorkerInfraProvider  — agent imports this
platforms/gcp/controller.py   # ControllerProvider   — admin only
platforms/gcp/worker_bootstrap.py / controller_bootstrap.py

The invariant becomes "worker-side modules don't import controller-side modules" — enforced at module granularity instead of package granularity. You lose the one-line package grep but get the cleaner taxonomy.

My recommendation

Three viable end states:

1. Keep setup/ as-is — if [Design] Iris multi-backend: Model D (federation peers + local backend contract) #6718's per-cluster agent still needs a slim backends/+platforms/ surface. (no change)
2. Merge into platforms/, group by cloud — simplest taxonomy; drop the package-level invariant. I can do the git mvs + reference updates + run pre-commit/tests.
3. Merge but keep module-level worker/controller split + lint rule — cleaner taxonomy, boundary still enforced.

Which do you want? If you just say "go with 2" (or 3) I'll implement it on this branch. I held off on the moves because each option silently changes the design intent you wrote in the PR description, and that felt like your call to make.
• branch iris-mb-1-platforms-setup

[rjpower]

@claude let's merge and keep the module-level controller/worker split.

We don't really care so much if another backend can import controller code. The key invariants we want are:

backends -> platforms, but never platform -> backend.

Thus code that has things like "controller script to boot the cluster with no direct controller module references" is fine for now.

[claude]

Claude finished @rjpower's task in 9m 21s —— View job

---

Merged setup/ into platforms/ (module-level worker/controller split kept)

Done — pushed as 0af61fb67.

• Mapped setup/ vs platforms/ contents
• Moved the controller/admin modules into platforms/, grouped by cloud
• Updated all imports and references
• Updated platforms/__init__.py docstring + fixed a stale doc reference
• pre-commit --all-files --fix → all hooks pass (ruff, black, pyrefly, …)
• Ran affected tests → 186 passed
• Committed & pushed

What changed

cluster/setup/ is gone; its modules now live under cluster/platforms/, grouped by cloud, with worker-side (workers.py) and controller-side (controller.py) modules kept as distinct files:

From	To
setup/factory.py	platforms/factory.py
setup/vm_lifecycle.py	platforms/vm_lifecycle.py
setup/gcp/controller.py	platforms/gcp/controller.py
setup/gcp/controller_bootstrap.py	platforms/gcp/controller_bootstrap.py
setup/k8s/controller.py	platforms/k8s/controller.py
setup/manual/controller.py	platforms/manual/controller.py

All git mv (rename history preserved). Each cloud now reads e.g. platforms/gcp/{workers,controller,worker_bootstrap,controller_bootstrap}.py.

Invariant kept vs dropped

Per your call, the only layering rule enforced is what already held: nothing in platforms/ (or backends/) imports a TaskBackend implementation from backends/ — only the shared backends.protocols / backends.types seam (which platforms/ already depended on). The previous package-level "a backend can't reach controller bring-up code" guarantee is intentionally gone, so a controller bootstrap script with no direct controller-module references is fine. The platforms/__init__.py docstring now states this.

Reference updates

iris.cluster.setup.* → iris.cluster.platforms.* across src (cli/main.py, composer.py, worker/main.py, local_cluster.py, the moved modules' internal imports) and tests (test_config.py, test_platform.py incl. mock.patch targets, test_coreweave.py, test_vm_lifecycle.py, gpu_gang_smoke.py). setup_scripts (the renamed worker setup-script generator) is untouched. Also corrected a stale iris.cluster.setup.cuda_toolchain_setup_script → iris.cluster.setup_scripts.cuda_toolchain_setup_script reference in docs/coreweave.md.

Testing

./infra/pre-commit.py --all-files --fix — all hooks ok. uv run pytest tests/cluster/backends/test_config.py tests/cluster/backends/gcp/test_platform.py tests/cluster/backends/k8s/test_coreweave.py tests/cluster/controller/test_vm_lifecycle.py — 186 passed. Module imports of all moved files verified.

Note: the PR description still describes the old three-tree (backends/platforms/setup) layout and its package-level invariant — worth a quick edit to match this two-tree result, but I left the description to you since it's your prose.
• branch iris-mb-1-platforms-setup