[iris] Add federation module: observable peers, not targetable#6826
Merged
Conversation
Add the iris.cluster.federation package: a controller's federation peers — remote Iris clusters it may delegate whole jobs to — become configurable and observable, without anything yet executing on them. - peers: config section on IrisClusterConfig (identity + reachability + trust, never capabilities). Trust reuses the rigging credentials_for path, so a peer connection presents the same client credentials every cross-cluster client does — no second credential system. - FederationPeer wraps one authenticated RemoteClusterClient per peer; the FederationManager owns the registry and a background capability heartbeat that learns what each peer can currently schedule (available:<device> markers). - ListPeers / PeerSummary RPC, distinct from ListBackends (backends are what a controller runs; peers are what it delegates to), plus GetClusterCapabilities, the heartbeat a peer answers with its own markers. - A submit-time PeerRouter, kept separate from the static meta-scheduler index. It selects local execution for every job, so nothing is handed off. Inert until a peers entry exists: with no peers, no connections are built, the heartbeat never starts, and every path is a no-op, so a single-cluster deployment is unchanged. Even with a peer configured, submissions still route and dispatch locally. Builds on the fold-exclusion schema seam; adds nothing to the DAG fold. Part of #6718
Contributor
|
Claude finished @rjpower's task in 3m 58s —— View job Code review
No blocking issues found. Checked for bugs and CLAUDE.md/AGENTS.md compliance. I traced every cross-module symbol the new code depends on and all wiring checks out:
Design/quality notes (all consistent with the PR's "inert until a
Tests are behavior-focused with injected fakes (stub connections/backends via the The PR description leads with what the change does, has no template scaffold or Testing section, and no self-credit — it reads as a proper commit message. No changes requested. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8f7f82e97
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+738
to
+742
| rpc ListPeers(Controller.ListPeersRequest) returns (Controller.ListPeersResponse); | ||
|
|
||
| // Federation capability heartbeat: report the availability markers this | ||
| // controller advertises to peers that may delegate work to it. | ||
| rpc GetClusterCapabilities(Controller.GetClusterCapabilitiesRequest) returns (Controller.GetClusterCapabilitiesResponse); |
There was a problem hiding this comment.
In auth/IAP deployments, unprovisioned browser or service-account identities resolve to the read-only dashboard role, and authorize_method default-denies any RPC not listed in DASHBOARD_READABLE_RPCS (lib/iris/src/iris/rpc/auth.py:50-56). These new observational federation RPCs are not added there, so ListPeers from the dashboard and capability heartbeats to an IAP-fronted peer using an unprovisioned programmatic identity fail with PERMISSION_DENIED before these handlers run. Please add the read-only federation methods to the allowlist when introducing the RPCs.
Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment.
🤖 Good catch — fixed in 9bd530e. Added ListPeers and GetClusterCapabilities to DASHBOARD_READABLE_RPCS alongside ListBackends, with a test asserting the dashboard role can call them. Both are read-only, so this is the correct tier.
Add ListPeers and GetClusterCapabilities to DASHBOARD_READABLE_RPCS. Both are read-only: without them, an unprovisioned IAP identity (the read-only dashboard role) is default-denied before the handler runs — the dashboard's peer view and a capability heartbeat from a peer asserting an unprovisioned programmatic identity would both fail with PERMISSION_DENIED.
The capability heartbeat now forwards the peer's BackendSummary list — the underlying cluster's static topology (kind, advertised devices) and dynamic state (availability, worker/task counts) — rather than deriving synthetic "available:<device>" marker strings. This conveys the real topology the parent needs, and reuses the peer's existing ListBackends read: no bespoke capability RPC, no invented marker vocabulary. PeerSummary.advertised_capabilities (repeated string) becomes PeerSummary.backends (repeated BackendSummary); the FederationPeer heartbeat calls the peer's ListBackends, already on the dashboard read allowlist.
Drop the design-seam narration from the class docstring; state the behavior.
rjpower
added a commit
that referenced
this pull request
Jul 2, 2026
…6835) Turn the federation router's peer arm live and add the first federated job that runs on a remote Iris cluster and reports back. Routing decides at submit time: prefer-local when any local backend is feasible, else match the job's constraints against a peer's live capability heartbeat and hand off to the first that can host it, and an explicit `cluster=<peer>` directive forces that peer (mutually exclusive with a local `backend` pin; the peer must be configured). A job no local backend can host is no longer fatal if a peer can take it. A handed-off job persists as a SENT `federated_jobs` handle (no local tasks) under a deterministic `remote_job_id` that folds this cluster's id into the job-name component, is delivered synchronously to the peer's `LaunchJob` with an explicit `FederationHandoff` field recording the requester and owner principal, and flips to `HANDED_OFF` on ack. A failed delivery is not fatal — the handle persists in `PENDING_HANDOFF` and the sync loop re-drives it under the same id (the peer's KEEP policy dedups), so handoff is exactly-once across a parent crash. Only a root job is handed off, and the `FederationHandoff` field is honored only from a trusted (admin) peer — a non-admin that sets it is denied, so it cannot forge a handoff to run a job as another user. The peer records the received job as a RECEIVED `federated_jobs` row naming the requester (the same table holds both directions, discriminated by `direction`), and writes an append-only `federation_changelog` in the same transaction as each job/task mutation. Each changelog row is stamped with the requester it belongs to, so `FederationSync` reports a requester only its own jobs without a join — and a tombstone survives the job delete. A second background loop pulls each peer once per tick via a new `FederationSync` RPC: the peer returns the jobs changed since the requester's cursor — or its full active set when the cursor is stale — and the parent mirrors job/task state onto its handle, applies tombstones, and advances the cursor in one transaction. A stale cursor (first contact, or below the retained changelog window) drives a full-resync set-replacement, so a job the parent never saw tombstoned is still reclaimed. Cancel bumps a versioned cancel intent and routes an idempotent `TerminateJob(remote_job_id)` to the peer; a peer `NOT_FOUND` (already terminal-and-pruned) satisfies the cancel, a transient failure is re-driven by the sync loop, and a cancel-before-handoff is terminated locally and never delivered. Federation stays inert until a `peers:` entry exists: with no peers, neither loop starts, routing is local, and the changelog is never written, so a single-cluster deployment is byte-identical apart from one new empty table and a `direction` discriminator column on `federated_jobs` (migration 0035). The backend seam is untouched — a peer is a remote cluster the job is handed to, not a backend. Two pieces are deferred to #6843: cross-cluster spend caps for federated roots (a zero-local-task federated root is not yet charged against the user's budget), and pruning the append-only `federation_changelog` (the sync staleness path already handles a pruned changelog, so a pruner needs no protocol change). Part of #6718 — PR3 of the multi-cluster federation rollout (follows #6821 PR1 and #6826 PR2).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the
iris.cluster.federationpackage: a controller's federation peers —remote Iris clusters it may delegate whole jobs to — become configurable and
observable, without anything yet executing on them. This is PR2 of the Iris
multi-cluster federation rollout, building on the fold-exclusion schema seam
from PR1 (#6821).
peers:config onIrisClusterConfigdeclares identity + reachability +trust, never capabilities (those are dynamic). Trust reuses the rigging
credentials_forpath, so a peer connection presents the same clientcredentials every cross-cluster client does — no second credential system.
FederationPeerwraps one authenticatedRemoteClusterClientper peer.FederationManagerowns the registry and a background capabilityheartbeat that forwards each peer's backends — the underlying cluster's static
topology (kind, advertised devices) and dynamic state (availability,
worker/task counts). The heartbeat reuses the peer's existing
ListBackendsread; there is no bespoke capability RPC or invented marker vocabulary.
ListPeers/PeerSummaryRPC, kept distinct fromListBackends(backends are what a controller runs; peers are what it delegates to).
PeerSummary.backendscarries the forwardedBackendSummarylist.PeerRouter, separate from the static meta-scheduler index,that selects local execution for every job — so nothing is handed off.
The master derisking lever holds: federation is inert until a
peers:entryexists. With no peers, no connections are built, the heartbeat never starts, and
every path is a no-op, so a single-cluster deployment is byte-identical. Even
with a peer configured, submissions still route and dispatch locally. The
backend seam never learns federation exists, and federation adds nothing to the
DAG fold.
Part of #6718