Add principle 14: couple through data at rest, not run status

Production jobs read whatever is on disk when they run and record how
stale it was. None of them asks whether the job producing its input
succeeded, or ran at all. That is why a failed ingest makes the 06:00
forecast slightly staler rather than absent.

The property was already implemented and already argued for in one place
(dropping the promoted_model lineage edge), but the docs never stated it,
and two places framed it as a defect awaiting a fix: the TODO on
power_time_series_and_metadata_schedule proposed a run-status sensor that
would only fire live_forecasts once the ingest had succeeded, and the
Airflow comparison table listed "run-status coordination" as planned
work. Both would have introduced exactly the cascade the design avoids.

live_forecasts wants to run at 00/06/12/18 regardless, so there is no
acceleration to be had from a sensor there. Sensors are still wanted on
the ingest assets, where they save booting Fargate for a no-op fetch;
that is what #324 is actually for, and the table row now says so.

Note the asymmetry that makes this work: the lineage edge stays. It is
what builds a developer's inputs on a laptop. What it must never become
is a runtime gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: JackKelly

docs/architecture/why-dagster-not-airflow.md

@@ -200,7 +200,7 @@ around the Dagster UI and would need rewriting.
 | `add_output_metadata` tables, asset catalog, lineage | every asset | Asset-event `extra` JSON (2.10+) in the events list | Partial — raw JSON, no rendered tables or history plots |
 | Asset checks — non-blocking WARN, attached to an asset, dedicated Checks view (`power_data_is_fresh`, `nwp_has_no_unexpected_nulls`) | power ingest, `ecmwf_ens` | Data-quality as ordinary tasks (`common.sql` check operators; Great Expectations / Soda / dbt-test); no first-class check primitive or Checks UI, blocking by default (as of 3.3.0) | Partial — the capability exists as tasks; the non-blocking severity and check-status surface do not |
 | `EcsRunLauncher` (laptop = subprocess, cloud = Fargate, switched by `dagster.yaml`) | control plane | ECS executor (Amazon provider, Fargate launch type) | Exists; per-*task* rather than per-run granularity |
-| Sensors / run-status coordination (planned, [#324](https://github.com/openclimatefix/nged-substation-forecast/issues/324)) | ingest → forecast ordering | Asset-triggered DAGs, event-driven scheduling | Parity, arguably cleaner in Airflow |
+| Data-arrival sensors (planned, [#324](https://github.com/openclimatefix/nged-substation-forecast/issues/324)) | firing the *ingest* jobs only when there is new data to fetch, to avoid booting Fargate for a no-op | Asset-triggered DAGs, event-driven scheduling | Parity, arguably cleaner in Airflow |
 
 The asset-checks row is worth one extra sentence, because the gap there is architectural rather
 than cosmetic: non-blocking WARN checks are the *mechanism* by which this service stays

docs/design-philosophy/design-principles.md

@@ -73,7 +73,7 @@ principle behind it is a claim we are merely hoping comes true.
    check in the repo is non-blocking `WARN`; there is deliberately no `ERROR`-severity check
    anywhere. *Serves:* [Hypothesis 1: a service that mostly runs
    itself](engineering-hypotheses.md#h1-a-service-that-mostly-runs-itself). *Detail:* [Inherent
-   Stability](inherent-stability.md), whose [ten rules](inherent-stability.md#the-rules) are the
+   Stability](inherent-stability.md), whose [eleven rules](inherent-stability.md#the-rules) are the
    fine-grained form of this principle together with the complexity-offline and strict-contracts
    principles.
 
@@ -401,6 +401,42 @@ principle behind it is a claim we are merely hoping comes true.
     [When would a REST API earn its keep?](../architecture/forecast-delivery.md#when-would-a-rest-api-earn-its-keep),
     [Considered but rejected designs](../architecture/production-deployment.md#considered-but-rejected-designs).
 
+14. **Production jobs are coupled through data at rest, never through run status.** Each scheduled
+    production job reads whatever is on disk at the moment it runs, and records how stale that input
+    turned out to be. No production job asks whether the job that produces its input succeeded, or
+    ran at all.
+    The common alternative is a chain of scheduled jobs — A at 06:00, B at 06:15, C at 06:30 — in
+    which B's real input is *the event of A having run*. That design has no way to distinguish "A
+    failed" from "A is still running" from "A had nothing to do", so one bad morning upstream takes
+    out every job downstream of it for the rest of the day, and recovery means replaying the chain in
+    order.
+    Ours cannot propagate a failure because there is no channel for it to propagate down.
+    `live_forecasts` does not care whether `ecmwf_ens` succeeded in the last 24 hours, or whether
+    this hour's telemetry pull ran: it selects the freshest NWP run genuinely present as of its own
+    init time and stamps the row with how old that was. A failed ingest makes the 06:00 forecast
+    slightly staler. It cannot make it late, and it cannot make it absent.
+    Note the deliberate asymmetry with the *lineage* graph, which is a different thing from a runtime
+    precondition. Dagster still knows that `live_forecasts` depends on `ecmwf_ens` and
+    `power_time_series_and_metadata`; that edge is what lets a developer materialise one asset on a
+    laptop and have its inputs built for them, and what makes the graph legible in the UI. What we
+    decline is letting that edge become a gate in production.
+    *Without it:* one failed 06:00 ingest suppresses the 06:00 forecast, and then the 12:00 and 18:00
+    ones behind it; NGED receive nothing at all rather than something slightly stale, which
+    [inverts the whole degradation ladder](inherent-stability.md#the-degradation-ladder); and someone
+    has to re-run the chain in order, out of hours, to catch up — the precise out-of-hours
+    intervention that
+    [T1.1](engineering-hypotheses.md#h1-a-service-that-mostly-runs-itself) predicts will never be
+    needed.
+    *Decided:* the three production jobs run on three independent schedules with no run-status
+    coupling between them, and the `:55` offset on the telemetry pull is an optimisation for
+    freshness, not a precondition — if it is missed, the forecast still runs. `promoted_model` was
+    deliberately *removed* from `live_forecasts`' deps once it became clear the model arrives by
+    filesystem path rather than by data flow: declaring the edge bought nothing at runtime and
+    created a permanently un-materialised parent on the production box, which has no MLflow and never
+    runs promotion. *Serves:*
+    [Hypothesis 1: a service that mostly runs itself](engineering-hypotheses.md#h1-a-service-that-mostly-runs-itself).
+    *Detail:* [Inherent Stability](inherent-stability.md#the-rules), rule 11.
+
 ## Deliberately absent
 
 We have **no availability service-level objective (SLO) and no error

docs/design-philosophy/inherent-stability.md

@@ -151,8 +151,8 @@ Nothing here is a 2am page. The uptime posture that makes that acceptable is arg
 ## The rules
 
 These are the imperative form of everything above: the checklist to follow when in doubt while
-changing production code. It is deliberately self-contained, so three of the ten restate a
-[design principle](design-principles.md) rather than adding anything new. Those three are marked
+changing production code. It is deliberately self-contained, so four of the eleven restate a
+[design principle](design-principles.md) rather than adding anything new. Those four are marked
 below — if you change one, change its matching principle too. The remaining seven are specific to
 degradation and appear nowhere else.
 
@@ -185,6 +185,12 @@ degradation and appear nowhere else.
 10. **Damp the corrections.** Bounded retries with backoff, rate limits on retraining and hysteresis
     on model promotion (the latter two designed but not built 🚧) are as much a part of this
     principle as the degradation ladder is.
+11. **Never make one production job's run status a precondition for another's.** Couple them through
+    data at rest: read whatever is on disk, record how stale it was, and carry on. A dependency in
+    the *lineage* graph is fine and useful — it is what builds a developer's inputs on a laptop — but
+    it must never become a runtime gate, because a gate turns one failed upstream run into a missing
+    forecast, which is rung 0 of the ladder reached by the one route the ladder cannot catch. *(The
+    coupling-through-data-at-rest principle, in imperative form.)*
 
 ## Where complexity should live
 

docs/live_service/operations.md

@@ -76,11 +76,13 @@ then point the task definition at the new tag — see
 
 Once a model is promoted, `live_forecasts` produces a new forecast automatically every 6 hours —
 at 00:00, 06:00, 12:00, and 18:00 UTC — via `live_forecasts_schedule`. `power_time_series_and_metadata`
-(a separate, hourly-scheduled job `live_forecasts` depends on but isn't ordered against) is
-itself scheduled 5 minutes *before* each hour so that hour's pull has landed by the time
-`live_forecasts` ticks — a cheap mitigation, not a guarantee; see
-`power_time_series_and_metadata_schedule`'s docstring (`defs/schedules.py`) for the more rigorous
-fix still to explore. This needs the Dagster daemon running (see
+(a separate, hourly-scheduled job `live_forecasts` depends on but is deliberately not ordered
+against) is itself scheduled 5 minutes *before* each hour so that hour's pull has landed by the
+time `live_forecasts` ticks. That offset is an optimisation for freshness, not a precondition: if
+the pull fails or runs long, the forecast still goes out on time against whatever is already on
+disk, and records how stale that input was — see
+[Inherent Stability, rule 11](../design-philosophy/inherent-stability.md#the-rules). This needs the
+Dagster daemon running (see
 [Prerequisites](#prerequisites-a-running-dagster-instance) above) to fire on time.
 
 To materialise one 6-hourly slot yourself — e.g. right after promoting a model, so you don't have

src/nged_substation_forecast/defs/schedules.py

@@ -28,13 +28,14 @@
 )
 """Fires at :55 past every hour — 5 minutes *before* the top of the hour — so this hour's pull
 has landed by the time ``live_forecasts_schedule`` ticks at 00/06/12/18 UTC.
-``live_forecasts`` depends on ``power_time_series_and_metadata`` (``defs/production_assets.py``)
-but the two run as separate jobs on separate schedules, so nothing actually enforces the
-ordering; this offset is a cheap mitigation, not a guarantee.
 
-TODO: explore a more rigorous fix — e.g. a run-status sensor that only fires
-``live_forecasts_job`` once that hour's ``power_time_series_and_metadata_job`` has actually
-succeeded, rather than assuming a fixed offset is always enough."""
+``live_forecasts`` declares ``power_time_series_and_metadata`` as a dep, but the two run as
+separate jobs on separate schedules and nothing enforces the ordering at runtime. That is
+deliberate, not a gap: the offset is an optimisation for freshness, and if it is missed —
+because this pull failed, or ran long — ``live_forecasts`` still runs on time against whatever
+is already on disk, and records how stale that input was. Making the ordering a precondition
+would convert one failed ingest into a missing forecast, which is the failure mode the whole
+design exists to avoid. See ``docs/design-philosophy/inherent-stability.md``, rule 11."""
 
 ecmwf_ens_job = define_asset_job(
     "ecmwf_ens_job",
@@ -69,5 +70,6 @@ def ecmwf_ens_schedule(context: ScheduleEvaluationContext) -> RunRequest:
 live_forecasts_schedule = build_schedule_from_partitioned_job(live_forecasts_job)
 """Ticks at 00/06/12/18 UTC, materialising the just-completed window with default run config
 (``availability_mode="live"``) — the schedule is always live; replays are manual, launched from
-the UI with ``availability_mode="replay"``. See ``power_time_series_and_metadata_schedule``'s
-docstring above for how the two schedules' relative timing is (loosely) coordinated."""
+the UI with ``availability_mode="replay"``. This slot fires on the clock regardless of whether
+the ingest jobs succeeded; see ``power_time_series_and_metadata_schedule``'s docstring above for
+why the two schedules are deliberately not ordered against each other."""