NGED Flexpectation is an NIA-funded project by Open Climate Fix to deliver state-of-the-art, probabilistic power forecasts for National Grid Electricity Distribution (NGED). The forecasts cover NGED's substations and customer meters, with a 14-day horizon at half-hourly resolution, updated every 6 hours. The goal is to help NGED optimise flexibility procurement and manage network congestion.
This repository is the research and production-harness codebase. The system is orchestrated with Dagster, uses XGBoost as the initial forecasting model, and stores all data and forecasts as Delta Lake tables on S3.
To external contributors: this repo is in early-stage development with frequent breaking changes, so it is not suitable for external contributions at the moment.
For detailed information about the project, including architecture, design philosophy, and user guides, please visit our documentation site.
This repo is a uv workspace: A single repo which contains multiple Python packages.
The full first-run walkthrough — including creating your .env, giving Dagster a persistent
home, and downloading data to train your first model — is the
Getting started guide.
The essentials:
- Ensure
uvis installed following their official documentation. - Install dependencies:
uv sync - Install pre-commit hooks:
uv run pre-commit install - Create your
.env:cp .env.example .env. Every setting has a working default, so you can leave it as-is; fill in theNGED_S3_BUCKET_*credentials only to ingest NGED telemetry. - Run Dagster:
uv run dg dev, then openhttp://localhost:3000in your browser.
- Check linting:
uv run ruff check . - Fix linting:
uv run ruff check . --fix— but see the warning below before running this over a marimo notebook. - Format code:
uv run ruff format . - Type checking:
uv run ty check - Markdown linting:
uv run pymarkdown scan -r docs README.md CLAUDE.md packages/*/README.md
Never run
ruff check --fixover a marimo notebook (packages/notebooks/*.py,packages/dashboard/{map_and_timeseries,view_forecasts}.py). When a fix needs a name the file does not import yet, ruff adds the import at module level — outsidewith app.setup:, where no cell can see it — and the notebook then fails with aNameErrorwhileruff checkreports success. The pre-commit hook is split so notebooks are checked but never auto-fixed; a--fixyou type yourself is not covered.
Markdown (README.md files, docs/*.md, and Python docstrings) is linted automatically by the pre-commit hook, but when developing code or docs it's a good idea to run the markdown lint command above yourself before committing, for faster feedback than waiting on the commit-time hook.
- Run all tests:
uv run pytest - Run tests with coverage:
uv run pytest --cov
- Run Dagster UI:
uv run dagster dev - Open
http://localhost:3000in your browser to see the project. - Run Marimo notebooks:
uv run marimo edit packages/notebooks/some_notebook.py - View MLflow experiments:
uv run mlflow ui --gunicorn-opts "--workers 1", then openhttp://localhost:5000. The--gunicorn-optsflag is required on Python 3.14 — MLflow's default uvicorn+FastAPI server fails to start there; see the ML experimentation docs.
The docs are built with MkDocs (Material theme). The tooling is part of the dev dependency group, so uv sync installs it.
- Serve docs locally with live reload:
uv run mkdocs serve, then openhttp://localhost:8000. The site rebuilds automatically as you edit files indocs/. - Build the static site:
uv run mkdocs build— renders the docs into thesite/directory.
Part of the Open Climate Fix community.
