Airflow pipelines for Open Climate Fix's production systems
Note
This repo is a migration of the dags folder from the
ocf-infrastructure
repository. Commit history and authorship has not been preserved,
so for previous iterations of the DAGs, see the original repository's history.
Copy the airflow_dags folder into your dags location:
$ cp -r airflow_dags /path/to/airflow/dagsOr use the build webserver image in your containerized airflow deployment:
$ docker pull ghcr.io/openclimatefix/airflow-dagsSee the docker-compose file in the ocf-infrastructure repository.
TODO
Try to avoid it! The DAG name is how airflow identifies the DAG in the database. If you change the name of a DAG, airflow will treat it as a new DAG. This means that the old DAG will still be in the database, but it will not be updated or run.
This project uses MyPy for static type checking and Ruff for linting. Installing the development dependencies makes them available in your virtual environment.
Use them via:
$ python -m mypy .
$ python -m ruff check .Be sure to do this periodically while developing to catch any errors early and prevent headaches with the CI pipeline. It may seem like a hassle at first, but it prevents accidental creation of a whole suite of bugs.
There are some additional dependencies to be installed for running the tests,
be sure to pass --extra=dev to the pip install -e . command when creating your virtualenv.
(Or use uv and let it do it for you!)
Run the unit tests with:
$ python -m unittest discover -s tests -p "test_*.py"On the directory structure:
- The official PyPA discussion on "source" and "flat" layouts.
- PR's are welcome! See the Organisation Profile for details on contributing
- Find out about our other projects in the here
- Check out the OCF blog for updates
- Follow OCF on LinkedIn
Part of the Open Climate Fix community.
