preparing release 0.5.9 (#130)

* Add contributors (#105)

* Add contributors
Co-authored-by: Mario Santa Cruz <[email protected]>

* Feature/masks (#104)

* add masks

Co-authored-by: Florian Pinault <[email protected]>

* Feature/new checkpoints (#106)

* add masks

* Feature/new datasets (#99)

* main changes

* bugfix

* few bugs and add unit tests

* work with more planetary computer ds

* add optional dependencies

* qa

* make test optional when adls is not installed (#110)

* make test optional when adls is not installed

* changelog

* tests

* tests

* split tests

* Xarray-zarr example dataset recipe (#108)

* add a working xarray-zarr example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: b8raoult <[email protected]>

* missing copyrigths

* missing copyrigths (#111)

* missing copyrigths

* fixing --test (changing only the behaviour of creating datasets with --test)

* more on testing

* fix tests

* Feature/support multiple lams to the Cutout class (#113)

* Enhance Cutout class to support multiple LAMs with hierarchical masking.

---------

Co-authored-by: Paulina Met. <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#112)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/psf/black-pre-commit-mirror: 24.8.0 → 24.10.0](psf/black-pre-commit-mirror@24.8.0...24.10.0)
- [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2)
- [github.com/tox-dev/pyproject-fmt: 2.2.4 → v2.5.0](tox-dev/pyproject-fmt@2.2.4...v2.5.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update .pre-commit-config.yaml (#120)

* fix qa for the notebook on multilam (#123)

* feature/fix qa (#124)

* fix qa for the notebook on multilam
* fix qa for the yaml for multilam

* Naming guidelines not available to all users. fixing #119 (#125)

* upload with ssh (#94)

* add copy to ssh target

* Feature/new checkpoints (#107)

* add masks
* save masks to checkpoint
* name supporting_arrays
* better support for cutout
* force np.datetime64 is seconds
---------

Co-authored-by: Florian Pinault <[email protected]>

* Feature/merge (#126)

* save masks to checkpoint
* force np.datetime64 is seconds
* Call filters from anemoi-transform
* when merging datasets, consider missing dates
* add gcd for frequency

---------

Co-authored-by: Florian Pinault <[email protected]>

* Feature/use anemoi transform (#127)

* Call filters from anemoi-transform

---------

Co-authored-by: Florian Pinault <[email protected]>

* Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)"

* redo "Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)"

* fix merge

* Update to documentation on using missing dates (#128)

* Updated docs on using datasets with missing dates

* Simplify ci: run on develop, and on sundays. And disable downstream-ci-hpc. And test only python 3.11. And test only once when PR are updated. And shortest name to read the full description on github.ci. And test with only ubuntu. (same change as for anemoi-utils ecmwf/anemoi-utils#42) (#129)

* skipping long tests (#132)

---------

Co-authored-by: Matthew Chantry <[email protected]>
Co-authored-by: b8raoult <[email protected]>
Co-authored-by: Mariah Pope <[email protected]>
Co-authored-by: Timothy Smith <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Baudouin Raoult <[email protected]>
Co-authored-by: paulina-t <[email protected]>
Co-authored-by: Paulina Met. <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Håvard Homleid Haugen <[email protected]>

Author: 10 people

Diff for: .github/workflows/ci.yml

@@ -42,13 +42,20 @@ jobs:
     with:
       anemoi-datasets: ecmwf/anemoi-datasets@${{ github.event.pull_request.head.sha || github.sha }}
       codecov_upload: true
+      skip_matrix_jobs: |
+        gnu@debian-11
+        [email protected]
+        gnu@debian-11
+        [email protected]
+        [email protected]
+        gnu@fedora-37
     secrets: inherit
 
-   # Build downstream packages on HPC
-  downstream-ci-hpc:
-    name: downstream-ci-hpc
-    if: ${{ !contains(github.repository, 'private') && (!github.event.pull_request.head.repo.fork && github.event.action != 'labeled' || github.event.label.name == 'approved-for-ci') }}
-    uses: ecmwf-actions/downstream-ci/.github/workflows/downstream-ci-hpc.yml@main
-    with:
-      anemoi-datasets: ecmwf/anemoi-datasets@${{ github.event.pull_request.head.sha || github.sha }}
-    secrets: inherit
+#   # Build downstream packages on HPC
+#  downstream-ci-hpc:
+#    name: downstream-ci-hpc
+#    if: ${{ !contains(github.repository, 'private') && (!github.event.pull_request.head.repo.fork && github.event.action != 'labeled' || github.event.label.name == 'approved-for-ci') }}
+#    uses: ecmwf-actions/downstream-ci/.github/workflows/downstream-ci-hpc.yml@main
+#    with:
+#      anemoi-datasets: ecmwf/anemoi-datasets@${{ github.event.pull_request.head.sha || github.sha }}
+#    secrets: inherit

Diff for: .github/workflows/python-pull-request.yml

@@ -1,12 +1,17 @@
 # This workflow will upload a Python Package using Twine when a release is created
 # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
 
-name: Code Quality checks for PRs
+name: Test PR
 
 on:
-  push:
   pull_request:
     types: [opened, synchronize, reopened]
+  push:
+    branches:
+      - develop
+  schedule:
+    - cron: "9 2 * * 0" # at 9:02 on sunday
+
 
 jobs:
   quality:
@@ -17,7 +22,7 @@ jobs:
   checks:
     strategy:
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.11"]
     uses: ecmwf-actions/reusable-workflows/.github/workflows/qa-pytest-pyproject.yml@v2
     with:
       python-version: ${{ matrix.python-version }}

Diff for: .pre-commit-config.yaml

@@ -27,7 +27,7 @@ repos:
     -   id: python-check-blanket-noqa # Check for # noqa: all
     -   id: python-no-log-warn # Check for log.warn
 - repo: https://github.com/psf/black-pre-commit-mirror
-  rev: 24.8.0
+  rev: 24.10.0
   hooks:
   - id: black
     args: [--line-length=120]
@@ -40,7 +40,7 @@ repos:
     - --force-single-line-imports
     - --profile black
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.6.9
+  rev: v0.7.2
   hooks:
   - id: ruff
     args:
@@ -59,13 +59,8 @@ repos:
   hooks:
   - id: rstfmt
     exclude: 'cli/.*' # Because we use argparse
-- repo: https://github.com/b8raoult/pre-commit-docconvert
-  rev: "0.1.5"
-  hooks:
-  - id: docconvert
-    args: ["numpy"]
 - repo: https://github.com/tox-dev/pyproject-fmt
-  rev: "2.2.4"
+  rev: "v2.5.0"
   hooks:
   - id: pyproject-fmt
 

Diff for: CHANGELOG.md

@@ -10,6 +10,13 @@ Keep it human-readable, your future self will thank you!
 
 ## [Unreleased](https://github.com/ecmwf/anemoi-datasets/compare/0.5.8...HEAD)
 
+
+### Added
+
+- Call filters from anemoi-transform
+- make test optional when adls is not installed Pull request #110
+
+
 ## [0.5.8](https://github.com/ecmwf/anemoi-datasets/compare/0.5.7...0.5.8) - 2024-10-26
 
 ### Changed
@@ -25,9 +32,13 @@ Keep it human-readable, your future self will thank you!
 - Various bug fixes
 - Control compatibility check in xy/zip
 - Add `merge` feature
+- Add support for storing `supporting_arrays` in checkpoint files
+- Allow naming of datasets components
+- Contributors file (#105)
 
 ### Changed
 
+- Upload with ssh (experimental)
 - Remove upstream dependencies from downstream-ci workflow (temporary) (#83)
 - ci: pin python versions to 3.9 ... 3.12 for checks (#93)
 - Fix `__version__` import in init

Diff for: CONTRIBUTORS.md

@@ -0,0 +1,13 @@
+## How to Contribute
+
+Please see the [read the docs](https://anemoi-training.readthedocs.io/en/latest/dev/contributing.html).
+
+
+## Contributors
+
+Thank you to all the wonderful people who have contributed to Anemoi. Contributions can come in many forms, including code, documentation, bug reports, feature suggestions, design, and more. A list of code-based contributors can be found [here](https://github.com/ecmwf/anemoi-datasets/graphs/contributors).
+
+
+## Contributing Organisations
+
+Significant contributions have been made by the following organisations: [DWD](https://www.dwd.de/), [MET Norway](https://www.met.no/), [MeteoSwiss](https://www.meteoswiss.admin.ch/), [RMI](https://www.meteo.be/) & [ECMWF](https://www.ecmwf.int/)

Diff for: docs/building/sources/xarray-zarr.rst

@@ -2,5 +2,17 @@
  xarray-zarr
 #############
 
+Here is an example recipe that builds a dataset using one of the many
+regridded versions of ERA5 hosted by Google in Analysis-Ready, Cloud
+Optimized format. See `here
+<https://cloud.google.com/storage/docs/public-datasets/era5>`_ for more
+information.
+
 .. literalinclude:: yaml/xarray-zarr.yaml
    :language: yaml
+
+Note that unlike the ``mars`` examples, there is no need to include a
+``grid`` specification. Also, in order to subselect the vertical levels,
+it is necessary to use the :ref:`join <building-join>` operation to join
+separate lists containing 2D variables and 3D variables. If all vertical
+levels are desired, then it is OK to specify a single source.

Diff for: docs/building/sources/yaml/xarray-zarr.yaml

@@ -1,3 +1,22 @@
+dates:
+  start: 2021-01-01T00:00:00
+  end: 2021-01-10T12:00:00
+  frequency: 6h
+
 input:
-  xarray-zarr:
-    url: https://...
+  join:
+      - xarray-zarr:
+          url: "gs://gcp-public-data-arco-era5/ar/1959-2022-6h-128x64_equiangular_conservative.zarr"
+          param:
+              - surface_pressure
+              - 2m_temperature
+              - 10m_u_component_of_wind
+              - 10m_v_component_of_wind
+
+      - xarray-zarr:
+          url: "gs://gcp-public-data-arco-era5/ar/1959-2022-6h-128x64_equiangular_conservative.zarr"
+          param:
+              - temperature
+          level:
+              - 1000
+              - 500

Diff for: docs/using/missing.rst

@@ -4,6 +4,16 @@
  Managing missing dates
 ########################
 
+*********************************************
+ Managing missing dates with anemoi-training
+*********************************************
+
+Anemoi-training has internal handling of missing dates, and will
+calculate the valid date indices used during training using the
+``missing`` property. Consequenctly, when training a model with
+anemoi-training, you should `not` specify a method to deal with missing
+dates in the dataloader configuration file.
+
 **************************************************
  Filling the missing dates with artificial values
 **************************************************

Diff for: pyproject.toml

@@ -1,14 +1,12 @@
-#!/usr/bin/env python
-# (C) Copyright 2024 ECMWF.
+# (C) Copyright 2024 Anemoi contributors.
 #
 # This software is licensed under the terms of the Apache Licence Version 2.0
 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+#
 # In applying this licence, ECMWF does not waive the privileges and immunities
 # granted to it by virtue of its status as an intergovernmental organisation
 # nor does it submit to any jurisdiction.
 
-# https://packaging.python.org/en/latest/guides/writing-pyproject-toml/
-
 [build-system]
 requires = [
   "setuptools>=60",
@@ -42,6 +40,7 @@ classifiers = [
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: Implementation :: PyPy",
 ]
@@ -97,6 +96,9 @@ optional-dependencies.tests = [
 optional-dependencies.xarray = [
   "gcsfs",
   "kerchunk",
+  "pandas",
+  "planetary-computer",
+  "pystac-client",
 ]
 
 urls.Documentation = "https://anemoi-datasets.readthedocs.io/"

Diff for: src/anemoi/datasets/__init__.py

@@ -1,6 +1,8 @@
-# (C) Copyright 2023 European Centre for Medium-Range Weather Forecasts.
+# (C) Copyright 2024 Anemoi contributors.
+#
 # This software is licensed under the terms of the Apache Licence Version 2.0
 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+#
 # In applying this licence, ECMWF does not waive the privileges and immunities
 # granted to it by virtue of its status as an intergovernmental organisation
 # nor does it submit to any jurisdiction.

Diff for: src/anemoi/datasets/__main__.py

@@ -1,12 +1,11 @@
-#!/usr/bin/env python
-# (C) Copyright 2024 ECMWF.
+# (C) Copyright 2024 Anemoi contributors.
 #
 # This software is licensed under the terms of the Apache Licence Version 2.0
 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+#
 # In applying this licence, ECMWF does not waive the privileges and immunities
 # granted to it by virtue of its status as an intergovernmental organisation
 # nor does it submit to any jurisdiction.
-#
 
 from anemoi.utils.cli import cli_main
 from anemoi.utils.cli import make_parser

Diff for: src/anemoi/datasets/commands/__init__.py

@@ -1,12 +1,11 @@
-#!/usr/bin/env python
-# (C) Copyright 2024 ECMWF.
+# (C) Copyright 2024 Anemoi contributors.
 #
 # This software is licensed under the terms of the Apache Licence Version 2.0
 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+#
 # In applying this licence, ECMWF does not waive the privileges and immunities
 # granted to it by virtue of its status as an intergovernmental organisation
 # nor does it submit to any jurisdiction.
-#
 
 import os