Renaming a variable changes the shape of the variable

[artificiallama]

I have only 1 variable (t_1000) in my anemoi-dataset file hrdps_2022.zarr.

I ran the anemoi-dataset with a short yaml file which only changes the name of the variable to T_1000. The input file (hrdps_2022.zarr) looks like,

Format version: 0.18.0

📅 Date      start: 2022-01-01 00:00
📅 Date      end  : 2022-12-31 18:00
⏰ Frequency  : 6h
🚫 Missing    : 0
🌎 Resolution : None
🌎 Field shape: [1290, 2540]

📅 Data start : None
📅 Data end   : None

📐 Shape      : 1,460 × 10 × 1 × 3,276,600 (178.2 GiB)
💽 Size       : 69.2 GiB (69.2 GiB)
📁 Files      : 1,515


                                  Statistics                                   
┏━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Index ┃ Variable       ┃    Min ┃    Max ┃     Mean ┃ Stdev ┃ Units         ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━┩
│     0 │ cos_julian_day │     -1 │      1 │   -0.189 │  0.66 │ dimensionless │
│     1 │ cos_latitude   │  0.332 │  0.889 │    0.621 │ 0.125 │ dimensionless │
│     2 │ cos_local_time │     -1 │      1 │ 2.63e-08 │ 0.707 │ dimensionless │
│     3 │ cos_longitude  │ -0.889 │  0.758 │  -0.0481 │ 0.429 │ dimensionless │
│     4 │ insolation     │      0 │  0.972 │    0.238 │  0.29 │ dimensionless │
│     5 │ sin_julian_day │     -1 │      1 │    0.139 │ 0.714 │ dimensionless │
│     6 │ sin_latitude   │  0.458 │  0.943 │    0.767 │ 0.102 │ dimensionless │
│     7 │ sin_local_time │     -1 │      1 │ 1.05e-09 │ 0.707 │ dimensionless │
│     8 │ sin_longitude  │     -1 │ -0.458 │   -0.896 │ 0.103 │ dimensionless │
│     9 │ t_1000         │    220 │    321 │      280 │  14.7 │ None          │
└───────┴────────────────┴────────┴────────┴──────────┴───────┴───────────────┘

anemoi-datasets create ./hrdps_rename.yaml /dataset/hrdps_2022_rename.zarr --overwrite

The yaml file is,

dates:
  start      : '2022-01-01 00:00'
  end        : '2022-12-31 18:00'
  frequency  : 6

input:
  anemoi-dataset:
      dataset: /dataset/hrdps_2022.zarr
      rename:
        t_1000: T_1000

However, this changes the shape of the variable. In the output file (hrdps_2022_rename.zarr) the shape turns out to be,
Field shape: [3276600] which was Field shape: [1290, 2540] in the source file.

The shape of the variable gives rise to an error during (stretched grid ) training which uses hrdps_2022_rename.zarr file. The error is,

Thinning only works latitude/longitude fields

Why does the shape change and how does one create a new dataset with new names while preserving the shape of the variable ?

[b8raoult]

Can you give us access to your input Zarr? Maybe a small subset?