Load observations

[malerany]

Implement retrieval of observational data from the corresponding database as a method of the ObservationData class defined in src/pyhanami/diags/Observations.py.

Migrated from GitLab: https://gitlab.earth.bsc.es/ces/hanami/pyhanami/-/issues/5

[malerany]

In GitLab by @kellekai on Jul 25, 2025, 10:07 UTC:

@malerany

I think that we can provide scripts inside pyhanami to download observational data from ERA5 and other sources that offer an api for the download. For ERA5 for example:

• https://confluence.ecmwf.int/display/CKB/How+to+download+ERA5#HowtodownloadERA5-4-DownloadERA5familydatathroughtheCDSAPI

There is even a python package that looks promising:

• https://pypi.org/project/era5cli/
• https://era5cli.readthedocs.io/en/stable/getting_started/ (user guide)

we can provide a function to the package that lets the user specify the diagnostics he wanna do and the variables, and automatically populate a folder with a certain structure with the ERA5 data (and others) that pyhanami can handle.

[malerany]

Hi @kellekai, perhaps we could provide a set of observations by default with the package (adding it to the repository as you suggested in #10), from ERA5 and NOAA for instance to cover the whole 1850-2025 period, and provide the script in case the user wants to use observations from other sources?

[malerany]

Following the discussion in issue #10, here is a compilation of observational datasets (the tab Obs_datasets contains the characteristics of each dataset and the tab Available_variables the variables available for each one): Observations_pyhanami

[malerany]

After discussion with @kellekai today, for now, we would like to include monthly means of the following datasets:

• ERA5
• JRA-55
• 20CRv3
• MERRA-2
• ORAS5
• EN4

I will check the variables already available in esarchive, make the corresponding requests for the missing ones, and report in here when each dataset is complete.

The idea would be to create scripts to download a dataset when needed and, once downloaded, keep it somewhere within the structure of the package to be used in the future if requested again (similar to what cartopy does).

[malerany]

I added to the list some satellite-based observation datasets that Chihiro recommended while in JAMSTEC, and that they used to evaluate NICAM16-S in (C. Kodama et al., 2021). They are mainly related to precipitation and cloud features, as he mentioned that reanalyses are very model-dependent and models still fall short at reproducing real precipitation.

[malerany]

I checked esarchive, and these are the stored monthly variables: Available_on_esarchive

Only the variables from EN4 are available for the entire period covered by the dataset. For the other datasets, most variables are present, but some years are missing. I will make the corresponding requests.

Regarding the years, should I request the full available period for all datasets up to 2025?

[malerany]

After discussion with @kellekai, we will consider the period 1950-2025. I created requests for the variables/years missing for each dataset in this period:

• ERA5: #3046
• JRA-55: #3047
• MERRA-2: #3049
• ORAS5: #3048

[malerany]

We also have Interpolated OLR data from NOAA available on esarchive to use for the ISO metrics (#3072). It is daily data from the period 1974-2022 and occupies ~772 MB.

I will check if we could use the new BSC Dataverse to store the data there and download it with Python when necessary, as @kellekai suggested. I just saw that there is an introductory training session for this new tool in October, I will attend to learn how it works.

[malerany]

I requested 6-hourly variables necessary for the TCs metrics from ERA5 #3100 and JRA55 #3099.

[malerany]

Regarding how to provide observational data to the user, a few months ago I talked with Pierre-Antoine and got this information on possible approaches (posted in the GitLab repository #54):

• b2share: it seems similar to Zenodo, they provide an official DOI and guarantee the maintenance of the datasets, but ‘the maximum size for a record is 20 GB; and the maximum size for a file is 10 GB’. Then, it might not be very convenient depending on how many years of observations we want to make available.
• FTP Public server: there is a FTP server for Earth Sciences that we could use. This is easier to use and, in principle, there is no size limit. However, each dataset is protected, so a username and a password are needed to access it, and it would require downloading the data to use it, as it is not directly accessible with Python. Nevertheless, using the python library ftplib, I think we could temporarily download the observations needed when executing a test, perform the necessary calculations, and delete the observations dataset from the local user storage afterward.
• THREDDS Data Server: this is a web server that provides metadata and data access for scientific datasets and has the advantage that it allows both to download files and to access the data directly with Python. This way, the data could be directly read from esarchive. However, he told me that it requires considerably more work to mount the server, so it is not the recommended approach.

[malerany]

Yesterday, I attended an introductory training session for the BSC Dataverse, a new institutional research (meta)data repository for the whole BSC. For now, everyone is able to browse through the metadata in there, but an external user would need to register and create an account to be able to download data.

However, within the ES department, they are working to implement a solution that would allow us to link there data from esarchive so that external users can access and download it without the need for an account (likely using the THREDDS server mentioned above). They are still working on this feature but, since in the next months pyhanami will only be used by people from BSC with access to esarchive, I would suggest keeping the data there for now and waiting until we can share it easily through the Dataverse. As the alternatives mentioned above seem less convenient if we need to provide a lot of data.

What do you think @kellekai?