Process Metoffice data ready, for ml team

[peterdudfield]

No description provided.

[peterdudfield]

Follows on from #31

[alirashidAR]

@peterdudfield I would like to take up this issue could you please assign it to me .

[peterdudfield]

THanks, please ask @jcamier for more details

[alirashidAR]

@jcamier, could you please provide a brief overview of this task? I’m currently occupied with mid-semester exams, so I’ll be able to work on it in 2 to 3 days. Thanks!

[peterdudfield]

@jcamier should be able to advise. Currently he is uploading lots of Metoffice data to huggingface. We need to process this so its in an easy to use format for the ML modelling. It should be a simialr shape to GFS data already online

[jcamier]

@alirashidAR I have a cron job that is getting the met office data from AWS netcdf files, converting to .zarr.zip and uploading them to hugging face ever hour due to rate limit constraints. Right now it is in March of 2023.
https://huggingface.co/datasets/openclimatefix/met-office-uk-deterministic-solar/tree/main/data/2023

The data is organized as follows data/<year>/<month>/<day>/.zarr.zip file for every hour of nwp forecast

I believe the next step is to be able to read this data using the ocf-data-sampler. I am currently reading up on ocf-data-sampler to get familiar with this. Maybe we can setup a time to discuss? Feel free to email me some dates/times that work for you.

[jcamier]

Here is a roadmap that Peter and I worked on a while back: