script to make samples

[peterdudfield]

clear script to make batches, do we need extra compute for this? We might have this already.

Something in https://github.com/openclimatefix/ocf-data-sampler might already be there

[alirashidAR]

Hi @peterdudfield,
I would love to contribute to this project and help with this issue. Could you please guide me on how I can get started? Let me know if there are specific tasks or areas where assistance is needed.

[peterdudfield]

This would be great, whats needed first is openclimatefix/PVNet#2 thought, so you might want to start there first

[siddharth7113]

@peterdudfield What exactly needs to be done in this? Can you provide any references, so I can get started on this ?
Also does need to be precceded by #60

[peterdudfield]

We have done something similar in PVnet, for example https://github.com/openclimatefix/PVNet/blob/dev-data-sampler/scripts/save_samples.py
but it wont be exactly the same

[peterdudfield]

Notes from our meeting:
@Sukh-P @siddharth7113 @jcamier

Flow diagram to show how ocf-data-sampler works: https://github.com/openclimatefix/ocf-data-sampler/tree/main/ocf_data_sampler/torch_datasets

The torch dataset in datasampler which can be used for sample creation:
https://github.com/openclimatefix/ocf-data-sampler/blob/main/ocf_data_sampler/torch_datasets/datasets/pvnet_uk.py#L171

The branch on PVNet which supports ocf-data-sampler:
PVnet current branch - https://github.com/openclimatefix/PVNet/tree/dev-data-sampler

The configuration file example which will need to be set to correct values for GFS/other NWPs
config - example - https://github.com/openclimatefix/PVNet/blob/dev-data-sampler/configs.example/datamodule/configuration/example_configuration.yaml

The script to run to make samples - https://github.com/openclimatefix/PVNet/blob/dev-data-sampler/scripts/save_samples.py

• 10x10 should be a grid size (image pixel height/width) for GFS. This should include the whole UK. Good to check this
• Will need to add a bit of loading logic to ocf-data-sampler here: https://github.com/openclimatefix/ocf-data-sampler/tree/main/ocf_data_sampler/load/nwp to get it to work with gfs, this can be copied/pasted from here https://github.com/openclimatefix/ocf_datapipes/blob/main/ocf_datapipes/load/nwp/providers/gfs.py

Would be good to commit data configuration in open-data-pvnet repo, so people can use it same. Later on we can add the PVnet ML configuration as well

Agree that using ocf-data-sampler would be good to use.

[jcamier]

@alirashidAR I believe you have worked on ocf-data-sampler. I am getting an AttributeError: 'Dataset' object has no attribute 'installedcapacity_mwp' when trying to use the PVNetUKRegionalDataset and having a config that points to the PVLive data on our s3 bucket: s3://ocf-open-data-pvnet/data/uk/pvlive/v0/target_data.nc
I only see the following attributes in the target_data.nc:
<xarray.Dataset> Size: 4MB
Dimensions: (index: 87699)
Coordinates:

• index (index) int64 702kB 0 1 2 3 4 ... 87695 87696 87697 87698
  Data variables:
  gsp_id (index) int64 702kB ...
  datetime_gmt (index) datetime64[ns] 702kB ...
  generation_mw (index) float64 702kB ...
  capacity_mwp (index) float64 702kB ...

It appears we have capacity_mwp and not installedcapacity_mwp . Do you have any thoughts on this as you have worked on the PVLive data? Trying to determine if this is a bug on PVNetUKRegionalDataset side or if we are missing an attribute in the pvlive target_data ?

[alirashidAR]

Hello @jcamier , yes i believe the installedcapacity_mwp attribute is missing from the target_data.

open-data-pvnet/src/open_data_pvnet/scripts/collect_pvlive_data.py

Line 17 in 8d3eeef

data = pv.get_data_between(start=start, end=end, extra_fields="capacity_mwp")

This script was used to collect the data.