You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml)[](https://github.com/openclimatefix/ocf-meta-repo?tab=readme-ov-file#overview-of-ocfs-nowcasting-repositories)
7
10
8
-
9
11
This project is used for training PVNet and running PVNet on live data.
10
12
11
13
PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
@@ -18,7 +20,6 @@ feature vector is put through an output network which outputs predictions of the
18
20
future GSP yield. National forecasts are made by adding all the GSP forecasts
19
21
together.
20
22
21
-
22
23
## Experiments
23
24
24
25
Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
@@ -28,8 +29,6 @@ Some slightly more structured notes on deliberate experiments we have performed
28
29
Some very rough, early working notes on this model are
29
30
[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.
30
31
31
-
32
-
33
32
## Setup / Installation
34
33
35
34
```bash
@@ -39,9 +38,11 @@ pip install .
39
38
```
40
39
41
40
The commit history is extensive. To save download time, use a depth of 1:
@@ -114,11 +113,14 @@ Then exit this environment, and enter back into the pvnet conda environment and
114
113
pip install -e <PATH-TO-ocf-data-sampler-REPO>
115
114
```
116
115
116
+
If you install the local version of `ocf-data-sampler` that is more recent than the version specified in PVNet, you might receive a warning. However, it should still function correctly.
117
+
117
118
## Generating pre-made batches of data for training/validation of PVNet
118
119
119
120
PVNet contains a script for generating batches of data suitable for training the PVNet models. To run the script you will need to make some modifications to the datamodule configuration.
120
121
121
122
Make sure you have copied the example configs (as already stated above):
123
+
122
124
```
123
125
cp -r configs.example configs
124
126
```
@@ -127,7 +129,6 @@ cp -r configs.example configs
127
129
128
130
We will use the following example config file for creating batches: `/PVNet/configs/datamodule/configuration/example_configuration.yaml`. Ensure that the file paths are set to the correct locations in `example_configuration.yaml`: search for `PLACEHOLDER` to find where to input the location of the files. You will need to comment out or delete the parts of `example_configuration.yaml` pertaining to the data you are not using.
129
131
130
-
131
132
When creating batches, an additional datamodule config located in `PVNet/configs/datamodule` is passed into the batch creation script: `streamed_batches.yaml`. Like before, a placeholder variable is used when specifying which configuration to use:
132
133
133
134
```yaml
@@ -151,6 +152,7 @@ Run the `save_samples.py` script to create batches with the parameters specified
151
152
```bash
152
153
python scripts/save_samples.py
153
154
```
155
+
154
156
PVNet uses
155
157
[hydra](https://hydra.cc/) which enables us to pass variables via the command
156
158
line that will override the configuration defined in the `./configs` directory, like this:
@@ -185,7 +187,6 @@ satellite:
185
187
186
188
ocf-data-sampler is currently set up to use 11 channels from the satellite data, the 12th of which is HRV and is not included in these.
187
189
188
-
189
190
### Training PVNet
190
191
191
192
How PVNet is run is determined by the extensive configuration in the config
@@ -194,13 +195,13 @@ files. The configs stored in `PVNet/configs.example` should work with batches cr
194
195
Make sure to update the following config files before training your model:
195
196
196
197
1. In `configs/datamodule/local_premade_batches.yaml`:
197
-
- update `batch_dir` to point to the directory you stored your batches in during batch creation
198
+
- update `batch_dir` to point to the directory you stored your batches in during batch creation
198
199
2. In `configs/model/local_multimodal.yaml`:
199
-
- update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
200
-
- `in_channels`: number of variables your NWP source supplies
201
-
- `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
200
+
- update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
201
+
- `in_channels`: number of variables your NWP source supplies
202
+
- `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
202
203
3. In `configs/local_trainer.yaml`:
203
-
- set `accelerator: 0` if running on a system without a supported GPU
204
+
- set `accelerator: 0` if running on a system without a supported GPU
204
205
205
206
If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
206
207
your customised config files:
@@ -228,7 +229,6 @@ python run.py
228
229
229
230
If you have successfully trained a PVNet model and have a saved model checkpoint you can create a backtest using this, e.g. forecasts on historical data to evaluate forecast accuracy/skill. This can be done by running one of the scripts in this repo such as [the UK GSP backtest script](scripts/backtest_uk_gsp.py) or the [the pv site backtest script](scripts/backtest_sites.py), further info on how to run these are in each backtest file.
230
231
231
-
232
232
## Testing
233
233
234
234
You can use `python -m pytest tests` to run tests
@@ -238,8 +238,11 @@ You can use `python -m pytest tests` to run tests
238
238
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
239
239
240
240
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
241
+
241
242
<!-- prettier-ignore-start -->
243
+
242
244
<!-- markdownlint-disable -->
245
+
243
246
<table>
244
247
<tbody>
245
248
<tr>
@@ -258,6 +261,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
0 commit comments