update readme and templates to reflect changes in datapipes #120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #120 +/- ##
=======================================
Coverage 57.76% 57.76%
=======================================
Files 26 26
Lines 1712 1712
=======================================
Hits 989 989
Misses 723 723 ☔ View full report in Codecov by Sentry. |
peterdudfield
approved these changes
Jan 10, 2024
zakwatts
changed the title
| @@ -23,3 +23,4 @@ hydra-core | |||
| python-dotenv | |||
| hydra-optuna-sweeper | |||
| rich | |||
| gcsfs | |||
There was a problem hiding this comment.
could you add a comment here, saying that this is only needed when getting data from gcsfs? Or is it needed more now?
There was a problem hiding this comment.
It's only needed when getting data from gcsfs. I'll add in a comment.
jacobbieker
reviewed
Jan 12, 2024
README.md
Outdated
| - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr" | ||
| - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr" | ||
| ``` | ||
| Datapipes is currently set up to use 12 channels from the satellite data which is the YEAR_nonhrv.zarr rather than YEAR_hrv.zarr. |
There was a problem hiding this comment.
11 channels, the 12th is HRV, which is not included in these ones.
configs.example/datamodule/configuration/template_configuration.yaml
Outdated
Show resolved
Hide resolved
jacobbieker
requested changes
Jan 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR mainly of edits to the documentation and template configs.
Commit history is very large and was causing issues when downloading, even with a GitHub buffer size increased to 1Gb I was running into download problems. Hence it has suggested that a clone depth can be used with cloning the repo.
Instructions on how to set up and get started with ocf_datapipes in order to create batches have been added alongside a more step by step guide through where to specify config files and google cloud set up instructions (with gcsfc being adding the requirements.txt)
Recent changes in ocf_datapipes also meant that the example configs were out of date, with multiple NWPs now being used the way the nwps in the config were set up have changed slightly (by an indentation, I believe to reflect it now being a dictionary of NWPs but I have not looked into this too much!)