Solar Consumer

This code can be used to download solar forecasts and save them to a PostgreSQL database. It fetches solar generation estimates for embedded solar farms and processes the data for analysis. We currently collect

• UK: Forecast can be retreived from NESO. Generation Data can be retrevied from PVLive.
• NL: Generation values from Ned NL, both national and region. National Forecast values from Ned NL too.
• DE: Generation values from ENTSOE for several TSOs.

Here are the different sources of data, and which methods can be used to save the results

Source	Country	CSV	Data Platform	DB (Legacy)	Site DB (Legacy)
PVLive	🇬🇧	✅	✅
NESO forecast	🇬🇧	✅		✅
Ned-nl	🇳🇱	✅			✅
Ned-nl forecast	🇳🇱	✅			✅
Germany (ENTSOE)	🇩🇪	✅			✅

Requirements

• Docker
• Docker Compose

Installation & Running

1. Clone the repository:

git clone https://github.com/openclimatefix/neso-solar-consumer.git
cd neso-solar-consumer

2. Copy the example environment file:

cp .example.env .env

3. Start the application:

docker compose up -d

The above command will:

• Start a PostgreSQL database container
• Build and start the NESO Solar Consumer application
• Configure all necessary networking between containers

To stop the application:

docker compose down

To view logs:

docker compose logs -f

Note: The PostgreSQL data is persisted in a Docker volume. To completely reset the database, use:

docker compose down -v

Documentation

The package provides three main functionalities:

1. Data Fetching: Retrieves solar forecast data from the NESO API
2. Data Formatting: Processes the data into standardized forecast objects
3. Data Storage: Saves the formatted forecasts to a PostgreSQL database

Key Components:

• fetch_data.py: Handles API data retrieval
• format_forecast.py: Converts raw data into forecast objects
• save_forecast.py: Manages database operations
• app.py: Orchestrates the entire pipeline

Environment Variables: (Can be found in the .example.env / .env file)

• DB_URL=postgresql://postgres:postgres@localhost:5432/neso_solar : Database Configuration
• COUNTRY="gb" : Country code for fetching data. Currently, other options are ["nl"]
• SAVE_METHOD="db": Ways to store the data. Currently other options are ["csv", "site-db"]
• CSV_DIR=None : Directory to save CSV files if SAVE_METHOD="csv".
• UK_PVLIVE_REGIME=in-day: For UK PVLive, the regime. Can be "in-day" or "day-after"
• UK_PVLIVE_N_GSPS=342: For UK PVLive, the amount of gsps we pull data for.
• UK_PVLIVE_BACKFILL_HOURS=2: For UK PVLive, the amount of backfill hours we pull, when regime="in-day"

Development

1. Set up the development environment:

pip install ".[dev]"

2. Run tests:

pytest

3. Format code:

black .

4. Run linter:

ruff check .

Running the Test Suite

The test suite includes unit tests and integration tests:

# Run all tests
pytest

# Run specific test file
pytest tests/test_fetch_data.py

# Run with coverage
pytest --cov=neso_solar_consumer

Continuous Integration (CI)

This reposistory has 2 main CI workflows - branch-ci and merged-ci.

• branch-ci is triggered on all pushes to any branch except main, and on any pull request that is opened, reopened or updated. It runs the tests suite, lints the project, and builds and pushes a dev image.
• merged-ci is triggered on any pull request merged into main. It bumps the git tag, and builds and pushes a container with that tag.

FAQ

Q: What format is the data stored in? A: The data is stored in PostgreSQL using SQLAlchemy models, with timestamps in UTC and power values in megawatts.

Q: How often should I run the consumer? A: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing and community

• PR's are welcome! See the Organisation Profile for details on contributing
• Find out about our other projects in the OCF Meta Repo
• Check out the OCF blog for updates
• Follow OCF on LinkedIn

Contributors

Peter Dudfield 🤔	Seao7 💻	Siddharth 👀 🚇 💻	Conor O Callaghan 📖	Ali Rashid ⚠️	Manzoor Ahmed Shaikh 💻	Anas Khan 📖
Peter Ireland 📖	vashisthrahul13 💻	rahul-ahuja 💻	Rahul Joon 💻	michael-gendy 🚇	Shohail Ismail 💻	Prafful Vyas 💻
Ramkumar R 💻

Part of the Open Climate Fix community.