Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft modular config structure #1174

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
2f555d0
Add a geometry config
ekatef Nov 6, 2024
f70a6b0
Add a heating config
ekatef Nov 6, 2024
7b3f4a4
Add monte-carlo config
ekatef Nov 6, 2024
792d4a8
Add a plotting config
ekatef Nov 6, 2024
770951e
Add a solving config
ekatef Nov 6, 2024
457c66a
Add a transport config
ekatef Nov 6, 2024
0e554c5
Add a weather config
ekatef Nov 6, 2024
a8ecb2c
Clean-up the default config
ekatef Nov 6, 2024
17b1345
Add an electricity config
ekatef Nov 6, 2024
2b46e5e
Move solar thermal to the weather config
ekatef Nov 6, 2024
f1a0c14
Add a hydrogen config
ekatef Nov 6, 2024
a38a287
Add a costs config
ekatef Nov 6, 2024
fcdef01
Rename a geometry config
ekatef Nov 11, 2024
a054730
Merge the config-related files
ekatef Nov 11, 2024
414a21c
Merge the solving-related files
ekatef Nov 11, 2024
829d978
Include the modular configs into the workflow
ekatef Nov 11, 2024
8b77535
Revert "Merge the config-related files"
ekatef Nov 11, 2024
ccd98ec
Consolidate the configs related to cross-sectoral modelling
ekatef Nov 11, 2024
a0139ff
Remove the default config
ekatef Nov 11, 2024
f8015a0
Merge branch 'main' into structure_config
ekatef Jan 2, 2025
aace26a
Move sector parameters to the sector config
ekatef Jan 2, 2025
b52adf1
Add the missed sector scenario
ekatef Jan 2, 2025
628807a
Add a missed shape parameter
ekatef Jan 2, 2025
41fe6d7
Comment-out the default configuration file
ekatef Jan 2, 2025
f3ab5c7
Draft the config output
ekatef Jan 2, 2025
bad96d8
Add main config
ekatef Jan 3, 2025
edd2ed9
Merge branch 'main' into structure_config
ekatef Jan 4, 2025
927b711
Merge branch 'main' into structure_config
ekatef Jan 12, 2025
5607fae
Merge remote-tracking branch 'origin/structure_config' into structure…
ekatef Jan 12, 2025
69c6cdb
Sketch a parser of the yaml-config into a css-formatted html
ekatef Jan 13, 2025
ad18cbc
Fix color names
ekatef Jan 13, 2025
85d5806
Minor fixes
ekatef Jan 16, 2025
390a89b
Apply header styling
ekatef Jan 16, 2025
508f9a4
Adjust font
ekatef Jan 16, 2025
19a76ed
Increase width of the color fill
ekatef Jan 16, 2025
7613227
Restore newlines
ekatef Jan 16, 2025
7eceaac
Adjust content
ekatef Jan 16, 2025
e2d2d6c
Add a demo
ekatef Jan 16, 2025
8e557ee
Add pre-commit fixes
ekatef Jan 16, 2025
ed20842
Merge branch 'main' into structure_config
ekatef Jan 28, 2025
b9a1022
Merge branch 'main' into structure_config
ekatef Jan 29, 2025
72b5968
Add output of config variable into a run-specific yaml
ekatef Jan 29, 2025
6a93bdd
Add extractor of the congig keys
ekatef Jan 29, 2025
5813a2f
Add a simple configuration parser
ekatef Jan 29, 2025
b734d90
Add a license string
ekatef Jan 29, 2025
aaeeabf
Add a configuration parser to Snakemake
ekatef Jan 29, 2025
c19a2bb
Add a config output function
ekatef Jan 29, 2025
8654e56
Add pre-commit fixes
ekatef Jan 29, 2025
3cc345a
Merge branch 'main' into structure_config
ekatef Jan 29, 2025
5a08314
Add some 'minimal' content for the default config
ekatef Jan 29, 2025
b20fa24
Merge remote-tracking branch 'origin/structure_config' into structure…
ekatef Jan 29, 2025
b934581
Remove debug prints
ekatef Jan 29, 2025
0bbd6b1
Merge branch 'main' into structure_config
ekatef Feb 5, 2025
6f3147b
Remove a duplicatied parameter
ekatef Feb 5, 2025
82bfb4f
Implement Leon's suggestions
ekatef Feb 5, 2025
e97684b
Add TODOs to the sector config
ekatef Feb 5, 2025
9b6f0dd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 5, 2025
73d1e5b
Merge branch 'main' into structure_config
ekatef Feb 10, 2025
9266fe6
Merge remote-tracking branch 'origin/structure_config' into structure…
ekatef Feb 10, 2025
c5eda56
Update the shapes sub-config
ekatef Feb 10, 2025
114753f
Re-arrange blocks in the shapes sub-config
ekatef Feb 10, 2025
ce91bfe
Fix formatting of the color codes
ekatef Feb 10, 2025
4686686
Update the sectoral config
ekatef Feb 10, 2025
43a1d0d
Update the clustering config
ekatef Feb 10, 2025
e9b2682
Fix the license indentificators
ekatef Feb 12, 2025
8c5dec9
Add reuse-ignore tags
ekatef Feb 12, 2025
4d96279
Add functionality to output a clean version of the config
ekatef Feb 12, 2025
37a50b4
Add output of the clean config to the workflow
ekatef Feb 12, 2025
515d7d0
Add docstrings
ekatef Feb 12, 2025
676c0a2
Overwrite the summary file
ekatef Feb 12, 2025
7444d53
Minor formatting fixes
ekatef Feb 12, 2025
6cc8a59
Shorten a list of renewable parameters for the default config
ekatef Feb 12, 2025
4e5811a
Rearrange the main congig
ekatef Feb 12, 2025
3670e96
Update the config documentation
ekatef Feb 12, 2025
d809319
Merge branch 'main' into structure_config
ekatef Feb 12, 2025
c258d54
Revise naming
ekatef Feb 14, 2025
caeee3c
Add processing of cutout inputs
ekatef Feb 18, 2025
a53bff1
Add processing of the cutout inputs into Snakemake
ekatef Feb 18, 2025
3abc3ee
Update naming
ekatef Feb 18, 2025
0917bf6
Fix refuse
ekatef Feb 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
743 changes: 0 additions & 743 deletions config.default.yaml

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions configs/config.costs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

# TODO: Needs to be adjusted for Africa.
costs:
year: 2030
version: v0.6.2
discountrate: [0.071] #, 0.086, 0.111]
# [EUR/USD] ECB: https://www.ecb.europa.eu/stats/exchange/eurofxref/html/eurofxref-graph-usd.en.html # noqa: E501
USD2013_to_EUR2013: 0.7532 # [EUR/USD] ECB: https://www.ecb.europa.eu/stats/exchange/eurofxref/html/eurofxref-graph-usd.en.html
rooftop_share: 0.14 # based on the potentials, assuming (0.1 kW/m2 and 10 m2/person)
fill_values:
FOM: 0
VOM: 0
efficiency: 1
fuel: 0
investment: 0
lifetime: 25
CO2 intensity: 0
discount rate: 0.07
marginal_cost: # EUR/MWh
solar: 0.01
onwind: 0.015
offwind: 0.015
hydro: 0.
H2: 0.
electrolysis: 0.
fuel cell: 0.
battery: 0.
battery inverter: 0.
emission_prices: # in currency per tonne emission, only used with the option Ep
co2: 0.
# investment: # EUR/MW
# CCGT: 830000
# FOM: # %/year
# CCGT: 3.35
# VOM: # EUR/MWh
# CCGT: 4.2
# fuel: # EUR/MWh
# gas: 10.1
# lifetime: # years
# CCGT: 25.0
# efficiency: # per unit
# CCGT: 0.58
lines:
length_factor: 1.25 #to estimate offwind connection costs
71 changes: 71 additions & 0 deletions configs/config.elec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

clean_osm_data_options: # osm = OpenStreetMap
names_by_shapes: true # Set the country name based on the extended country shapes
threshold_voltage: 51000 # [V] minimum voltage threshold to keep the asset (cable, line, generator, etc.) [V]
tag_substation: "transmission" # Filters only substations with 'transmission' tag, ('distribution' also available)
add_line_endings: true # When "True", then line endings are added to the dataset of the substations
generator_name_method: OSM # Methodology to specify the name to the generator. Options: OSM (name as by OSM dataset), closest_city (name by the closest city)
use_custom_lines: "OSM_only" # Use OSM (OSM_only), customized (custom_only), or both data sets (add_custom)
path_custom_lines: false # If exists, provide the specific absolute path of the custom file e.g. (...\data\custom_lines.geojson)
use_custom_substations: "OSM_only" # Use OSM (OSM_only), customized (custom_only), or both data sets (add_custom)
path_custom_substations: false # If exists, provide the specific absolute path of the custom file e.g. (...\data\custom_substations.geojson)
use_custom_cables: "OSM_only" # Use OSM (OSM_only), customized (custom_only), or both data sets (add_custom)
path_custom_cables: false # If exists, provide the specific absolute path of the custom file e.g. (...\data\custom_cables.geojson)

build_osm_network: # Options of the build_osm_network script; osm = OpenStreetMap
group_close_buses: true # When "True", close buses are merged and guarantee the voltage matching among line endings
group_tolerance_buses: 5000 # [m] (default 5000) Tolerance in meters of the close buses to merge
split_overpassing_lines: true # When True, lines overpassing buses are splitted and connected to the bueses
overpassing_lines_tolerance: 1 # [m] (default 1) Tolerance to identify lines overpassing buses
force_ac: false # When true, it forces all components (lines and substation) to be AC-only. To be used if DC assets create problem.

base_network:
min_voltage_substation_offshore: 51000 # [V] minimum voltage of the offshore substations
min_voltage_rebase_voltage: 51000 # [V] minimum voltage in base network

electricity:
base_voltage: 380.
voltages: [132., 220., 300., 380., 500., 750.]
co2limit: 7.75e+7 # European default, 0.05 * 3.1e9*0.5, needs to be adjusted for Africa
co2base: 1.487e+9 # European default, adjustment to Africa necessary
agg_p_nom_limits: data/agg_p_nom_minmax.csv
hvdc_as_lines: false # should HVDC lines be modeled as `Line` or as `Link` component?
automatic_emission: false
automatic_emission_base_year: 1990 # 1990 is taken as default. Any year from 1970 to 2018 can be selected.

operational_reserve: # like https://genxproject.github.io/GenX/dev/core/#Reserves
activate: false
epsilon_load: 0.02 # share of total load
epsilon_vres: 0.02 # share of total renewable supply
contingency: 0 # fixed capacity in MW


lines:
ac_types:
132.: "243-AL1/39-ST1A 20.0"
220.: "Al/St 240/40 2-bundle 220.0"
300.: "Al/St 240/40 3-bundle 300.0"
380.: "Al/St 240/40 4-bundle 380.0"
500.: "Al/St 240/40 4-bundle 380.0"
750.: "Al/St 560/50 4-bundle 750.0"
dc_types:
500.: "HVDC XLPE 1000"
s_max_pu: 0.7
s_nom_max: .inf
length_factor: 1.25
under_construction: "zero" # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity

links:
p_max_pu: 1.0
p_nom_max: .inf
under_construction: "zero" # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity

transformers:
x: 0.1
s_nom: 2000.
type: ""
52 changes: 52 additions & 0 deletions configs/config.geometry.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

# definition of the Coordinate Reference Systems
crs:
geo_crs: EPSG:4326 # general geographic projection, not used for metric measures. "EPSG:4326" is the standard used by OSM and google maps
distance_crs: EPSG:3857 # projection for distance measurements only. Possible recommended values are "EPSG:3857" (used by OSM and Google Maps)
area_crs: ESRI:54009 # projection for area measurements only. Possible recommended values are Global Mollweide "ESRI:54009"

cluster_options:
simplify_network:
to_substations: false # network is simplified to nodes with positive or negative power injection (i.e. substations or offwind connections)
algorithm: kmeans # choose from: [hac, kmeans]
feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.
exclude_carriers: []
remove_stubs: true
remove_stubs_across_borders: true
p_threshold_drop_isolated: 20 # [MW] isolated buses are being discarded if bus mean power is below the specified threshold
p_threshold_merge_isolated: 300 # [MW] isolated buses are being merged into a single isolated bus if a bus mean power is below the specified threshold
s_threshold_fetch_isolated: 0.05 # [-] a share of the national load for merging an isolated network into a backbone network
cluster_network:
algorithm: kmeans
feature: solar+onwind-time
exclude_carriers: []
alternative_clustering: false # "False" use Voronoi shapes, "True" use GADM shapes
distribute_cluster: ["load"] # Distributes cluster nodes per country according to ['load'],['pop'] or ['gdp']
out_logging: true # When "True", logging is printed to console
aggregation_strategies:
generators: # use "min" for more conservative assumptions
p_nom: sum
p_nom_max: sum
p_nom_min: sum
p_min_pu: mean
marginal_cost: mean
committable: any
ramp_limit_up: max
ramp_limit_down: max
efficiency: mean

build_shape_options:
gadm_layer_id: 1 # GADM level area used for the gadm_shapes. Codes are country-dependent but roughly: 0: country, 1: region/county-like, 2: municipality-like
update_file: false # When true, all the input files are downloaded again and replace the existing files
out_logging: true # When true, logging is printed to console
year: 2020 # reference year used to derive shapes, info on population and info on GDP
nprocesses: 3 # number of processes to be used in build_shapes
worldpop_method: "standard" # "standard" pulls from web 1kmx1km raster, "api" pulls from API 100mx100m raster,
# false (not "false") no pop addition to shape which is useful when generating only cutout
gdp_method: "standard" # "standard" pulls from web 1x1km raster, false (not "false") no gdp addition to shape which useful when generating only cutout
contended_flag: "set_by_country" # "set_by_country" assigns the contended areas to the countries according to the GADM database, "drop" drops these contended areas from the model
42 changes: 42 additions & 0 deletions configs/config.heating.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

sector:
district_heating:
potential: 0.3 #maximum fraction of urban demand which can be supplied by district heating
#increase of today's district heating demand to potential maximum district heating share
#progress = 0 means today's district heating share, progress=-1 means maximum fraction of urban demand is supplied by district heating
progress: 1
# 2020: 0.0
# 2030: 0.3
# 2040: 0.6
# 2050: 1.0
district_heating_loss: 0.15
reduce_space_heat_exogenously: true # reduces space heat demand by a given factor (applied before losses in DH)
# this can represent e.g. building renovation, building demolition, or if
# the factor is negative: increasing floor area, increased thermal comfort, population growth
reduce_space_heat_exogenously_factor: 0.29 # per unit reduction in space heat demand
# the default factors are determined by the LTS scenario from http://tool.european-calculator.eu/app/buildings/building-types-area/?levers=1ddd4444421213bdbbbddd44444ffffff11f411111221111211l212221
# 2020: 0.10 # this results in a space heat demand reduction of 10%
# 2025: 0.09 # first heat demand increases compared to 2020 because of larger floor area per capita
# 2030: 0.09
# 2035: 0.11
# 2040: 0.16
# 2045: 0.21
# 2050: 0.29

tes: true
tes_tau: # 180 day time constant for centralised, 3 day for decentralised
decentral: 3
central: 180
boilers: true
oil_boilers: false
chp: true
micro_chp: false
solar_thermal: true
heat_pump_sink_T: 55 #Celsius, based on DTU / large area radiators; used un build_cop_profiles.py
time_dep_hp_cop: true #time dependent heat pump coefficient of performance
solar_cf_correction: 0.788457 # = >>>1/1.2683
36 changes: 36 additions & 0 deletions configs/config.hydrogen.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

sector:
hydrogen:
network: true
H2_retrofit_capacity_per_CH4: 0.6
network_limit: 2000 #GWkm
network_routes: gas # "gas or "greenfield". If "gas" -> the network data are fetched from ["sector"]["gas"]["network_data"]. If "greenfield" -> the network follows the topology of electrical transmission lines
gas_network_repurposing: true # If true -> ["sector"]["gas"]["network"] is automatically false
underground_storage: false
hydrogen_colors: false
set_color_shares: false
blue_share: 0.40
pink_share: 0.05

hydrogen_underground_storage: true
shipping_hydrogen_liquefaction: false
shipping_average_efficiency: 0.4 #For conversion of fuel oil to propulsion in 2011

gadm_level: 1
h2_cavern: true
marginal_cost_storage: 0
methanation: true
helmeth: true
dac: true
SMR: true
SMR CC: true
cc_fraction: 0.9
cc: true
airport_sizing_factor: 3

min_part_load_fischer_tropsch: 0.9
37 changes: 37 additions & 0 deletions configs/config.monte-carlo.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# SPDX-FileCopyrightText: PyPSA-Earth and PyPSA-Eur Authors
#
# SPDX-License-Identifier: CC0-1.0

version: 0.4.1

monte_carlo:
# Description: Specify Monte Carlo sampling options for uncertainty analysis.
# Define the option list for Monte Carlo sampling.
# Make sure add_to_snakefile is set to true to enable Monte-Carlo
options:
add_to_snakefile: false # When set to true, enables Monte Carlo sampling
samples: 9 # number of optimizations. Note that number of samples when using scipy has to be the square of a prime number
sampling_strategy: "chaospy" # "pydoe2", "chaospy", "scipy", packages that are supported
seed: 42 # set seedling for reproducibilty
# Uncertanties on any PyPSA object are specified by declaring the specific PyPSA object under the key 'uncertainties'.
# For each PyPSA object, the 'type' and 'args' keys represent the type of distribution and its argument, respectively.
# Supported distributions types are uniform, normal, lognormal, triangle, beta and gamma.
# The arguments of the distribution are passed using the key 'args' as follows, tailored by distribution type
# normal: [mean, std], lognormal: [mean, std], uniform: [lower_bound, upper_bound],
# triangle: [mid_point (between 0 - 1)], beta: [alpha, beta], gamma: [shape, scale]
# More info on the distributions are documented in the Chaospy reference guide...
# https://chaospy.readthedocs.io/en/master/reference/distribution/index.html
# An abstract example is as follows:
# {pypsa network object, e.g. "loads_t.p_set"}:
# type: {any supported distribution among the previous: "uniform", "normal", ...}
# args: {arguments passed as a list depending on the distribution, see the above and more at https://pypsa.readthedocs.io/}
uncertainties:
loads_t.p_set:
type: uniform
args: [0, 1]
generators_t.p_max_pu.loc[:, n.generators.carrier == "onwind"]:
type: lognormal
args: [1.5]
generators_t.p_max_pu.loc[:, n.generators.carrier == "solar"]:
type: beta
args: [0.5, 2]
Loading
Loading