Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transport improvements for 2024-W32 #213

Merged
merged 77 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
82ec270
Update freight-activity.csv for n=R12_CHN
khaeru Aug 2, 2024
b3ca481
Dispatch message_data transport workflow from dev branch
khaeru Aug 2, 2024
8b412fb
Add path_fallback(… where="local") option
khaeru Aug 5, 2024
2620f61
Use path_fallback in .tools.iea.web
khaeru Aug 5, 2024
8520626
Fill scale-1 factor with 1.0 in .transport.base
khaeru Aug 5, 2024
d9e4b00
Give explicit "dimensionless" in .factor.Constant
khaeru Aug 6, 2024
d846402
Add ldv-t-share.csv & entry in .transport.files
khaeru Aug 6, 2024
638799b
Add .ldv.stock to calculate stock/sales
khaeru Aug 6, 2024
8593d94
Add .transport.Config.ldv_stock_method
khaeru Aug 6, 2024
5df1d64
Switch method in .ldv.prepare_computer()
khaeru Aug 6, 2024
f490a62
Update transport/R12/pdt-cap-ref.csv for #213
r-aneeque Aug 6, 2024
3f0f54e
Update tests for #213
khaeru Aug 6, 2024
0e758f2
Add "million * v" to transport/set.yaml
khaeru Aug 7, 2024
a64e493
Add initial_activity_up in .ldv.constraint_data()
khaeru Aug 7, 2024
c3326e4
Add extend_y(…, dim="y") keyword argument
khaeru Aug 7, 2024
07d64e1
Update R12_CHN mode share values
r-aneeque Aug 7, 2024
5500c4a
Update transport/R12/load-factor-ldv.csv for #213
r-aneeque Aug 7, 2024
9aec937
Update R12/ldv-activity.csv for #213
r-aneeque Aug 7, 2024
3d53849
Move tests of .transport.files to parallel module
khaeru Aug 7, 2024
251991d
Reorganize .transport.test_files
khaeru Aug 7, 2024
78c0543
Handle SDMX structural metadata for transport inputs
khaeru Aug 7, 2024
9ab564b
Add ldv-age.csv and entry in .transport.files
khaeru Aug 12, 2024
788014f
Compute sales_fraction_annual for each (n, t)
khaeru Aug 12, 2024
a99979a
Add data/transport/mode-share/debug.csv for testing
khaeru Aug 12, 2024
13abc9a
Update transport/R12/mode-share/default.csv for #213
r-aneeque Aug 13, 2024
c62d445
Update transport/R12/ldv-age.csv for #213
r-aneeque Aug 14, 2024
c986459
Update base scenarios for transport SSP workflow
khaeru Aug 16, 2024
2312510
Handle non-int age in sales_fraction_annual()
khaeru Aug 19, 2024
b2861c6
Refer to exogenous file data using keys
khaeru Aug 20, 2024
023840c
Add SDMX metadata for transport inputs
khaeru Aug 20, 2024
2ea8989
Further update tests for #213
khaeru Aug 20, 2024
9b07d8f
Update docs for #213
khaeru Aug 2, 2024
a9b3775
Update transport/R12/ldv-t-share.csv for #213
r-aneeque Aug 20, 2024
144a801
Handle full dimensionality of LDV data
khaeru Aug 21, 2024
f85a429
Divide LDV CAP_NEW by period duration
khaeru Aug 21, 2024
cc29404
Omit t=ICE_conv from bound_new_capacity_up
khaeru Aug 21, 2024
9e0aedc
Omit LDV technologies from .non_ldv.bound_activity_lo()
khaeru Aug 21, 2024
d6e043a
Add iis=1 to default MESSAGEix-Transport solve
khaeru Aug 21, 2024
7b65a3c
Add .transport.files.lifetime_ldv and CSV files
khaeru Aug 23, 2024
3659201
Handle LDV technical_lifetime from files
khaeru Aug 23, 2024
059c876
Remove .transport.Config.ldv_lifetime
khaeru Aug 23, 2024
18f58b0
Correct docstring of broadcast_y_yv_ya
khaeru Aug 23, 2024
967bfb4
Update transport/T12/lifetime-ldv.csv for #213
r-aneeque Aug 25, 2024
9c884f4
Update R14/load-factor-ldv.csv
khaeru Aug 23, 2024
d1f3b19
Quiet logging in .exo_data.prepare_computer()
khaeru Aug 23, 2024
1b79b33
Update test_get_ldv_data()
khaeru Aug 23, 2024
5ef953c
Xfail two transport tests for #213
khaeru Aug 23, 2024
d6608df
Add .transport.operator.maybe_select
khaeru Aug 26, 2024
aa7cf98
Add ExoDatasource.key, .get_keys()
khaeru Aug 26, 2024
d9d9f8e
Adjust existing ExoDataSource classes, usage for #213
khaeru Aug 26, 2024
165815f
Add .transport.data.MaybeAdaptR11Source
khaeru Aug 26, 2024
c0ba35d
Add .transport.ldv.LDV class for data
khaeru Aug 26, 2024
0b19492
Use .ldv.lDV in .ldv.prepare_computer()
khaeru Aug 26, 2024
6615ba7
Add transport/R12/ldv-inv_cost.csv
khaeru Aug 26, 2024
f214746
Allow make_output_path(..., name=Path(...))
khaeru Aug 26, 2024
6af5006
Collect .transport.base outputs in a subdirectory
khaeru Aug 26, 2024
456ada8
Adjust transport/R12/ldv-inv_cost.csv for #213
r-aneeque Aug 26, 2024
f6755c9
Output final-energy.csv, fe-share{,-max,-min}.csv
khaeru Aug 27, 2024
090d3c8
Disable .transport.factor "pdt non-active"
khaeru Aug 27, 2024
7759113
Expand and improve test_get_ldv_data()
khaeru Aug 27, 2024
b79e7a8
Also backfill in .transport.operator.extend_y()
khaeru Aug 27, 2024
2b9931b
Include periods from 1995 in LDV parameter data
khaeru Aug 27, 2024
65de031
Add .transport.operator.broadcast_n()
khaeru Aug 27, 2024
f68bb16
Remove minimum_activity entry for (R12_AFR, gas)
khaeru Aug 29, 2024
ba492e7
Include road freight in .non_ldv.constraint_data()
khaeru Aug 29, 2024
2d872a5
Add stub of .transport.base.share_constraints()
khaeru Aug 29, 2024
e7a831d
Also produce bound_activity_up in .non_ldv.other()
khaeru Aug 29, 2024
6d63405
Add transport input data flow from act-non_ldv.csv
khaeru Aug 29, 2024
2f05c64
Constrain techs via transport/R12/act-non_ldv.csv
r-aneeque Aug 29, 2024
e4b1915
Add SOLVE_CONFIG global in .transport.workflow
khaeru Aug 30, 2024
cb293b4
Update transport/base-scenario-url.json
khaeru Aug 30, 2024
4001cea
Add emission tax in .transport.workflow.generate()
khaeru Aug 30, 2024
859a075
Implement .transport.base.share_constraints()
khaeru Sep 1, 2024
f8e6594
Test format_share_constraints()
khaeru Sep 1, 2024
d6a56c3
Expand description of ICAE_ffv
khaeru Sep 4, 2024
fd1ff62
Add #213 to doc/whatsnew
khaeru Aug 2, 2024
7c50871
Xfail transport…test_report_bare_solved temporarily
khaeru Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/transport.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
GH_TOKEN: ${{ secrets.MESSAGE_DATA_DISPATCH_TOKEN }}
run: |
gh workflow run \
${{ env.target }} --ref fix/transport-workflow \
${{ env.target }} --ref=dev \
--field ref=${{ github.ref }} \
--field sha=${{ github.sha }} \
--field base=${{ env.base }} \
Expand Down
1 change: 1 addition & 0 deletions doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ def setup(app: "sphinx.application.Sphinx") -> None:
"issue": ("https://github.com/iiasa/message-ix-models/issue/%s", "GH #%s"),
"pull": ("https://github.com/iiasa/message-ix-models/pull/%s", "PR #%s"),
"gh-user": ("https://github.com/%s", "@%s"),
"source": ("https://github.com/iiasa/message-ix-models/blob/main/%s", "%s"),
}

# -- Options for sphinx.ext.intersphinx ------------------------------------------------
Expand Down
15 changes: 9 additions & 6 deletions doc/transport/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@ MESSAGEix-Transport

.. warning::

MESSAGEix-Transport is **under development**.
1. MESSAGEix-Transport is **under development**.
For details, see:

- The code and data documented on these pages were recently :doc:`migrated </migrate>` from :mod:`.message_data`.
The text may still contain references to the old location.
- For details, see the `project board <https://github.com/orgs/iiasa/projects/29>`_.
- `Issues and PRs labeled 'transport' <https://github.com/iiasa/message-ix-models/issues?q=label%3Atransport>`_ on the :mod:`.message_ix_models` GitHub repository.
- The `project board <https://github.com/orgs/iiasa/projects/29>`_ (IIASA only).

2. The code and data documented on these pages were recently :doc:`migrated </migrate>` from the private :mod:`.message_data` repository to the public :mod:`message_ix_models` repository.
The text may still contain references to the old location.

:mod:`message_ix_models.model.transport` adds a technology-rich representation of transport to models in the MESSAGEix-GLOBIOM family.
The resulting “model variant” is variously referred to as:
Expand All @@ -19,7 +22,7 @@ The resulting “model variant” is variously referred to as:
MESSAGEix-Transport extends the formulation described by McCollum et al. (2017) :cite:`mccollum-2017` for the older, MESSAGE V framework that predated MESSAGEix.
Some inherited information about the older model is collected at :doc:`old`.

Information about MESSAGEix-Transport, its inputs, configuration, implementation, and output, are organized according to this diagram:
This documentation of MESSAGEix-Transport, its inputs, configuration, implementation, and output, are organized according to this diagram:

.. figure:: https://raw.githubusercontent.com/khaeru/doc/main/image/data-stages.svg

Expand All @@ -34,7 +37,7 @@ Information about MESSAGEix-Transport, its inputs, configuration, implementation
input
output

On this page:
Other topics covered on this page:

.. contents::
:local:
Expand Down
26 changes: 14 additions & 12 deletions doc/transport/input.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ This page describes the structure and format of inputs required for building MES

Both input data and configuration are stored in files under :file:`message_ix_models/data/transport/`.

In most cases, these files are read from a subdirectory like :file:`/data/transport/{nodes}/`, where `nodes` denotes the :mod:`message_ix_models` :doc:`node code list </pkg-data/node>`—for instance, "R12"—for which MESSAGEix-Transport will be built.
This value is retrieved from the :attr:`Context.regions <.model.Config.regions>` setting.
In most cases, these files are read from a subdirectory like :file:`/data/transport/{nodes}/`, where `nodes` denotes the :mod:`message_ix_models` :doc:`node code list </pkg-data/node>`—for instance, "R12"—for which MESSAGEix-Transport will be built.
This value is retrieved from the :attr:`.model.Config.regions` setting on a :class:`.Context` object.

- If the file data or configuration settings have a node (:math:`n`) dimension, the file **must** be placed in such a subdirectory.
- If the file data or configuration settings have a node (|n|) dimension, the file **must** be placed in such a subdirectory.
Data for one node list is not usable for base models using a different node list.
- For other data, a node list–specific file **may** be used.
If none exists, the file of the same name in :file:`/data/transport/` is used as a default.
For example, :file:`/data/transport/R12/set.yaml` is used if it exists; if not, then :file:`/data/transport/set.yaml` is used.
If none exists, the file of the same name in :file:`/data/transport/` is used as a default.
For example, :file:`/data/transport/R12/set.yaml` is used if it exists; if not, then :file:`/data/transport/set.yaml` is used.

.. _transport-config:

Expand All @@ -27,8 +27,8 @@ General (:file:`config.yaml`, required)
---------------------------------------

The contents of this configuration file exactly map to the attributes of the class :class:`transport.Config <.transport.config.Config>`.
The class stores all the settings understood by the code for building, solving, and reporting MESSAGEix-Transport, including their default values.
(There is no common default like :file:`/data/transport/config.yaml`.)
The class stores all the settings understood by the code for building, solving, and reporting MESSAGEix-Transport.
The class also defines the default values for each setting (there is no file :file:`/data/transport/config.yaml` containing defaults.)
It also has methods for reading the configuration from file; see the detailed description of :meth:`.Config.from_context`.

The following is the configuration file for a base model with R12 nodes:
Expand All @@ -41,14 +41,14 @@ Technology code list (:file:`technology.yaml`)
This file gives the code list for the MESSAGE ``technology`` concept/set/dimension.
Some annotations (``iea-eweb-flow``, ``input``, ``report``) and the :attr:`~sdmx.model.common.Code.child` hierarchy give information about technologies' grouping according to transport modes.

→ View :source:`data/transport/technology.yaml` on GitHub
→ View :source:`message_ix_models/data/transport/technology.yaml` on GitHub

Code lists for other MESSAGE sets (:file:`set.yaml`)
----------------------------------------------------

This file gives code lists for other MESSAGE concepts/sets/dimensions.

→ View :source:`data/transport/set.yaml` on GitHub
→ View :source:`message_ix_models/data/transport/set.yaml` on GitHub

.. _transport-data-files:

Expand All @@ -65,8 +65,10 @@ Through :func:`.transport.build.main` (ultimately, :func:`.transport.build.add_e
The file has column headers "node", "technology", and "value".

Not all files are currently or always used in model-building computations.
Some submodules of :mod:`~message_ix_models.model.transport` use additional data files via other mechanisms.
Most of the files have a header comment including a precise description of the quantity, source of the data, and units of measurement; in some cases extended information is below (where a header comment would be too long).
Some submodules of :mod:`~.model.transport` use additional data files via other mechanisms.
Most of the files have a header comment including a precise description of the quantity, source of the data, and units of measurement.
In some cases extended information is below (where a header comment would be too long).
The :program:`git` history of files, or the GitHub "blame" view can also be used to inspect the edit history of each file, line by line.

:file:`ldv-activity.csv` → ``ldv activity:n:exo``
-------------------------------------------------
Expand Down Expand Up @@ -223,7 +225,7 @@ Other files
Other data sources
==================

:mod:`~message_ix_models.model.transport` makes use of the :mod:`message_ix_models.tools.exo_data` mechanism to retrieve data from common (not transport-specific) sources.
:mod:`~.model.transport` makes use of the :mod:`.tools.exo_data` mechanism to retrieve data from common (not transport-specific) sources.
:class:`.DataSourceConfig`, :attr:`.transport.Config.ssp`, and other settings determine which sources and quantities are used.

These include:
Expand Down
2 changes: 1 addition & 1 deletion doc/transport/old.rst
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ The tables in this sheet have been preserved as the following files:
- “Vehicle class splits” → ldv_class.csv.
- “Regional cost multipliers” → config.yaml keys ``factor / cost / ldv 2010``, ``ldv cost catch-up year``.
- “Annual driving distances by consumer type” → config.yaml keys ``ldv activity``, ``factor / activity / ldv``.
- “Vehicle lifetimes by consumer type” → config.yaml key ``ldv lifetime``.
- “Vehicle lifetimes by consumer type” → :file:`lifetime-ldv.csv`.
- “Suburbanization rates” → suburb_area_share.csv.

.. admonition:: PNK
Expand Down
5 changes: 4 additions & 1 deletion doc/whatsnew.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ Next release
============

- Fix and update :doc:`/api/tools-costs` (:pull:`219`)
- Fix naming of GDP and population columns in SSP data aggregation (:pull:`219`).

- Fix naming of GDP and population columns in SSP data aggregation.

- Update :doc:`/transport/index` (:pull:`213`).

v2024.8.6
=========
Expand Down
Loading
Loading