Namespaces (#17)

Provide new high-level functions:

- `concatenate_functions_tree`
- `create_input_structure_tree`

allowing for nested dicts as tree representations. The standard case is nested with `qualified_names`, which `"__".join()` the paths. Qualified names need to be used as function arguments in case a function from a different module is needed.


---------

Co-authored-by: Hans-Martin von Gaudecker <[email protected]>
Co-authored-by: Marvin Immesberger <[email protected]>

Author: 3 people

.github/workflows/main.yml

@@ -1,67 +1,53 @@
+---
 name: main
 on:
   push:
-    branches:
-    - main
+    branches: [main]
   pull_request:
     branches:
-    - '*'
-
+      - '*'
 # Automatically cancel a previous run.
 concurrency:
   group: ${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
-
 jobs:
-
   run-tests:
-
     name: Run tests for ${{ matrix.os }} on ${{ matrix.python-version }}
     runs-on: ${{ matrix.os }}
-
     strategy:
       fail-fast: false
       matrix:
-        os: ['ubuntu-latest', 'macos-latest', 'windows-latest']
+        os: [ubuntu-latest, macos-latest, windows-latest]
         python-version: ['3.9', '3.10', '3.11']
-
     steps:
       - uses: actions/checkout@v2
       - uses: conda-incubator/setup-miniconda@v2
         with:
           auto-update-conda: true
           python-version: ${{ matrix.python-version }}
-
       - name: Install core dependencies.
         shell: bash -l {0}
         run: conda install -c conda-forge tox-conda
-
       - name: Run pytest.
         shell: bash -l {0}
         run: tox -e pytest -- -m "not slow" --cov-report=xml --cov=./
-
       - name: Upload coverage report.
         if: runner.os == 'Linux' && matrix.python-version == '3.9'
         uses: codecov/codecov-action@v1
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
-
   docs:
-
     name: Run documentation.
     runs-on: ubuntu-latest
-
     steps:
       - uses: actions/checkout@v2
       - uses: conda-incubator/setup-miniconda@v2
         with:
           auto-update-conda: true
           python-version: 3.9
-
       - name: Install core dependencies.
         shell: bash -l {0}
         run: conda install -c conda-forge tox-conda
-
       - name: Build docs
         shell: bash -l {0}
         run: tox -e sphinx

.github/workflows/publish-to-pypi.yml

@@ -1,34 +1,31 @@
+---
 name: PyPI
-
 on: push
-
 jobs:
   build-n-publish:
     name: Build and publish Python 🐍 distributions 📦 to PyPI
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
-
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v1
-      with:
-        python-version: 3.8
-
-    - name: Install pypa/build
-      run: >-
-        python -m
-        pip install
-        build
-        --user
-    - name: Build a binary wheel and a source tarball
-      run: >-
-        python -m
-        build
-        --sdist
-        --wheel
-        --outdir dist/
-    - name: Publish distribution 📦 to PyPI
-      if: startsWith(github.ref, 'refs/tags')
-      uses: pypa/gh-action-pypi-publish@master
-      with:
-        password: ${{ secrets.PYPI_API_TOKEN }}
+      - uses: actions/checkout@master
+      - name: Set up Python 3.8
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.8
+      - name: Install pypa/build
+        run: >-
+          python -m
+          pip install
+          build
+          --user
+      - name: Build a binary wheel and a source tarball
+        run: >-
+          python -m
+          build
+          --sdist
+          --wheel
+          --outdir dist/
+      - name: Publish distribution 📦 to PyPI
+        if: startsWith(github.ref, 'refs/tags')
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -84,6 +84,9 @@ ipython_config.py
 # VS Code
 .vscode
 
+# IntelliJ
+.idea/
+
 # pyenv
 .python-version
 

.pre-commit-config.yaml

@@ -1,80 +1,51 @@
+---
 repos:
   - repo: meta
     hooks:
       - id: check-hooks-apply
       - id: check-useless-excludes
-        # - id: identity  # Prints all files passed to pre-commits. Debugging.
+      # - id: identity  # Prints all files passed to pre-commits. Debugging.
+  - repo: https://github.com/lyz-code/yamlfix
+    rev: 1.17.0
+    hooks:
+      - id: yamlfix
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v5.0.0
     hooks:
       - id: check-added-large-files
-        args:
-          - --maxkb=1300
+        args: [--maxkb=10000]
       - id: check-case-conflict
       - id: check-merge-conflict
       - id: check-vcs-permalinks
+      - id: check-yaml
       - id: check-toml
       - id: debug-statements
       - id: end-of-file-fixer
       - id: fix-byte-order-marker
-        types:
-          - text
+        types: [text]
       - id: forbid-submodules
       - id: mixed-line-ending
-        args:
-          - --fix=lf
+        args: [--fix=lf]
         description: Forces to replace line ending by the UNIX 'lf' character.
-      - id: name-tests-test
-        args:
-          - --pytest-test-first
       - id: no-commit-to-branch
-        args:
-          - --branch
-          - main
+        args: [--branch, main]
       - id: trailing-whitespace
       - id: check-ast
       - id: check-docstring-first
-  - repo: https://github.com/psf/black
-    rev: 23.3.0
-    hooks:
-      - id: black
-        language_version: python3.10
-  - repo: https://github.com/asottile/blacken-docs
-    rev: 1.13.0
+  - repo: https://github.com/adrienverge/yamllint.git
+    rev: v1.35.1
     hooks:
-      - id: blacken-docs
-  - repo: https://github.com/PyCQA/docformatter
-    rev: v1.5.1
-    hooks:
-      - id: docformatter
-        args:
-          - --in-place
-          - --wrap-summaries
-          - '88'
-          - --wrap-descriptions
-          - '88'
-          - --blank
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.0.261
+      - id: yamllint
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.7.2
     hooks:
+      # Run the linter.
       - id: ruff
-  - repo: https://github.com/nbQA-dev/nbQA
-    rev: 1.7.0
-    hooks:
-      - id: nbqa-black
-      - id: nbqa-ruff
-  - repo: https://github.com/asottile/setup-cfg-fmt
-    rev: v2.2.0
-    hooks:
-      - id: setup-cfg-fmt
-  - repo: https://github.com/mgedmin/check-manifest
-    rev: '0.49'
-    hooks:
-      - id: check-manifest
+        types_or: [python, jupyter]
         args:
-          - --no-build-isolation
-        additional_dependencies:
-          - setuptools-scm
-          - toml
+          - --fix
+      # Run the formatter.
+      - id: ruff-format
+        types_or: [python, jupyter]
 ci:
   autoupdate_schedule: monthly

.readthedocs.yml

@@ -1,10 +1,8 @@
+---
 version: 2
-
 build:
   image: latest
-
 python:
   version: 3.8
-
 conda:
   environment: docs/rtd_environment.yml

codecov.yml

@@ -1,19 +1,16 @@
+---
 codecov:
   notify:
-    require_ci_to_pass: yes
-
+    require_ci_to_pass: true
 coverage:
   precision: 2
   round: down
-  range: "50...100"
+  range: 50...100
   status:
     patch:
       default:
         target: 80%
     project:
       default:
         target: 80%
-
-ignore:
-  - ".tox/**/*"
-  - "setup.py"
+ignore: [.tox/**/*, setup.py]

docs/rtd_environment.yml

@@ -1,7 +1,5 @@
-channels:
-  - conda-forge
-  - nodefaults
-
+---
+channels: [conda-forge, nodefaults]
 dependencies:
   - python=3.9
   - pip
@@ -17,6 +15,4 @@ dependencies:
   - sphinx-copybutton
   - sphinx-panels
   - pydata-sphinx-theme>=0.3.0
-
-  - pip:
-      - ../
+  - pip: [../]

docs/source/examples.rst

@@ -83,6 +83,91 @@ outputs. And we would like to have them as a dictionary. We can do this as follo
     {"h": 0.6, "f": 5, "g": 3.0}
 
 
+Functions from different namespaces
+-----------------------------------
+
+In large projects, function names can become lengthy when they share the same namespace.
+Using dags, we can concatenate functions from different namespaces.
+
+Suppose we define the following function in the module `linear_functions.py`:
+
+.. code-block:: python
+
+    def f(x):
+        return 0.5 * x
+
+In another module, called `parabolic_functions.py`, we define two more functions. Note,
+that there is a function `f` in this module as well.
+
+.. code-block:: python
+
+    def f(x):
+        return x**2
+
+    def h(f, linear_functions__f):
+        return (f + linear_functions__f) ** 2
+
+The function `h` takes two inputs:
+- `f` from `parabolic_functions.py`, referenced directly as f within the current
+namespace.
+- `f` from `linear_functions.py`, referenced using its namespace with a double
+underscore separator (`linear_functions__f`).
+
+Using `concatenate_functions_tree`, we are able to combine the functions from both
+modules.
+
+First, we need to define the functions tree, which maps functions to their namespace.
+The functions tree can be nested to an arbitrary depth.
+
+.. code-block:: python
+    from linear_functions import f as linear_functions__f
+    from parabolic_functions import f as parabolic_functions__f
+    from parabolic_functions import h as parabolic_functions__h
+
+    # Define functions tree
+    functions = {
+        "linear_functions": {"f": linear_functions__f},
+        "parabolic_functions": {
+            "f": parabolic_functions__f,
+            "h": parabolic_functions__h
+        },
+    }
+
+Next, we define the input structure, which maps the parameters of the functions to their
+namespace. The input structure can also be created via the
+`create_input_structure_tree` function.
+
+.. code-block:: python
+    # Define input structure
+    input_structure = {
+        "linear_functions": {"x": None},
+        "parabolic_functions": {"x": None},
+    }
+
+
+Finally, we combine the functions using `concatenate_functions_tree`.
+
+.. code-block:: python
+    # Get combined function
+    combined = concatenate_functions_tree(
+        functions,
+        input_structure=input_structure,
+        targets={"parabolic_functions": {"h": None}},
+    )
+
+    # Call combined function
+    combined(inputs={
+        "linear_functions": {"x": 2},
+        "parabolic_functions": {"x": 1},
+    })
+
+.. code-block:: python
+
+    {"h": 4.0}
+
+Importantly, dags does not allow for branches with trailing underscores in the
+definition of the functions tree.
+
 Renaming the output of a function
 ---------------------------------