Snakemake --dag | dot not working

[Eric-Nitschke]

Checklist

• I am using the current main branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

When running

snakemake --dag solve_all_networks | dot -Tsvg > dag.svg

to produce the rule graph for your PyPSA-Earth setup, dot will fail with the following error message (or similar):

Error: <stdin>: syntax error in line 1 near 'Set'

This is caused by other print statements that occur when running the Snakefile. As of right now these are:

• the PuLP-warning described in issue Use new snakemake-minimal version to remove Gurobi Warning #1304.
• Demand data folder: [...] caused by get_load_path_gegis in prepare_demand_profiles.py.
• for PyPSA-Earth versions older than 0.6.0: The namespace pypsa.networkclustering is deprecated […] caused py the outdated PyPSA version.

A somewhat dirty (local) fix for this would be the following:

• find pulp/apis/gurobi_api.py in your environment, identify the try-except block with gurobipy.setParam("_test", 0) and change it to always return true. as it isn't working properly anyway.
• in prepare_demand_profiles.py add snakemake.logging import logger as snake_logger and change logger to snake_logger in get_load_path_gegis.
• find pypsa/networkclustering.py in your environment and remove the warning.

I think the change suggested for prepare_demand_profiles.py could be included in general, as it is not caused by any dependency. The PuLP error will most likely resolve itself, once PyPSA-Earth updates to Snakemake 8 (see #1304). However, it could be nice to write a general fix based on PuLP commit 42f91ab that addresses the issue.

Error Message

Error: <stdin>: syntax error in line 1 near 'Set'

[davide-f]

Hello @Eric-Nitschke :)
Thanks for reporting; indeed this issue will be solved permanently with the update of snakemake to version >=8.

A temporary solution that may be used is:

1. Save the output of the dag into a temporary txt, e.g.
   snakemake --dag solve_all_networks >> out.txt
2. Remove the first four lines from out.txt that lead to that issues and save the file
3. dot out.txt -Tsvg -o dag.svg

The above could also be automated in linux, but would not work in windows

Good that you also identified the point in prepare_demand_profiles.
To simplify, we can drop the logger.info from that location and place it in retrieve_databundle, hopefully leading to the same logging.

Would you like adding a PR where we update to snakemake >= 8 and fix these issues?
In full transparency, as it leads to breaking changes with packages down the line, it may be safer to wait some/few weeks before merging the PR, but we could test the workflow functionality with the update of snakemake