feat: workflow and config templates #29

ameynert · 2025-01-30T23:04:29Z

Populates the config.yml, config_schema.yml, and Snakefile with "hello world" type examples.
Adds a "Development" section to the template repo README.md
Fleshes out the cookecutter README.md with examples of how to document inputs and outputs, and how to configure and run the workflow

ameynert · 2025-01-31T23:18:41Z

README.md

+## Development
+
+Read the [Snakemake Best Practices](https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html) and the [Fulcrum Snakemake](https://www.notion.so/fulcrumgenomics/Snakemake-3d836708c9bc47ca868ee9a09ada7d0d) documentation. The text below is adapted from the latter.
+


Maybe this text should be used to update the Notion page instead? The template is only intended for internal use.

I'd be fine moving most/all of the below text to the Notion page and linking from here. (Just use suggest changes so N/T/C can review.) That way, we're not maintaining it in two locations.

And I find it easier to make edits and suggestions to longform text in Notion or gdocs rather than a PR

msto

This is looking really nice! The new docs and schemas are awesome

README.md

{{cookiecutter.project_slug}}/Snakefile

{{cookiecutter.project_slug}}/config/config.yml

{{cookiecutter.project_slug}}/Snakefile

{{cookiecutter.project_slug}}/workflow/{{cookiecutter.project_slug}}.smk

msto · 2025-02-04T14:05:40Z

{{cookiecutter.project_slug}}/config/config_schema.yml

+properties:
+  experiment:
+    type: string
+    description: Name of the experiment.
+    example: "{{cookiecutter.project_slug}}"
+
+  samples:
+    type: array
+    description: List of samples.
+    example: ["sample1", "sample2"]
+
+  p_value_cutoff:
+    type: number
+    description: P-value cutoff for statistical significance.
+    default: 0.05
+
+required:
+  - experiment
+  - samples


Do you think we should update the workflow to use these config params? Or, choose contrived example params that could be used in the workflow?

msto · 2025-02-05T18:24:41Z

README.md

+
+If there is a single Snakemake workflow, it should be named `Snakefile` and kept at the top level of the repository. If there are multiple workflows, name them according to their function and give them the extension `.smk`.
+
+Workflow files should mainly contain `rules`.  Any additional code should be added to a separate `Python` toolkit, for example to parse the configuration object, handle samplesheet input, or to organize reference data.


Suggested change

Workflow files should mainly contain `rules`. Any additional code should be added to a separate `Python` toolkit, for example to parse the configuration object, handle samplesheet input, or to organize reference data.

Workflow files should mainly contain `rules`. Any additional code should be added to a separate Python toolkit, for example to parse the configuration object, handle samplesheet input, or to organize reference data.

msto · 2025-02-05T18:25:44Z

README.md

+
+The following should be followed:
+
+1. All rules should have descriptive names


Suggested change

1. All rules should have descriptive names

1. All rules should have descriptive names.

msto · 2025-02-05T18:28:55Z

README.md

+The following should be followed:
+
+1. All rules should have descriptive names
+2. All rules should have a short docstring describing what the rule does, and what tool(s) it uses. If there are any custom input or output file formats, describe them, e.g. for a CSV/TSV file list the expected field names. Follow the docstring conventions for Python, e.g.


Suggested change

2. All rules should have a short docstring describing what the rule does, and what tool(s) it uses. If there are any custom input or output file formats, describe them, e.g. for a CSV/TSV file list the expected field names. Follow the docstring conventions for Python, e.g.

2. Each rule should have a short docstring describing its behavior and any tools it uses. The docstring should also describe the required fieldnames or schema of any custom input or output file formats.

Follow Python's docstring conventions, e.g.:

msto · 2025-02-05T18:29:44Z

README.md

+    Inputs:
+        file: TSV file with fields name, count, and source describing counts of ...
+    Params:
+        min_count: Minimum count for output.
+    Output:
+        file: Filtered version of input.file containing only rows with count > params.min_count


Suggested change

Inputs:

file: TSV file with fields name, count, and source describing counts of ...

Params:

min_count: Minimum count for output.

Output:

file: Filtered version of input.file containing only rows with count > params.min_count

Inputs:

file: TSV file with fields "name", "count", and "source" describing counts of ...

Params:

min_count: Minimum count for output.

Output:

file: Filtered version of `input.file` containing only rows where `count > params.min_count`.

msto · 2025-02-05T18:31:41Z

README.md

+1. All rules should have descriptive names
+2. All rules should have a short docstring describing what the rule does, and what tool(s) it uses. If there are any custom input or output file formats, describe them, e.g. for a CSV/TSV file list the expected field names. Follow the docstring conventions for Python, e.g.
+
+```python


Can you indent all code blocks in this section so they are appropriately aligned within their bullet item when rendered to markdown?

msto · 2025-02-05T18:38:57Z

{{cookiecutter.project_slug}}/README.md

+
+## Inputs
+
+TODO: Describe workflow input file formats here. Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.


Suggested change

TODO: Describe workflow input file formats here. Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.

> [!WARNING]

> **After creating a new project, describe workflow input file formats here. **

>

> Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.

msto · 2025-02-05T18:39:49Z

{{cookiecutter.project_slug}}/README.md

+
+## Outputs
+
+TODO: Describe workflow outputs here. Consider using a `tree` output style format to describe the expected output file structure, URLs to third party file format descriptions, and tables as in [Inputs](#inputs) for custom output file formats.


Suggested change

TODO: Describe workflow outputs here. Consider using a `tree` output style format to describe the expected output file structure, URLs to third party file format descriptions, and tables as in [Inputs](#inputs) for custom output file formats.

> [!WARNING]

> **After creating a new project, describe workflow outputs here.**

>

> Consider using a `tree` output style format to describe the expected output file structure, URLs to third party file format descriptions, and tables as in [Inputs](#inputs) for custom output file formats.

msto · 2025-02-05T18:41:28Z

{{cookiecutter.project_slug}}/README.md

+The [workflow configuration schema](config/config_schema.yml) describes the parameters for the workflow.
+To set the parameters for a specific run of the workflow, write a [configuration file](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html) using the schema and the [example config/config.yml](config/config.yml) as a guide.
+
+If you do not specify a workflow file with e.g. `-s myworkflow.smk`, Snakemake will look, in this order, for a file called `Snakefile`, `snakefile`, `workflow/Snakefile`, or `workflow/snakefile`.


How much of the Snakemake basics do we think are necessary to include in this template?

Some of this feels more appropriate as links to the relevant sections of the Snakemake docs, or assumed as prior knowledge for the user

msto · 2025-02-05T18:43:19Z

{{cookiecutter.project_slug}}/Snakefile

+            touch {output.indexes}
+        ) &> {log}
+        """
+


Suggested change

msto · 2025-02-05T18:43:52Z

{{cookiecutter.project_slug}}/Snakefile

+            touch {output.reads}
+        ) &> {log}
+        """
+


Suggested change

two lines between rules per snakefmt

feat: template workflow and config/config_schema

6e5e4d1

ameynert self-assigned this Jan 30, 2025

ameynert mentioned this pull request Jan 30, 2025

WIP: template refresh #24

Draft

7 tasks

ameynert added 2 commits January 30, 2025 15:20

feat: added development section to README

c84ef78

feat: moved snakefile and config up a level, added config schema doc

412660b

ameynert commented Jan 31, 2025

View reviewed changes

ameynert added 3 commits January 31, 2025 15:20

feat: tweak config validation description

10a0725

feat: added input and output description suggestions to README

1c5896c

fix: extra character in code block

f491ad6

ameynert marked this pull request as ready for review January 31, 2025 23:38

ameynert requested review from clintval, tfenne and nh13 as code owners January 31, 2025 23:38

ameynert assigned msto and unassigned ameynert Jan 31, 2025

msto reviewed Feb 4, 2025

View reviewed changes

ameynert added 3 commits February 4, 2025 16:09

feat: expand recommendations

aa04242

fix: typo

8563056

feat: expanded example Snakefile, use docstrings on rules

6d87fa8

msto reviewed Feb 5, 2025

View reviewed changes

msto assigned ameynert and unassigned msto Feb 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: workflow and config templates #29

feat: workflow and config templates #29

ameynert commented Jan 30, 2025 •

edited

Loading

ameynert Jan 31, 2025 •

edited

Loading

msto Feb 5, 2025

msto left a comment

msto Feb 4, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

msto Feb 5, 2025

		## Development

		Read the [Snakemake Best Practices](https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html) and the [Fulcrum Snakemake](https://www.notion.so/fulcrumgenomics/Snakemake-3d836708c9bc47ca868ee9a09ada7d0d) documentation. The text below is adapted from the latter.


		If there is a single Snakemake workflow, it should be named `Snakefile` and kept at the top level of the repository. If there are multiple workflows, name them according to their function and give them the extension `.smk`.

		Workflow files should mainly contain `rules`. Any additional code should be added to a separate `Python` toolkit, for example to parse the configuration object, handle samplesheet input, or to organize reference data.


		The following should be followed:

		1. All rules should have descriptive names

-. All rules should have a short docstring describing what the rule does, and what tool(s) it uses. If there are any custom input or output file formats, describe them, e.g. for a CSV/TSV file list the expected field names. Follow the docstring conventions for Python, e.g.
+. Each rule should have a short docstring describing its behavior and any tools it uses. The docstring should also describe the required fieldnames or schema of any custom input or output file formats.
+Follow Python's docstring conventions, e.g.:


		## Inputs

		TODO: Describe workflow input file formats here. Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.

-TODO: Describe workflow input file formats here. Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.
+> [!WARNING]
+> **After creating a new project, describe workflow input file formats here. **
+>
+> Use the [configuration schema](config/config_schema.yml) for simple input descriptions, e.g. `reference` is "Path to reference genome in FASTA format with .fai and .dict indexes". Include URLs to input descriptions for 3rd party tools.


		## Outputs

		TODO: Describe workflow outputs here. Consider using a `tree` output style format to describe the expected output file structure, URLs to third party file format descriptions, and tables as in [Inputs](#inputs) for custom output file formats.

feat: workflow and config templates #29

Are you sure you want to change the base?

feat: workflow and config templates #29

Conversation

ameynert commented Jan 30, 2025 • edited Loading

ameynert Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameynert commented Jan 30, 2025 •

edited

Loading

ameynert Jan 31, 2025 •

edited

Loading