Merge paraphase output #549

rannick · 2025-02-07T16:21:30Z

change vcf headers, merge paraphase vcf output by sample, then by case.
Goal: help storage and preprocessing for Scout visualisation

PR checklist

…erging paraphase json

github-actions · 2025-02-07T16:22:50Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 85399d3

+| ✅ 181 tests passed       |+
#| ❔  23 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

nextflow_config - Config variable not found: validation.help.beforeText
nextflow_config - Config variable not found: validation.help.afterText
nextflow_config - Config variable not found: validation.summary.beforeText
nextflow_config - Config variable not found: validation.summary.afterText
pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0

❔ Tests ignored:

files_exist - File is ignored: CITATIONS.md
files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-nallo_logo_light.png
files_exist - File is ignored: docs/README.md
files_exist - File is ignored: docs/images/nf-core-nallo_logo_light.png
files_exist - File is ignored: docs/images/nf-core-nallo_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: conf/igenomes_ignored.config
files_exist - File is ignored: conf/test_full.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: assets/nf-core-nallo_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-nallo_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-nallo_logo_dark.png
files_unchanged - File ignored due to lint config: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/test.config
files_exist - File found: docs/output.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-nallo_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowNallo.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 0.5.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.skip_qc= false
nextflow_config - Config default value correct: params.skip_snv_calling= false
nextflow_config - Config default value correct: params.skip_genome_assembly= false
nextflow_config - Config default value correct: params.skip_alignment= false
nextflow_config - Config default value correct: params.skip_methylation_pileups= false
nextflow_config - Config default value correct: params.skip_repeat_calling= false
nextflow_config - Config default value correct: params.skip_repeat_annotation= false
nextflow_config - Config default value correct: params.skip_phasing= false
nextflow_config - Config default value correct: params.skip_snv_annotation= false
nextflow_config - Config default value correct: params.skip_sv_calling= false
nextflow_config - Config default value correct: params.skip_sv_annotation= false
nextflow_config - Config default value correct: params.skip_cnv_calling= false
nextflow_config - Config default value correct: params.skip_call_paralogs= false
nextflow_config - Config default value correct: params.skip_rank_variants= false
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/genomic-medicine-sweden/test-datasets/630ff7c9bd62f3446070ebd685df380a53926366/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.preset= revio
nextflow_config - Config default value correct: params.snv_caller= deepvariant
nextflow_config - Config default value correct: params.sv_caller= severus
nextflow_config - Config default value correct: params.phaser= longphase
nextflow_config - Config default value correct: params.hifiasm_mode= hifi-only
nextflow_config - Config default value correct: params.hifiasm_preset=
nextflow_config - Config default value correct: params.alignment_processes= 8
nextflow_config - Config default value correct: params.snv_calling_processes= 13
nextflow_config - Config default value correct: params.vep_cache_version= 110
nextflow_config - Config default value correct: params.filter_snvs_expression=
nextflow_config - Config default value correct: params.filter_svs_expression=
nextflow_config - Config default value correct: params.deepvariant_model_type= PACBIO
nextflow_config - Config default value correct: params.minimap2_read_mapping_preset= map-hifi
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: ci_master.yml
actions_schema_validation - Workflow validation passed: keep_a_changelog.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: build-docs.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - HIFIASM found in conf/base.config and Nextflow scripts.
base_config - SAMTOOLS_MERGE found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.1.2

Run details

nf-core/tools version 3.1.2
Run at 2025-02-12 13:33:12

Co-authored-by: Felix Lenner <[email protected]>

…icine-sweden/nallo into merge_paraphase_vcfs

fellen31

Nice! Needs some publishDir changes I think.

There's also a Paralogous genes section in the output docs. Could you add these new files to that table, and perhaps a note on which files has post-processed after paraphase?

fellen31 · 2025-02-12T09:34:16Z

bin/merge_json_sample_files_into_family.py

+def merge_json_files(json_files, output_file):
+    """
+    This script merges multiple JSON files into a single JSON file.
+    Each input file's content is stored under a key derived from the first part of its filename (before the first dot).


Another +1 for #504 😄

Absolutely!

tests/samplesheet_multisample_bam.nf.test.snap

tests/samplesheet_multisample_ont_bam.nf.test.snap

CHANGELOG.md

…merged vcf

conf/modules/call_paralogs.config

subworkflows/local/call_paralogs.nf

fellen31

Thanks a lot for the help!

docs/output.md

Co-authored-by: Felix Lenner <[email protected]>

change vcf headers, merge by sample, merge by case, first draft for m…

4dfd924

…erging paraphase json

rannick self-assigned this Feb 7, 2025

rannick added 11 commits February 10, 2025 11:47

add script to merge json files

869b5e5

update changelog

2eb1a32

fixes

b814862

add arg for bcftools query

6fa3ed0

add prefix to CREATE_SAMPLES_FILE

8aa04eb

use arg2

d4ed5bc

modify create_sample_files to add the haplotype in the sample name

79ef3c1

haplotype specific sample names, force single

8b67981

create empty channels to use

a1da33a

update paraphase

031c525

Merge branch 'dev' into merge_paraphase_vcfs

b679649

rannick marked this pull request as ready for review February 11, 2025 11:01

rannick requested a review from a team as a code owner February 11, 2025 11:01

rannick and others added 2 commits February 11, 2025 12:42

updated snapshots

e19550f

update snapshots

2b4bf2a

This comment was marked as resolved.

Sign in to view

rannick and others added 8 commits February 11, 2025 13:49

update snapshots 2

a7785d1

merge

e10942b

Update CHANGELOG.md

a21177c

Co-authored-by: Felix Lenner <[email protected]>

Update modules/local/merge_json.nf

fc8aa50

Co-authored-by: Felix Lenner <[email protected]>

Update modules/local/merge_json.nf

f51fb52

Co-authored-by: Felix Lenner <[email protected]>

Update conf/modules/call_paralogs.config

229d234

Co-authored-by: Felix Lenner <[email protected]>

Update conf/modules/call_paralogs.config

01cdabc

Co-authored-by: Felix Lenner <[email protected]>

Update modules/local/merge_json.nf

7209212

Co-authored-by: Felix Lenner <[email protected]>

This comment was marked as off-topic.

Sign in to view

rannick and others added 2 commits February 11, 2025 14:34

Apply suggestions from code review

9933930

Co-authored-by: Felix Lenner <[email protected]>

case to family and aligned bracket

f368718

rannick and others added 14 commits February 11, 2025 15:00

update script

2ab7ac6

update script

5e499c7

update script

5401f1f

add to software references

03686d0

fix config issue

e37086a

remove snapshots

8af0bda

update snaphshots

061b9c9

exclude vcf and tbi unpublished

6a7a06a

update snaps

8aab0c6

remove snap

cf47dda

add publish dir

283e233

update snap

85f219e

revert change

e872d81

Merge branch 'merge_paraphase_vcfs' of https://github.com/genomic-med…

3045eea

…icine-sweden/nallo into merge_paraphase_vcfs

fellen31 reviewed Feb 12, 2025

View reviewed changes

rannick and others added 7 commits February 12, 2025 11:31

paralogs false default publish, remove publishing for all but family …

98785d8

…merged vcf

use family_id directory

04229d5

prettier

fd32b4b

updated modules added

aff2a3a

update output.md

43aa9f5

publish family json

7148781

update snapshots

6042515

fellen31 reviewed Feb 12, 2025

View reviewed changes

conf/modules/call_paralogs.config Outdated Show resolved Hide resolved

subworkflows/local/call_paralogs.nf Show resolved Hide resolved

rannick and others added 2 commits February 12, 2025 13:35

use sample and family directories, update mapping for family merging

88ff629

update snapshots

fb6abd4

fellen31 approved these changes Feb 12, 2025

View reviewed changes

docs/output.md Outdated Show resolved Hide resolved

rannick and others added 2 commits February 12, 2025 14:28

Update docs/output.md

54b8cc7

Co-authored-by: Felix Lenner <[email protected]>

prettier

85399d3

rannick merged commit 5049054 into dev Feb 12, 2025
23 checks passed

rannick deleted the merge_paraphase_vcfs branch February 12, 2025 13:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge paraphase output #549

Merge paraphase output #549

rannick commented Feb 7, 2025 •

edited

Loading

github-actions bot commented Feb 7, 2025 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

This comment was marked as resolved.

This comment was marked as off-topic.

fellen31 left a comment

fellen31 Feb 12, 2025

rannick Feb 12, 2025

fellen31 left a comment

Merge paraphase output #549

Merge paraphase output #549

Conversation

rannick commented Feb 7, 2025 • edited Loading

PR checklist

github-actions bot commented Feb 7, 2025 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

This comment was marked as resolved.

This comment was marked as off-topic.

fellen31 left a comment

Choose a reason for hiding this comment

fellen31 Feb 12, 2025

Choose a reason for hiding this comment

rannick Feb 12, 2025

Choose a reason for hiding this comment

fellen31 left a comment

Choose a reason for hiding this comment

rannick commented Feb 7, 2025 •

edited

Loading

github-actions bot commented Feb 7, 2025 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️