topic channel output breaks storeDir

[CormacKinsella]

Bug report

Expected behavior and actual behavior

Expected

• including a storeDir directive in a process will catch only file outputs. I expected it to ignore topic channels since they are collected elsewhere.

Actual

• storeDir doesn't ignore topic channels and fails to store any files:

Steps to reproduce the problem

• Run script below - no files stored, warning and error thrown
• Comment out the topic channel line - works fine

nextflow.preview.topic = true

process FOO {

	storeDir 'output'

	output:
	path "output.txt"
	tuple val(task.process), val('grep'), eval('grep --version | head -n 1'), topic: versions

	script:
	"""
	touch output.txt
	"""
}

workflow {
	FOO()
}

Program output

WARN: [FOO] StoreDir can only be used when using 'file' outputs

ERROR ~ No such file or directory: /home/cormac/sandbox/output/.command.env

Environment

• Nextflow version: 24.10.4.5934
• Java version: openjdk version "21.0.6" 2025-01-21
• Operating system: Linux
• Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

[bentsherman]

I think the problem is not the topic channel, but the eval output. storeDir can only be used with val and path outputs.

It's a bit uglier, but you could make it work by converting the eval output to a path output:

process FOO {
    storeDir 'output'

    output:
    path "output.txt"
    tuple val(task.process), val('grep'), path('version.grep.txt'), topic: versions

    script:
    """
    touch output.txt

    grep --version | head -n 1 > version.grep.txt
    """
}

[mahesh-panchal]

There's still a bug there it looks like. If you do that, the process still runs even with the path.

nextflow.preview.topic = true

process FOO {
    storeDir 'output'

    output:
    path "output.txt"
    tuple val(task.process), val('grep'), path('version.grep.txt'), topic: versions

    script:
    """
    touch output.txt

    grep --version | head -n 1 > version.grep.txt
    """
}
workflow {
	FOO()
}

terminal:

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [gigantic_torricelli] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[17/de4e61] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [tender_snyder] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[d1/23d825] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf -resume

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [chaotic_bohr] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[fd/1beef3] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

It creates a new process each time. It's not using the stored version.

It's the same outputting a val.

nextflow.preview.topic = true

process FOO {

	storeDir 'output'

	output:
	path "output.txt"
	tuple val(task.process), val('grep'), val(version_text), topic: versions

	script:
        def version = 'grep --version'.execute() | 'head -n 1'.execute()
        version.waitFor()
        version_text = version.text
	"""
	touch output.txt
	"""
}

workflow {
	FOO()
}

terminal:

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main.nf` [voluminous_pare] DSL2 - revision: ecf4603cee

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[14/a3c426] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main.nf` [lethal_mclean] DSL2 - revision: ecf4603cee

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[d1/1af4e7] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

[CormacKinsella]

Thanks for the reply Ben.

When running the workaround above, while it does correctly produce the output, it still breaks storeDir - the process is neither stored nor cached and repeats on a rerun. Something about the tuple?

Also, in the workaround provided the version report ends up in the storeDir directory, which in the use case of storing genome indexes alongside input FASTA is not ideal, if we're also trying to produce a unified report of the software versions used by a pipeline. This would typically need to be stored elsewhere, e.g. results/package_versions

Applying a map operator to the topic channel itself only reports the path to the version.grep.txt file, which while usable is not desirable.

If storeDir could match specific output patterns, like the old publishDir option.

Edit: @mahesh-panchal beat me to it :)