BIDS filter not working

[bilas003]

What happened?

BIDS-filter is not working to filter out the set of files with the latest release of the build. The job ran for 4 days and then timed out . Can anyone help?

What command did you use?

Used a slurm job to run the pipeline,see the attached image.

What version of NiBabies are you using?

nibabies-24.0.0.sif

Relevant log output

Time -out

Add any additional information or context about the problem here.

[mgxd]

Can you share the output log and the contents of the filter file you are using? Also looks like your line break on --derivatives is not properly spaced.

FYI - The latest release is 24.1.0, and we just cut a release candidate (25.0.0rc1) of a new minor series slated to come out later this week.

[bilas003]

BIDS -filter json

{
"bold":{"datatype":"func","suffix":"bold","session":"V02"},
"sbref":{"datatype": "func", "suffix": "sbref","session":"V02"},
"fmap": {"datatype": "fmap","session":"V02"},
"t2w": {"datatype": "anat", "suffix": "T2w" , "run":"2", "session": "V02","SeriesDescription":"anat-t2w","ProtocolName":"anat-t2w"}

}

I corrected the derivatives flag and ran again.

[mgxd]

Were any T2w images found? I don't think --bids-filter-file will filter metadata, only bids entities directly in the filename.

[erikglee]

@mgxd, nibabies uses pybids to find files based on the bids filter file, right? @bilas003 had tried running some independent tests with pybids that involved a filter file with fields that reference the JSON metadata. In her example, it looked like pybids came up with the right files based on the filter settings.

[erikglee]

Hi @bilas003, @mgxd just informed me that another good option to test would be trying to put null (without quotations) for the t2w "acq" entity in the BIDS Filter File. This should be another strategy to avoid the localizer images (since they have "acq-mrsLoc" in their name) during processing, without Nibabies needing to search through the JSON meta-data.

Could you try running another test where you replace the t2w filter file content to be:
"t2w": {"datatype": "anat", "suffix": "T2w" , "run":"2", "session": "V02", "acq": null}

[bilas003]

Hi @mgxd @erikglee

Added the acq tag as null to bids-filter.

The new error received is
ValueError: 'acq' is not a recognized entity. If you're sure you want to impose this constraint, set invalid_filters='allow'.

[mgxd]

You'll have to use the full entity name - acquisition

[bilas003]

The job has been running for 3 days now. The error log shows FileNotFound (see screenshot)

[mgxd]

There are a few moving parts here, can you clarify the following?

1. What does your bids directory tree look like for the subject you are running?
2. What does your filter file look like now?

Additionally, some suggestions:

• --skip-bids-validation should not be used when debugging querying a BIDS dataset
• If you are just starting analysis, try using 25.0.1 as it's the most up to date version
• --participant-label <id> can be used in your command to narrow down processing for a single subject, simplifying debugging
• --age-months is deprecated and shouldn't be used

[bilas003]

Hi @mgxd

Here is the tree structure

filter like is

I followed your suggestions and made changes with latest version of sif , the job fails with below err

[mgxd]

RE: filter - Which file are you trying to isolate? I don't see anything matching your T2w criteria...you are looking for:
anat/sub-<>_ses-V02_run-2_T2w...

RE: the validation failure, I'm not quite sure what ERR_STRING_TOO_LONG is complaining about, perhaps @effigies @oesteban @rwblair have seen this before

[bilas003]

Hi Mattias
For anat scans we are trying to filter out run_2 V02 . I have attached the screenshot with files highlighted.

[mgxd]

--bids-filter is inclusive, so only the subsequent matches will be used. It seems like you are looking to use it to exclude - this behavior is not implemented. Your best bet would be to move the file to somewhere like sourcedata/raw/sub-<>/ses-V02/anat/, where it will be ignored.

FYI - if multiple files are found for t1w/t2w, they are used to form an average.

[erikglee]

Hi Mathias, thanks for the replies! That sounds good, I think we misunderstood your comment about setting "acquisition" to null. We thought you were saying that this could create exclusion like behavior.

Another way that we could use BIDS Filter Files to get the file selection we want is to have the BIDS Filter File use information from the anat JSON sidecars such as "SeriesDescription":"anat-t2w. This is what we were originally trying, and it worked as expected in pybids, but it wasn't giving the expected behavior in Nibabies.

Do you think it would be within scope for Nibabies to accept filter files that use metadata information from within JSON sidecars?

[mgxd]

I tested this, and can confirm our data collection already filters based on JSON sidecars as well as filename entities.

I just added #452 to improve the transparency of which files are being used.

[erikglee]

Thanks! @bilas003 could you download the new 'unstable' version of Nibabies from Docker Hub and share what you are seeing when you use the --verbose flag (with the filter file you were originally using in your Jan 27 message)?

[bilas003]

Hi @mgxd
for the unstable version and filter file including "SeriesDescription":"anat-t2w, the pipeline throws below err

{
"bold":{"datatype":"func","suffix":"bold","session":"V02"},
"sbref":{"datatype": "func", "suffix": "sbref","session":"V02"},
"fmap": {"datatype": "fmap","session":"V02"},
"t2w": {"datatype": "anat", "suffix": "T2w" , "run":"2", "session": "V02","SeriesDescription": "anat-t2w"}
}

[effigies]

I haven't seen that error, but if the dataset is valid BIDS, then just use --skip-bids-validation in nibabies.

[mgxd]

Picked Ross's brain about it earlier - it's a node error, likely from either a really large JSON or TSV.

@bilas003 it would be nice to verify the BIDS directory is actually valid, especially since the problem seems to be with indexing.

[erikglee]

Thanks for the replies. Given that we have accelerometry/motion data stored in TSV files (following https://bids-specification.readthedocs.io/en/stable/modality-specific-files/motion.html), we definitely do have very large (~600MB) TSV files.

@bilas003 is now also trying running processing with --skip-bids-validation following @effigies suggestion. We had previously processed this subject successfully without the filter file, so the formatting should be pretty good, although I think there are still some minor errors detected by the validator (unrelated to any fMRI/T1w/T2w/fmap files).

[mgxd]

has this been resolved?

[erikglee]

We had ran a couple of tests that accidentally had some odd features, so they were inconclusive. We are starting some more tests right now and will hopefully know tomorrow.

[erikglee]

Okay things are working pretty well now! So we are currently using the unstable version of Nibabies (as of a couple of weeks ago) with the skip bids validation flag activated. We have run two tests with filter files. The first test used a filter file like this:

{
    "bold":{"datatype":"func","suffix":"bold","session":"V02"},
    "sbref":{"datatype": "func", "suffix": "sbref","session":"V02"},
    "fmap": {"datatype": "fmap","session":"V02"},
    "t2w": {"datatype": "anat", "suffix": "T2w" , "run":"1", "session": "V02","SeriesDescription": "anat-t2w"}

}

And Nibabies correctly looked through the JSON metadata to find the "SeriesDescription" information that we wanted to use for T2w filtering. The only unexpected behavior that we observed was that Nibabies still grabbed a T1w image even though it wasn't specified in the filter file. Following that we ran this second test with a SeriesDescription that will never occur in practice:

{
    "bold":{"datatype":"func","suffix":"bold","session":"V02"},
    "sbref":{"datatype": "func", "suffix": "sbref","session":"V02"},
    "fmap": {"datatype": "fmap","session":"V02"},
    "t2w": {"datatype": "anat", "suffix": "T2w" , "run":"1", "session": "V02","SeriesDescription": "anat-t2w"},
    "t1w": {"datatype": "anat", "suffix": "T1w" , "session": "V02","SeriesDescription": "SKIPT1w"}

}

And with this test Nibabies avoided the T1w image from the BIDS directory during processing. In both cases the visual report still said that there was 1 Input T1w image (with no reference to the T2w image). But in the second example this was the only trace of T1w involvement that we noticed.

I imagine we may run an additional test or two to look at cases where other images are the target of filtering. But this looks promising so far!