What would you like to see added in this software?
The analyze command can result in a core dump if there are issues reading some of the collected files. Unless the file is discarded/excluded manually, the process is unable to proceed. As the analysis is parallelized, it may be difficult to identify what dataset file has issues.
At present, this happens with the OpenNeuro ds006833 dataset:
niquery analyze /mnt/data/openneuro/dataset_files /mnt/data/openneuro/dataset_features --suffix bold -f
2025-12-21 16:36:42 - INFO - Script called with arguments:
in_dirname: /mnt/openneuro/dataset_files
out_dirname: /mnt/openneuro/dataset_features
suffix: ('bold',)
force: True
2025-12-21 16:36:42 - INFO - Characterizing 604 datasets...
Filtering modality files: 91%|█████████ | 548/604 [00:06<00:00, 60.69it/s]
Segmentation fault (core dumped)
(When debugging on PyCharm, the segfault is marked as Process finished with exit code 139 (interrupted by signal 11:SIGSEGV))
The core dump error may suggest an issue with e.g. having insufficient memory, etc. However, it is verified that memory consumption is low when executing the command, and that removing the corresponding TSV file allows the process to proceed.
The source of the issue with the dataset remains to be investigated at this time.
Possibilities:
- Perform a basic check of the files before proceeding with the analysis.
- Add a try/except block in
filter_modality_records to deal with unexpected issues.
- Add a flag to the analyze command to exclude datasets.
Do you have any interest in helping implement the feature?
Yes
Additional information / screenshots
N/A
What would you like to see added in this software?
The
analyzecommand can result in a core dump if there are issues reading some of the collected files. Unless the file is discarded/excluded manually, the process is unable to proceed. As the analysis is parallelized, it may be difficult to identify what dataset file has issues.At present, this happens with the OpenNeuro
ds006833dataset:(When debugging on PyCharm, the segfault is marked as
Process finished with exit code 139 (interrupted by signal 11:SIGSEGV))The core dump error may suggest an issue with e.g. having insufficient memory, etc. However, it is verified that memory consumption is low when executing the command, and that removing the corresponding TSV file allows the process to proceed.
The source of the issue with the dataset remains to be investigated at this time.
Possibilities:
filter_modality_recordsto deal with unexpected issues.Do you have any interest in helping implement the feature?
Yes
Additional information / screenshots
N/A