Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Podman and shifter profiles not working #142

Closed
Qiangong2 opened this issue Apr 5, 2024 · 6 comments
Closed

Podman and shifter profiles not working #142

Qiangong2 opened this issue Apr 5, 2024 · 6 comments
Labels
bug Something isn't working
Milestone
1.1.0

Comments

@Qiangong2
Copy link

Description of the bug

When you try and run an alphafold2 job (and presumably colabfold as well) using the conda profile, it always errors out with:

python3: can't open file '/app/alphafold/run_alphafold.py': [Errno 2] No such file or directory

It's almost like it thinks it's running in a container, when it's not.

Command used and terminal output

/global/common/software/m2789/proteinfold/nextflow run nf-core/proteinfold --input $PSCRATCH/fastas.csv --outdir /pscratch/sd/k/kstine/out3 --mode alphafold2 --alphafold2_db /global/cfs/cdirs/m2789/nogaleslab-new/alphafold --full_dbs false --alphafold2_model_preset multimer --use_gpu true -profile conda
N E X T F L O W  ~  version 23.10.1
Launching `https://github.com/nf-core/proteinfold` [adoring_varahamihira] DSL2 - revision: 22a2ada9c2 [master]


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/proteinfold v1.0.0-g22a2ada
------------------------------------------------------
Core Nextflow options
  revision                        : master
  runName                         : adoring_varahamihira
  launchDir                       : /global/cfs/cdirs/m2789/nogaleslab-new/user-scratch/kstine/alpha
  workDir                         : /global/cfs/cdirs/m2789/nogaleslab-new/user-scratch/kstine/alpha/work
  projectDir                      : /global/homes/k/kstine/.nextflow/assets/nf-core/proteinfold
  userName                        : kstine
  profile                         : conda
  configFiles                     : /global/homes/k/kstine/.nextflow/assets/nf-core/proteinfold/nextflow.config, /global/cfs/cdirs/m2789/nogaleslab-new/user-scratch/kstine/alpha/nextflow.config

Global options
  input                           : /pscratch/sd/k/kstine/fastas.csv
  outdir                          : /pscratch/sd/k/kstine/out3
  use_gpu                         : true

Alphafold2 options
  alphafold2_db                   : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold
  alphafold2_model_preset         : multimer

Alphafold2 DBs and parameters links options
  bfd_path                        : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/bfd/*
  small_bfd_path                  : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/small_bfd/*
  alphafold2_params_path          : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/alphafold_params_*/*
  mgnify_path                     : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/mgnify/*
  pdb70_path                      : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/pdb70/**
  pdb_mmcif_path                  : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/pdb_mmcif/**
  uniclust30_path                 : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/uniclust30/**
  uniref90_path                   : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/uniref90/*
  pdb_seqres_path                 : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/pdb_seqres/*
  uniprot_path                    : /global/cfs/cdirs/m2789/nogaleslab-new/alphafold/uniprot/*

Colabfold DBs and parameters links options
  colabfold_db_path               : null/colabfold_envdb_202108
  uniref30_path                   : null/uniref30_2202
  colabfold_alphafold2_params_path: null/params/alphafold_params_2021-07-14
  colabfold_alphafold2_params_tags: [AlphaFold2-multimer-v1:alphafold_params_colab_2021-10-27, AlphaFold2-multimer-v2:alphafold_params_colab_2022-03-02, AlphaFold2-ptm:alphafold_params_2021-07-14]

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/proteinfold for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/proteinfold/blob/master/CITATIONS.md
------------------------------------------------------
WARN: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  There is a problem with your Conda configuration!

  You will need to set-up the conda-forge and bioconda channels correctly.
  Please refer to https://bioconda.github.io/
  The observed channel order is 
  [conda-forge]
  but the following channel order is required:
  [conda-forge, bioconda, defaults]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
executor >  local (2)
[b0/61ac86] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:INPUT_CHECK:SAMPLESHEET_CHECK (fastas.csv)      [100%] 1 of 1 ✔
[86/9693b8] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1) [  0%] 0 of 1
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:CUSTOM_DUMPSOFTWAREVERSIONS                     -
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:MULTIQC                                         -
ERROR ~ Error executing process > 'NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1)'

executor >  local (2)
[b0/61ac86] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:INPUT_CHECK:SAMPLESHEET_CHECK (fastas.csv)      [100%] 1 of 1 ✔
[86/9693b8] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1) [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:CUSTOM_DUMPSOFTWAREVERSIONS                     -
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:MULTIQC                                         -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/proteinfold] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1)'

executor >  local (2)
[b0/61ac86] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:INPUT_CHECK:SAMPLESHEET_CHECK (fastas.csv)      [100%] 1 of 1 ✔
[86/9693b8] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1) [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:CUSTOM_DUMPSOFTWAREVERSIONS                     -
[-        ] process > NFCORE_PROTEINFOLD:ALPHAFOLD2:MULTIQC                                         -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/proteinfold] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1)'

Caused by:
  Process `NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 (Periphilin4TASORMPP8complex_T1)` terminated with an error exit status (2)

Command executed:

  if [ -f pdb_seqres/pdb_seqres.txt ]
      then sed -i "/^\w*0/d" pdb_seqres/pdb_seqres.txt
  fi
  if [ -d params/alphafold_params_* ]; then ln -r -s params/alphafold_params_*/* params/; fi
  python3 /app/alphafold/run_alphafold.py         --fasta_paths=Periphilin4TASORMPP8complex.fasta         --model_preset=multimer --pdb_seqres_database_path=./pdb_seqres/pdb_seqres.txt --uniprot_database_path=./uniprot/uniprot.fasta          --db_preset=reduced_dbs --small_bfd_database_path=./small_bfd/bfd-first_non_consensus_sequences.fasta         --output_dir=$PWD         --data_dir=$PWD         --uniref90_database_path=./uniref90/uniref90.fasta         --mgnify_database_path=./mgnify/mgy_clusters_2018_12.fa         --template_mmcif_dir=./pdb_mmcif/mmcif_files         --obsolete_pdbs_path=./pdb_mmcif/obsolete.dat         --random_seed=53343         --use_gpu_relax=true --max_template_date 2020-05-14
  
  cp "Periphilin4TASORMPP8complex"/ranked_0.pdb ./"Periphilin4TASORMPP8complex".alphafold.pdb
  cd "Periphilin4TASORMPP8complex"
  awk '{print $6"\t"$11}' ranked_0.pdb | uniq > ranked_0_plddt.tsv
  for i in 1 2 3 4
      do awk '{print $6"\t"$11}' ranked_$i.pdb | uniq | awk '{print $2}' > ranked_"$i"_plddt.tsv
  done
  paste ranked_0_plddt.tsv ranked_1_plddt.tsv ranked_2_plddt.tsv ranked_3_plddt.tsv ranked_4_plddt.tsv > plddt.tsv
  echo -e Positions"\t"rank_0"\t"rank_1"\t"rank_2"\t"rank_3"\t"rank_4 > header.tsv
  cat header.tsv plddt.tsv > ../"Periphilin4TASORMPP8complex"_plddt_mqc.tsv
  cd ..
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2":
      python: $(python3 --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  2

Command output:
  (empty)

Command error:
  python3: can't open file '/app/alphafold/run_alphafold.py': [Errno 2] No such file or directory

Work dir:
  /global/cfs/cdirs/m2789/nogaleslab-new/user-scratch/kstine/alpha/work/86/9693b820526728e1f97d8074b34525

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Relevant files

nextflow.log

System information

Version: 23.10.1.5891
Hardware: HPC
Executor: local & slurm
Container engine: conda
OS: SLES 15
nf-core/proteinfold: current master
@Qiangong2 Qiangong2 added the bug Something isn't working label Apr 5, 2024
@JoseEspinosa JoseEspinosa added wontfix This will not be worked on and removed bug Something isn't working labels Apr 8, 2024
@JoseEspinosa
Copy link
Member

Yes, this is right it can not be run with conda and actually we recently added an error to the modules that can not be run using conda on #138. I strongly encourage you to use Docker or Singularity to run the pipeline.

@Qiangong2
Copy link
Author

@JoseEspinosa That's unfortunate, because the shifter and podman profiles also don't work. It seems to be trying to copy the container image before the work dir is created, so it errors out:

Command error:
  
  2024-04-08T09:19:01 Pulling Image: docker:quay.io/biocontainers/python:3.8.3, status: READY
  4965e8f9078ba50c7148d49dcbc41c1827f21cb74329013deeca366204f0e317
  4965e8f9078ba50c7148d49dcbc41c1827f21cb74329013deeca366204f0e317
  .command.run: line 299: NXF_TASK_WORKDIR=/pscratch/sd/k/kstine/alpha/work/c0/deffb96301b785a285e64201a225d4: No such file or directory

Work dir:
  /pscratch/sd/k/kstine/alpha/work/c0/deffb96301b785a285e64201a225d4

As you can see, the work dir is declared the same as where the container should be extracted and the work dir exists.

Docker and singularity are both not available on our systems. This is why we were hoping to utilize the conda environment instead.

@JoseEspinosa
Copy link
Member

That's indeed bad, we will take a look and try to fix the podman and/or shifter profiles as it should work with them. Thanks for reporting.

@JoseEspinosa JoseEspinosa changed the title Can't open '/app/alphafold/run_alphafold.py' error when running under conda Podman and shifter profiles not working Apr 9, 2024
@JoseEspinosa JoseEspinosa added bug Something isn't working and removed wontfix This will not be worked on labels Apr 9, 2024
@JoseEspinosa JoseEspinosa added this to the 1.1.0 milestone Apr 9, 2024
@JoseEspinosa
Copy link
Member

JoseEspinosa commented Apr 9, 2024 •
edited
Loading

It would be great if we could get more information @Qiangong2
Where are you running the pipeline in an HPC cluster?
Also, could you share the command you use to launch the pipeline and from the directory where the process is running (/pscratch/sd/k/kstine/alpha/work/c0/deffb96301b785a285e64201a225d4) the .command.run, .command.log and .command.err files?
🙏

@JoseEspinosa
Copy link
Member

I have been testing the pipeline with podman and it works on my hands, maybe check if something is misconfigured in your system.

@JoseEspinosa
Copy link
Member

As there has been not more feedback, I close the issue now, feel free to reopen it in case you need any further assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
1.1.0
Development

No branches or pull requests
2 participants
@JoseEspinosa @Qiangong2