Doc.Singularity.Usecases.UTK.Batchjob

Example Use-Case for Running HPC Batch Jobs with X-SCAPE and Singularity

While it's possible to run X-SCAPE as an interactive job, users might find it more efficient to write job scripts to submit to the scheduler. Multiple jobs can be submitted and executed without having to remain logged in to an interactive shell.

This wiki page demonstrates a simple use-case that can be adapted according to your project's needs.

In this example, we will work with three files: 1) X-SCAPE's user XML file, 2) a Bash script, and 3) a job script.

Modifying X-SCAPE's User XML File

Shown here is the jetscape_user_PP_1910.05481.xml file provided in the config folder of the X-SCAPE installation. Modify the <outputFilename> tag to include a path to where your generated data will be written. Replace your_user_id with your user id. testOutputReplace is a specific string of text we will search for in the subsequent bash script.

<?xml version="1.0"?>

<jetscape>
  
  <nEvents> 1000 </nEvents> 
  <outputFilename>/lustre/haven/user/your_user_id/testOutputReplace</outputFilename>
  <JetScapeWriterHepMC> on </JetScapeWriterHepMC>
  
  <!-- Hard Process -->
  <Hard>
    <PythiaGun>
      <pTHatMin>235</pTHatMin>
      <pTHatMax>1000</pTHatMax>
      <eCM>5020</eCM>
    </PythiaGun>
  </Hard>
  
  <!--Eloss Modules -->
  <Eloss>
    <Matter> 
      <Q0> 1.0 </Q0>
      <in_vac> 1 </in_vac>
      <vir_factor> 0.25 </vir_factor>
      <recoil_on> 0 </recoil_on>
      <broadening_on> 0 </broadening_on>
      <brick_med> 0 </brick_med>
    </Matter>
  </Eloss>
  
  <!-- Jet Hadronization Module -->
  <JetHadronization>
    <name>colorless</name>
  </JetHadronization>
  
</jetscape>

Writing a BASH Script to Call the runJetscape Program

If we run JETSCAPE multiple times and pass in this same user XML file, the generated data will be overwritten because the <outputFilename> text is the same. The following BASH script will replace the text in the <outputFile> tag with the PBS_JOBID of the current batch job. Different jobs will have different job IDs, so the output data will not be overwritten.

Create the following BASH script jetjob.sh in your home directory. You may name the script what you like.

#!/bin/bash

mkdir -p /lustre/haven/user/your_user_id/${PBS_JOBID}

cp ~/jetscape/JETSCAPE/config/publications_config/arXiv_1910.05481/jetscape_user_PP_1910.05481.xml ~/jetscape/JETSCAPE/config/temp${PBS_JOBID}.xml

sed -i "s/testOutputReplace/out${PBS_JOBID}/" ~/jetscape/JETSCAPE/config/temp${PBS_JOBID}.xml

cd ~/jetscape/JETSCAPE/build
./runJetscape ../config/temp${PBS_JOBID}.xml

rm ~/jetscape/JETSCAPE/config/temp${PBS_JOBID}.xml

mv /lustre/haven/user/your_user_id/out${PBS_JOBID}* /lustre/haven/user/your_user_id/${PBS_JOBID}

• Replace the instances of your_user_id with your user id.
• The mkdir command creates a folder named according to the current PBS_JOBID.
• The cp command creates a temporary copy of the user XML file with the PBS_JOBID appended to the filename. It is in this temporary file that the text of the <outputFile> tag will be updated.
• The sed command searches for the specific text "testOutputReplace" and replaces that text with "out${PBS_JOBID} where ${PBS_JOBID} is the job ID of the current job.
• The cd command navigates to the build directory where runJetscape will be called along with the user XML file for this specific job.
• The rm command deletes the temporary user XML file when the job is complete.
• The mv command moves the output file(s) created by this job to the output folder designated for this job.

Writing a Job Script to Call the jetjob BASH Script

Create the following job script jet.job in your home directory. You may name the script what you like.

#PBS -S /bin/bash
#PBS -A ACF-<your_institution_name><your_project_id_number>
#PBS -o /lustre/haven/user/your_user_id
#PBS -e /lustre/haven/user/your_user_id
#PBS -M your_email_address

#PBS -l nodes=1:ppn=4,walltime=01:00:00

singularity exec jet.sif ~/jetjob.sh

• Replace your_institution_name, your_project_id_number, your_user_id, and your_email_address with the data relevant to your account.
• Adjust ppn and walltime according to your needs.

Submit the Job

Execute the qsub command from your home directory.

qsub jet.job