Skip to content

Instructions

Jump to bottom
Annette Osprey edited this page Mar 30, 2023 · 5 revisions

Instructions for running log-analysis code

Either add to the runs in an existing directory (e.g. N1280) or make a new directory for a new set of runs. These instructions assume you are adding a new Archer2 suite to the N1280 set.

Process the log files

  1. Gather the cylc-run logs.

    • Copy all the files from cylc-run/log*. These should be copied from the HPC, as puma may not have a full set of files.
    • These should be archived to Jasmin. Most of the previous runs are here: /gws/nopw/j04/hrcm/aosprey/logs.
    • They will also need to be on untarred on the machine where you wish to run the analysis. The script scripts/log-files/untar_logs will extract all tar.gz files in the current directory.

  2. Create a processing script.

    • In the N1280/process_logs (or equivalent) directory, choose an existing run to use as a template, e.g. process_u-cd936.sh.
    • Edit the variables at the top of the file with the relevant paths.
    • By default this just extracts times for the atmos task.

  3. Run the script.

    • Run the script from the command-line.
    • This produces the data to be analysed under files/processed/.

Note that there are some example for processing the transfer task, but these only work with rsync transfers. New scripts will need to be written for processing gridftp transfer logs. In particular, we need to extract the size of the data transferred, and the time taken. It is also not clear what happens when a transfer is interrupted. Does the transfer re-copy everything or just the remaining files?

Run the performance analysis

  1. Add suite details to the spreadsheet files/suite-info.csv.

    • Most of the information can be gathered from looking at the Rose suite setup.
    • The start and end times can be derived from looking at the job logs. In particular the start time is taken to be the time the first model job is submitted (not including any false starts). The end time should be when the last post-processing task completes, for running suites it's just the time of the latest log files.

  2. Add new suite to the analysis scripts in analyse_data/.

    • See for details of each of the analysis scripts.
    • Ensure the analysis functions are in the environment. export PYTHONPATH=/path-to-dir/log-analysis/scripts/analysis:$PYTHONPATH
    • Add the suite id to the list of archer2 suites at the top of the file.
    • Make any other changes as necessary (e.g. adding a colour for plotting), and run the scripts.

Clone this wiki locally