Automated Processing

Group: Leah Morabito, Neal Jackson, James Petley, Frits Sweijen

Main aim: To advance the automated post-processing of LoTSS at high res

---

Overall updates

• The lotss-hba-survey repository we were using has been moved under the organisation; this was accompished after Martin accepted a pull request from Leah's fork and then the organisation forked from Martin's repo. A team was set up under the organisation to provide access. This is read-only for everyone else.
• There were some minor issues to firefight this week with things like macaroons, but these are now resolved.

Automated processing of calibrators

This is ongoing and indivudals are at different states in the process. We use a script that runs in a screen session to do automated staging, downloading, untarring, processing, and uploads solutions to back to a central location. This week we had some issues with staging but still managed to get things running again at Manchester, this time with only the cluster-specific modifications required, which means the processing should be the same in each location.

Automated processing of fields

• Two fields are downloaded to azimuth, but there were memory issues running setup.cwl from VLBI-cwl, related to the Ateamclipper.py script. Tried testing the new setup.cwl with the DP3 ateamclip step but then around 40 subbands failed for an as of yet undiagnosed reason. Transferred a dataset to cosma (Durham) to tset setup.cwl there as have not had memory issues before. That is still running at time of writing (9am on 16 Feb).
• Lots of book-keeping updates to the monitor_lbfields.py script. These include:
  • adding slurm scripts for the different 'tasks' (steps) for running the VLBI-cwl sub-workflows
  • testing the new syntax of create_ms_list.py to generate the mslist.json files
  • updating functions for checking solutions and moving some to calibrator_utils.py
  • selecting a calibrator to use when there is more than one, based on (in order of importance):
    • number of good (unflagged) international stations
    • least amount of flagging in faraday table
    • least amount of flagging in bandpass
  • still in progress this morning: need to update file structure to include obsid instead of just field name since some fields have more than one observation

Next steps

• Finish testing setup.cwl and start testing next task
• incorporate the necessary 'widefield' modules in -- high priority is writing a script to run a light version of ddf-pipeline which re-generates the required solutions for the vlbi processing
• step that applies ddf-pipeline solutions (?)
• other book-keeping steps to glue the different tasks together

---

James Notes

I started by getting the latest version of monitor_lbcals.py running on the dataweb login of cosma.

After some teething problems the system was working to the best of the LTA staging capabilities.

I implemented the new create_ms_list.py syntax which looks something like create_ms_list.py LINC calibrator <DATA_PATH> and this seemed to work well.

I also tested Martin's calibrator solutions checks which Leah had written a wraparound function for. This worked well and can go straight into the monitor_lbfields.py script.

I have created several of the job submission scripts which can be found in the slurm directory. They have been created to work with the new version of the create_ms_list.py script but will probably need some editing for specific inputs. The bookkeeping tasks are not yet defined and will decide where a lot of files and catalogues can be found.

Current statistics:

---