Pipeline philosophy

The pipeline is built in a modular way, where the desired outcome can be achieved by stringing the different modules together. The modules should be interoperable, but this is still in development to ensure the "glue" between modules works every time.

The setup and running of LINC and VLBI-cwl modules can be accomplished using the runners in https://github.com/tikk3r/flocs/tree/fedora-py3/runners although the submission scripts may need cluster-specific adjustments.

Below is a flowchart showing the different modules. These are described individually below with an overview of each module, and links to more detailed descriptions.

Module 0: Pipeline preparation

Required before any pipeline run.

Run the plot_field.py script using instructions here, which queries online databases for LotSS and LBCS to construct catalogues of sources in the field, and provides an initial graphical representation. The catalogues that this step generates are required for VLBI-cwl to run, so this step cannot be skipped. It will also provide a short summary of the observation parameters, to help the user understand if the data is suitable for LOFAR-VLBI processing. This step can and should be run before downloading the data to avoid wasting time and resources on an unsuitable dataset!

Module 1: LINC

Required before any pipeline run.

Option 1: Suitable prefactor3 or LINC solutions exist for the target field, i.e. the international stations were included in the calibrator run. In this case, all you need is the final cal_solutions.h5 from the target run (which contains the calibrator solutions). Check that the solutions are good before proceeding.

Option 2: Suitable calibrator / target solutions do not already exist. You need to run LINC (https://git.astron.nl/RD/LINC) for the calibrator and the target to generate the final cal_solutions.h5 from the target run (which contains the calibrator solutions). Do not change any of the default settings for either the calibrator or target, but please do check that your solutions are good before proceeding.

Module 1A: WIDE-FIELD MODE: ddf-pipeline

Required if you are planning on doing widefield processing, otherwise optional.

What it does: This is the direction-dependent processing for the Dutch array. Solutions from this are used to subtract sources outside the field of view of the international stations when performing widefield, high resolution imaging. If you are only processing "postage stamp" images of individual targets in the field, this step is not required.

Module 2: VLBI-cwl: setup

This module applies all relevant solutions from the previous modules and performs A-team clipping. It is a sub-workflow of the delay-calibration.cwl workflow but does not need to be run separately. This is the longest-running step of the VLBI-CWL workflow.

Module 3: VLBI-cwl: concatenate-flag

This module concatenates subbands into bands of 10 (2 MHz bandwidth) and performs aoflagging.

Module 3A: WIDE-FIELD MODE: subtract

This module uses the 6 arcsecond imaging output, generated by the DDF-pipeline (to Module 1A), to subtract sources beyond the 2.5 by 2.5 degrees field of view.

Module 4: VLBI-cwl: phaseup-concat

This module splits out and performs self-calibration on an in-field calibrator to correct the bulk (direction independent) phases and delays on the international stations.

Module 5: VLBI-cwl: split-directions

This module splits out target directions, applies the DI delay calibrator solutions, and performs self-calibration.

Module 6: WIDE-FIELD MODE: select and merge directions

This module selects the best calibrator sources, based on the images and solutions from all performed self-calibration cycles, after which the phase and (optionally) amplitude solutions from all selected calibrators are merged together into one giant h5parm.

Module 7A: WIDE-FIELD MODE: Intermediate resolution imaging

This model returns as a final product a wide-field image with a resolution of 1 arcsecond. This resolution allows extra averaging of the UV data, which speeds up imaging by a factor 4 compared to 0.3 arcsecond imaging (Ye et al. 2024; de Jong et al. in prep.)

Module 7B: WIDE-FIELD MODE: High resolution imaging

This model returns as a final product a wide-field image with a sub-arcsecond resolution. The highest resolution with the current available LOFAR stations is 0.3 arcseconds (Sweijen et al. 2022; de Jong et al. in prep). To complement the intermediate resolution, it is also possible to go for a 0.6 arcsecond resolution, which speeds up the the imaging by a factor 2 compared to 0.3 arcsecond imaging.