Add possibility to produce DL3 with ctapipe

.codespell-ignores

@@ -3,3 +3,4 @@ usera
 nd
 studi
 referenc
+livetime

docs/api-reference/irf/cuts/index.rst

@@ -0,0 +1,27 @@
+.. _irf:
+
+**********************************************
+Cuts (`~ctapipe.irf.cuts`)
+**********************************************
+
+.. currentmodule:: ctapipe.irf.cuts
+
+This sub-module contains functionalities representing and applying cuts.
+
+:ref:`quality_cuts` is for quality selection of the events and :ref:`selection_cuts` contain the 'gamma-like' events selections
+
+Submodules
+==========
+
+.. toctree::
+    :maxdepth: 1
+
+    quality_cuts
+    selection_cuts
+
+
+Reference/API
+=============
+
+.. automodapi:: ctapipe.irf.cuts
+    :no-inheritance-diagram:

docs/api-reference/irf/cuts/quality_cuts.rst

@@ -0,0 +1,12 @@
+.. _preprocessing:
+
+*******************************
+Quality cuts
+*******************************
+
+
+Reference/ API
+==============
+
+.. automodapi:: ctapipe.irf.cuts.quality_cuts
+    :no-inheritance-diagram:

docs/api-reference/irf/cuts/selection_cuts.rst

@@ -0,0 +1,12 @@
+.. _preprocessing:
+
+*******************************
+Selection cuts
+*******************************
+
+
+Reference/ API
+==============
+
+.. automodapi:: ctapipe.irf.cuts.selection_cuts
+    :no-inheritance-diagram:

docs/api-reference/irf/dl3.rst

@@ -0,0 +1,12 @@
+.. _dl3:
+
+*******
+DL3
+*******
+
+
+Reference/ API
+==============
+
+.. automodapi:: ctapipe.irf.dl3
+    :no-inheritance-diagram:

docs/api-reference/irf/index.rst

@@ -29,8 +29,10 @@ Submodules
 .. toctree::
     :maxdepth: 1
 
-    optimize
+    optimize/index
+    cuts/index
     irfs
+    dl3
     benchmarks
     binning
     preprocessing

docs/api-reference/irf/optimize/algorithm.rst

@@ -0,0 +1,12 @@
+.. _preprocessing:
+
+*******************************
+Cuts Optimization Results
+*******************************
+
+
+Reference/ API
+==============
+
+.. automodapi:: ctapipe.irf.optimize.algorithm
+    :no-inheritance-diagram:

docs/api-reference/irf/optimize/index.rst

@@ -0,0 +1,30 @@
+.. _irf:
+
+**********************************************
+Cuts optimization (`~ctapipe.irf.optimize`)
+**********************************************
+
+.. currentmodule:: ctapipe.irf.optimize
+
+This sub-module contains functionalities for optimizing cuts.
+
+The cut optimization as well as the calculations of the irf components and the benchmarks
+are done using the `pyirf <https://pyirf.readthedocs.io/en/stable/>`_ package.
+
+:ref:`algorithm` contain the algorithm for computing the cuts while :ref:`results` contain the object storing the optimize cuts
+
+Submodules
+==========
+
+.. toctree::
+    :maxdepth: 1
+
+    algorithm
+    results
+
+
+Reference/API
+=============
+
+.. automodapi:: ctapipe.irf.optimize
+    :no-inheritance-diagram:

docs/api-reference/irf/optimize/results.rst

@@ -0,0 +1,12 @@
+.. _preprocessing:
+
+*******************************
+Cuts Optimization Results
+*******************************
+
+
+Reference/ API
+==============
+
+.. automodapi:: ctapipe.irf.optimize.results
+    :no-inheritance-diagram:

docs/api-reference/tools/index.rst

@@ -111,6 +111,9 @@ Reference/API
 .. automodapi:: ctapipe.tools.compute_irf
     :no-inheritance-diagram:
 
+.. automodapi:: ctapipe.tools.create_dl3
+    :no-inheritance-diagram:
+
 .. automodapi:: ctapipe.tools.dump_instrument
    :no-inheritance-diagram:
 

docs/user-guide/tools/dl3_guide.rst

@@ -0,0 +1,97 @@
+.. _dl3_guide:
+
+***************************************************************
+How to produce DL3 for observations using ``ctapipe`` tools
+***************************************************************
+
+The guide explains how to obtain a DL3 file (gamma-like events with IRFs) for your observations.
+
+.. note::
+   * This guide assumes you have Monte Carlo simulations processed to obtain RF and IRFs. For more details, see :doc:`IRF guide <irf_guide>` and :doc:`DL2 guide <dl2_guide>`.
+   * Use the same ctapipe configuration files for processing MC and observations in all steps, ensuring the same processing is applied to both.
+
+Setup
+=====
+
+First define the following environment variables:
+
+* ``DL0_FOLDER`` — directory with the DL0 data
+* ``DL1_FOLDER`` — directory with the DL1 data
+* ``DL2_FOLDER`` — directory with the DL2 data
+* ``DL3_FOLDER`` — directory with the DL3 data
+* ``RF_FOLDER`` — directory with the random forest models for the reconstruction
+* ``IRF_FOLDER`` — directory with the IRFs and associated cuts file
+* ``CONFIG_DL1`` — configuration file for the DL0→DL1 processing, e.g. ``optimize_dl0_to_d1.yaml``
+* ``CONFIG_DL3`` — configuration file for the DL2→DL3 processing of observations
+  (quality cuts must be identical to those used for optimized cuts and IRFs),
+  e.g. ``optimize_dl2_to_d3_obs.yaml``
+
+Running the tools
+=================
+
+1) DL0 → DL1
+------------
+
+Process your DL0 file to DL1 level:
+
+.. code-block:: bash
+
+   ctapipe-process \
+     --input "$DL0_FOLDER/MyRun_subrun_xxx.dl0.h5" \
+     --output "$DL1_FOLDER/MyRun_subrun_xxx.dl1.h5" \
+     --config "$CONFIG_DL1" \
+     --provenance-log "$DL1_FOLDER/MyRun_subrun_xxx.dl1.provenance.log" \
+     --log-file "$DL1_FOLDER/MyRun_subrun_xxx.dl1.log"
+
+If your observation is divided into subruns, merge them:
+
+.. code-block:: bash
+
+   ctapipe-merge --progress \
+     --telescope-events \
+     --dl1-parameters \
+     --no-dl1-images \
+     --single-ob \
+     -i "$DL1_FOLDER" \
+     -o "$DL1_FOLDER/MyRun.dl1.h5" \
+     -l "$DL1_FOLDER/MyRun_subrun_xxx.dl1.log"
+
+2) DL1 → DL2 (apply RF models)
+------------------------------
+
+Apply the RF models trained on MC to reconstruct events and obtain the DL2 file:
+
+.. code-block:: bash
+
+   ctapipe-apply-models \
+     --input "$DL1_FOLDER/MyRun.dl1.h5" \
+     --output "$DL2_FOLDER/MyRun.dl2.h5" \
+     --reconstructor "$RF_FOLDER/energy_regressor.pkl" \
+     --reconstructor "$RF_FOLDER/particle_classifier.pkl" \
+     --reconstructor "$RF_FOLDER/disp_reconstructor.pkl" \
+     --provenance-log "$DL2_FOLDER/MyRun.provenance.log" \
+     --log-file "$DL2_FOLDER/MyRun.dl2.log" \
+     --log-level INFO
+
+.. note::
+   The option ``--reconstructor "$RF_FOLDER/particle_classifier.pkl"`` is only required for **monoscopic** reconstructions.
+
+3) DL2 → DL3
+------------
+
+You could finally produce your DL3 file:
+
+.. code-block:: bash
+
+   ctapipe-create-dl3 \
+     --dl2-file "$DL2_FOLDER/MyRun.dl2.h5" \
+     --output "$DL3_FOLDER/MyRun.dl3.fits.gz" \
+     --irfs-file "$IRF_FOLDER/MyIRFs.fits" \
+     --cuts "$IRF_FOLDER/MyCuts.fits" \
+     -c "$CONFIG_DL3" \
+     --no-optional-columns \
+     --provenance-log "$DL3_FOLDER/MyRun.dl3.provenance.log" \
+     --log-file "$DL3_FOLDER/MyRun.dl3.log" \
+     --log-level INFO
+
+The DL3 file is now ready to be used for high level analysis.

docs/user-guide/tools/index.rst

@@ -29,6 +29,7 @@ Data Processing Tools
   Calculate gamma/hadron and direction cuts (e.g. for IRF calculation).
 * `ctapipe-compute-irf <ctapipe.tools.compute_irf.IrfTool>`: Calculate an IRF with or without applying a direction cut
   and optionally benchmarks.
+* `ctapipe-create-dl3 <ctapipe.tools.create_dl3.DL3Tool>`: Create a DL3 file (gamma-like events and IRFs) from a DL2 file and an IRF fits file
 * `ctapipe-store-astropy-cache <ctapipe.tools.store_astropy_cache.main>`: Store astropy downloadable data
   in a given directory. Useful to run ctapipe in clusters where worker nodes might not have internet access.
 
@@ -52,3 +53,4 @@ The following pages contain examples on how to use the command-line tools.
 
   dl2_guide
   irf_guide
+  dl3_guide

pyproject.toml

@@ -108,6 +108,7 @@ ctapipe-process = "ctapipe.tools.process:main"
 ctapipe-merge = "ctapipe.tools.merge:main"
 ctapipe-optimize-event-selection = "ctapipe.tools.optimize_event_selection:main"
 ctapipe-compute-irf = "ctapipe.tools.compute_irf:main"
+ctapipe-create-dl3 = "ctapipe.tools.create_dl3:main"
 ctapipe-fileinfo = "ctapipe.tools.fileinfo:main"
 ctapipe-quickstart = "ctapipe.tools.quickstart:main"
 ctapipe-calculate-pixel-statistics = "ctapipe.tools.calculate_pixel_stats:main"