Karabo DAQ

Overview

The Karabo DAQ package (also known as pcLayer) contains the core devices of the Data AcQuisition of the Karabo SCADA (Supervisory Control And Data Acquisition) system, the data aggregators. These store data in HDF5 files.

Three types of data can be stored:

• Slow data are Karabo device properties, received via the Karabo broker.
• Fast data is Karabo pipeline data, received via TCP.
• XTDF (XFEL train data format) data is produced by the high-data rate detectors developed for the European XFEL (AGIPD, DSSC and LPD) and is received via UDP.

The DataAggregator device supports slow and fast data whereas the DataAggregatorXtdf supports XTFD data. DataAggregatorAgipd inherits from DataAggregatorXtdf and offers an AGIPD specific data reduction option.

One or more data aggregators are orchestrated by Karabo devices in the Karabo RunManagement package, the RunManager and the RunAssistant.

In contrast to the Karabo data logging system that deals only with slow data, the Karabo DAQ is run based and data is stored once per short time intervals. These intervals are identified by a unique timeId that is distributed by a Karabo time server. If slow data does not change compared to the previous time interval, the values of the previous interval is replicated. In contrast to that, if fast or XTDF data is not received by the data aggregator for a given time interval, no data is written.

Contact

For questions, please contact opensource@xfel.eu.

License and Contributing

This software is released by the European XFEL GmbH as is and without any warranty under the Mozilla Public License 2.0 (Core) license.

If you have questions on contributing to the project, please get in touch at opensource@xfel.eu.

External contributors, i.e. anyone not contractually associated to the European XFEL GmbH, are asked to sign a Contributor License Agreement (CLA):

• people contributing as individuals should sign the Individual CLA
• people contributing on behalf of an organization should sign the Entity CLA.

The CLAs can be found in the Contributor_License_Agreement.md and Entity_Contributor_License_Agreement.md documents located in the root folder of this repository. Please send signed CLAs to opensource@xfel.eu. We'll get in touch with you then. We ask for your understanding that we cannot accept external contributions without a CLA in place. Importantly, with signing the CLA you acknowledge that

• European XFEL retains all copyrights of Karabo DAQ,
• European XFEL may relicense Karabo DAQ under other appropriate open source licenses which the Free Software Foundation classifies as Free Software licenses.

However, you are welcome to already suggest modifications you'd like to contribute by opening a merge/pull request before you send the CLA.

Source Code

The sources for this project can be found at https://github.com/European-XFEL/Karabo-DAQ

Dependencies

• Karabo Hdf5: https://github.com/European-XFEL/Karabo-Hdf5
• Karabo RunManagement: https://github.com/European-XFEL/Karabo-RunManagement

The test code distributed in this repository depends in addition on packages that are still in the process of being published as open source:

• Karabo XtdfProducers
• Karabo DataGenerator
• Karabo TimeServer

Compiling and Testing

1. Activate the Karabo installation against which to compile the package:

cd <Karabo installation root directory>
source ./activate

2. Checkout the source code

mkdir -p $KARABO/devices
cd $KARABO/devices
git clone https://github.com/European-XFEL/Karabo-DAQ.git pcLayer

3. Goto the device source root directory and call the wrapper script around cmake:

cd $KARABO/devices/pcLayer
./.install.sh Release

Besides Release mode, also Debug, Simulation, and RelWithDebInfo are supported. To make the devices available for the C++ device server, the shared library in dist/Release/cmake/ must be copied to or linked from $KARABO/devices/pcLayer/dist/Release/cmake/.

4. C++ Unit Testing

cd $KARABO/devices/pcLayer/build

cmake -DCMAKE_BUILD_TYPE=Debug \
    -DBoost_NO_BOOST_CMAKE=ON \
    -DBoost_NO_SYSTEM_PATHS=ON \
    -DCMAKE_PREFIX_PATH=\"$KARABO\;$KARABO/extern\" \
    -DCMAKE_INSTALL_PREFIX=$KARABO \
    -DBUILD_TESTS=1 \
    -B build .
cd pcLayer
cmake --build . -j10

(or any other number of cores than 10)

5. Python Integration Tests

The Karabo DAQ device package is shipped with python integration tests. In order to make the device visible to any device-server you have to install the package to Karabo's own Python environment.

The local testing has a quick route:

source $KARABO/activate
cd <directory_of_pcLayer>
pip install -e integration_tests/

daq-install-all-dependencies

Note: Please be aware that daq-install-all-dependencies installs also the runManagement project. This way all dependencies are setup.

cd integration_tests
pytest -vv

Debugging a running service with gdb

It can be important to debug a data aggregator with gdb. Please have a look at the following

which karabo-cppserver  # gives you the path to the executable
karabo-stop serverID  # stop the daemontools server
cat var/data/serverId/run  # get additional server options including serverId, i.e. {OPTIONS}
gdb /path/to/karabo/cpp/server

and then in gdb

run {OPTIONS}