Merge branch 'main' into common-mirror

Author: Villtord

docs/explanations/decisions/0006-devices-shared-between-endstations.md

@@ -0,0 +1,27 @@
+# 6. Handle devices shared between multiple endstations
+
+Date: 2025-08-27
+
+## Status
+
+Proposed
+
+## Context
+
+Some beamlines have multiple endstations with shared hardware in the optics or experiment hutch, and could potentially be trying to control it at the same time. Any device in the common hutch should only be fully controlled by one endstation at a time - the one that is taking data - but still be readable from the other endstations.
+
+## Decision
+
+The current solution is to have a separate blueapi instance for the shared hutch in order to be able to control the access to all the devices defined there.
+For all hardware in the shared optics hutch, the architecture should follow this structure:
+
+- There is a base device in dodal that sends a REST call to the shared blueapi with plan and devices names, as well as the name of the endstation performing the call.
+- There are read-only versions of the shared devices in the endstation blueapi which inherit from the base device above and set up the request parameters.
+- The real settable devices are only defined in the shared blueapi and should never be called directly from a plan.
+- The shared blueapi instance also has an ``AccessControl`` device that reads the endstation in use for beamtime from a PV.
+- Every plan should then be wrapped in a decorator that reads the ``AccessControl`` device, check which endstation is making the request and only allows the plan to run if the two values match.
+
+
+:::{seealso}
+[Optics hutch implementation on I19](https://diamondlightsource.github.io/i19-bluesky/main/explanations/decisions/0004-optics-blueapi-architecture.html) for an example.
+:::

docs/how-to/create-beamline.rst renamed to docs/how-to/create-beamline.md

@@ -1,13 +1,11 @@
-Creating a new beamline
-=======================
+# Creating a new beamline
 
 A beamline is a collection of devices that can be used together to run experiments, they may be read-only or capable of being set.
 They include motors in the experiment hutch, optical components in the optics hutch, the synchrotron "machine" and more.
 
-Beamline Modules
-----------------
+## Beamline Modules
 
-Each beamline should have its own file in the ``dodal.beamlines`` folder, in which the particular devices for the 
+Each beamline should have its own file in the ``dodal.beamlines`` folder, in which the particular devices for the
 beamline are instantiated. The file should be named after the colloquial name for the beamline. For example:
 
 * ``i03.py``
@@ -16,14 +14,14 @@ beamline are instantiated. The file should be named after the colloquial name fo
 
 Beamline modules (in ``dodal.beamlines``) are code-as-configuration. They define the set of devices and common device
 settings needed for a particular beamline or group of similar beamlines (e.g. a beamline and its digital twin). Some
-of our tooling depends on the convention of *only* beamline modules going in this package. Common utilities should 
+of our tooling depends on the convention of *only* beamline modules going in this package. Common utilities should
 go somewhere else e.g. ``dodal.utils`` or ``dodal.beamlines.common``.
 
 The following example creates a fictitious beamline ``w41``, with a simulated twin ``s41``.
 ``w41`` needs to monitor the status of the Synchrotron and has an AdAravisDetector.
 ``s41`` has a simulated clone of the AdAravisDetector, but not of the Synchrotron machine.
 
-.. code-block:: python
+```python
 
     from ophyd_async.epics.adaravis import AravisDetector
 
@@ -85,3 +83,4 @@ The following example creates a fictitious beamline ``w41``, with a simulated tw
             drv_suffix=CAM_SUFFIX,
             fileio_suffix=HDF5_SUFFIX,
         )
+```

docs/how-to/move-code.md

@@ -0,0 +1,94 @@
+
+# Moving code from another repo
+
+In the process of writing code in other DLS repos you may come to realise that it makes more sense to be in ``dodal``. It is a good idea to keep the history for this code, you can do this by doing the following (we will be using moving devices from https://github.com/DiamondLightSource/hyperion as an example):
+
+* Clone the codebase you are copying from:
+
+```bash
+    git clone [email protected]:DiamondLightSource/hyperion.git clone_for_history
+    cd clone_for_history/
+```
+
+* Remove the remote to avoid any mistaken pushes:
+
+```bash
+    git remote rm origin
+```
+
+* Filter out only the directory/file you want to move:
+
+```bash
+    pip install git-filter-repo
+    git-filter-repo --path file/to/move --path /other/file/to/move -f
+```
+
+* Clean everything up:
+
+```bash
+    git reset --hard
+    git gc --aggressive
+    git prune
+    git clean -fd
+```
+
+* Add a note to every commit message to mention it's been moved::
+
+```bash
+    git filter-branch --msg-filter 'sed "$ a \
+    NOTE: Commit originally came from https://github.com/DiamondLightSource/hyperion"' -f -- --all
+```
+
+* If you have been using Github [issue references](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls#issues-and-pull-requests) in the old repository modify these to point to be more explicit (Note that this assumes the old repo uses ``#123`` notation and only ever references issues from it's own repo)::
+
+```bash
+    git filter-branch -f --msg-filter 'sed "s|#[0-9]\+|DiamondLightSource/hyperion&|g"' -- --all
+```
+
+* Prepare the code to be in the correct structure for dodal::
+
+```bash
+    mkdir -p src/dodal/devices
+    mv path/to/device src/dodal/devices/
+```
+
+* At this point it's a good idea to check the log ``git log`` and the general directory structure to ensure it looks mostly correct
+
+* Add and commit this (locally)::
+
+```bash
+    git add .
+    git commit -m "Prepare for import into dodal"
+```
+
+* In a different folder clone ``dodal``, remove the origin (for now) to be safe and create a branch::
+
+```bash
+    git clone [email protected]:DiamondLightSource/dodal.git
+    cd dodal
+    git remote rm origin
+    git checkout -b add_code_from_hyperion
+```
+
+* Add the source repo as a remote for ``dodal``::
+
+```bash
+    git remote add source /path/to/source/old_repo/.git
+```
+
+* Pull from the source repo::
+
+```bash
+    git pull --no-rebase source main --allow-unrelated-histories
+```
+
+* This is another point where it's a good idea to check the log ``git log`` and the general directory structure to ensure it looks mostly correct
+
+* Remove the source remote and re-add origin::
+
+```bash
+    git remote rm source
+    git remote add origin [email protected]:DiamondLightSource/dodal.git
+```
+
+* Tidy up the code so that it fits into the ``dodal`` repo e.g. in the Hyperion case we had to change the tests to import from ``hyperion`` to import from ``dodal`` and add some more dependencies.

docs/how-to/move-code.rst

@@ -7,6 +7,9 @@ Testing is essential to maintain the integrity and reliability of the codebase.
 - **Unit Tests**: Place unit tests for individual components in the `tests` directory, but take care to mirror the file structure of the `src` folder with the corresponding code files. Use the `test_*.py` naming convention for test files.
 - **System Tests**: Tests that interact with DLS infrastructure, network, and filesystem should be placed in the top-level `systems_test` folder. This separation ensures that these tests are easily identifiable and can be run independently from unit tests.
 
+Useful functions for testing that can be reused across multiple tests for common devices and for external plan repositories belong in the `dodal/testing` directory. For example, when mocking a `Motor` device, all of the signals will default to zero, which will cause errors when trying to move. The `patch_motor` and `patch_all_motors` functions, found in `dodal.testing`, will populate the mocked motor with useful default values for the signals so that it can still be used in tests.
+
+
 ## Writing a test for a device
 We aim for high test coverage in dodal with small, modular test functions. To achieve this, we need to test the relevant methods by writing tests for the class/method we are creating or changing, checking for the expected behaviour. We shouldn't need to write tests for parent classes unless we alter their behaviour.
 

docs/how-to/write-tests.md

@@ -1,32 +1,27 @@
-Zocalo Interaction
-==================
+# Zocalo Interaction
 
-.. image:: ../assets/zocalo.png
-  :alt: Diagram of zocalo
+![Diagram of zocalo](../assets/zocalo.png)
 
-Zocalo jobs are triggered based on their ISPyB DCID using the ``ZocaloTrigger`` class in a callback subscribed to the 
-Bluesky plan or ``RunEngine``. These can trigger processing for any kind of job, as zocalo infers the necessary 
+Zocalo jobs are triggered based on their ISPyB DCID using the ``ZocaloTrigger`` class in a callback subscribed to the
+Bluesky plan or ``RunEngine``. These can trigger processing for any kind of job, as zocalo infers the necessary
 processing from data in ISPyB.
 
-Results are received using the ``ZocaloResults`` device, so that they can be read into a plan and used for 
-decision-making. Currently the ``ZocaloResults`` device is only made to handle X-ray centring results. It subscribes to 
+Results are received using the ``ZocaloResults`` device, so that they can be read into a plan and used for
+decision-making. Currently the ``ZocaloResults`` device is only made to handle X-ray centring results. It subscribes to
 a given zocalo RabbitMQ channel the first time that it is triggered.
 
-Zocalo Service
-==============
+# Zocalo Service
 
-The Zocalo service processes incoming messages using recipes which describe routing of messages between processing 
-steps. You can see `source for the recipes here`_.
-
-.. _source for the recipes here: https://gitlab.diamond.ac.uk/scisoft/zocalo/-/tree/master/recipes
+The Zocalo service processes incoming messages using recipes which describe routing of messages between processing
+steps. You can see [source for the recipes here](https://gitlab.diamond.ac.uk/scisoft/zocalo/-/tree/master/recipes).
 
 You can find more information about Zocalo at https://confluence.diamond.ac.uk/display/SSCC/How+to+create+an+MX+processing+pipeline
 
-Gridscans
----------
+## Gridscans
 
-The Zocalo Service receives messages of the following form for both the xy and xz gridscans::
+The Zocalo Service receives messages of the following form for both the xy and xz gridscans:
 
+```
     {
         'recipes': ['mimas'],
         'parameters': {
@@ -39,9 +34,11 @@ The Zocalo Service receives messages of the following form for both the xy and x
             'guid': 'd6f117bb-c856-4df8-b9bc-2d3c625e9fd5'
         }
     }
+```
 
-Zocalo is then sent stop messages::
+Zocalo is then sent stop messages:
 
+```
    {
         'recipes': ['mimas'],
         'parameters': {
@@ -50,22 +47,22 @@ Zocalo is then sent stop messages::
             'guid': '9a96e59c-da30-494c-8380-c7a5c828c2c9'
         },
    }
+```
 
 these tell zocalo that the data is now ready to be processed.
 
 Zocalo then uses the ISPyB DataCollection ID to fetch the corresponding info from ISPyB
 
-The messages that zocalo receives can be found in Graylog in the Zocalo stream, from there you can find the log of 
+The messages that zocalo receives can be found in Graylog in the Zocalo stream, from there you can find the log of
 the recipe processing using the path to the logbook that comes from messages like these::
     Message saved in logbook at /dls/tmp/zocalo/dispatcher/2024-12/cb/62d35b-7cc9-4f1b-868b-712e82aa0271
 
-From the zocalo graylog you can also see that once the gridscan nexus file is picked up (for the CPU gridscan) it 
-starts a recipe::
+From the zocalo graylog you can also see that once the gridscan nexus file is picked up (for the CPU gridscan) it
+starts a recipe:
 
+```
     {'recipes': ['per-image-analysis-gridscan-i03-no-really'], 'parameters': {'ispyb_dcid': 16085803, 'filename': '{filename}', 'start_frame_index': '{start_frame_index}', 'number_of_frames': '{number_of_frames}', 'message_index': '{message_index}', 'guid': 'd10f8b8c-57a5-4dc4-acaf-a22f8d2bbf60'}, 'recipe': <workflows.recipe.recipe.Recipe object at 0x7f0587f13110>}
+```
 
-
-If you look at the recipe json, Zocalo then runs Per-Image-Analysis on each frame and then assembles the results in 
-the `DLS X-Ray Centring service`_.
-
-.. _DLS X-Ray Centring service: https://github.com/DiamondLightSource/python-dlstbx/blob/a8fcbd30335bf13f5e35df78badfc60397500535/src/dlstbx/services/xray_centering.py
+If you look at the recipe json, Zocalo then runs Per-Image-Analysis on each frame and then assembles the results in
+the [DLS X-Ray Centring service](https://github.com/DiamondLightSource/python-dlstbx/blob/a8fcbd30335bf13f5e35df78badfc60397500535/src/dlstbx/services/xray_centering.py).