Merge remote-tracking branch 'origin/master' into software-catalog

Author: joonas-somero

docs/apps/lastools.md

@@ -13,7 +13,7 @@ tags:
 
 LAStools is included in following modules:
 
-* lastools: 2025 (more exactly 250219), 2023 (230914) and 2022 (220613)
+* lastools: 2025 (more exactly 250304), 2023 (230914) and 2022 (220613)
 * geoconda: 3.11.9, 3.10.9 and 3.10.6 (all with older 20171231)
 
 Load one of these modules, for example the newest version (default):

docs/apps/python.md

@@ -88,6 +88,19 @@ with the command `python3 --version`, and the full path of the command with
 `which python3` (to see if you are using the system Python or one from the
 modules listed above).
 
+!!! info ""
+
+    Note that most of the pre-installed Python environment modules are
+    self-contained and mutually exclusive environments, so it does not
+    make sense to for example load both python-data and pytorch
+    modules. The module loaded last will be the only active one, and
+    the module load command will warn about this, for example:
+
+    ```
+    Lmod is automatically replacing "python-data/3.10-24.04" with "pytorch/2.5".
+    ```
+
+
 ### Custom Python environments
 
 While the pre-installed Python environments suffice for many applications,

docs/cloud/pouta/multiattach.md

@@ -440,6 +440,49 @@ You can enable them if you wish with `pcs`:
     $> pcs resource create sharedfs1 --group shared_vg1 ocf:heartbeat:Filesystem device="/dev/shared_vg1/shared_lv1" directory="/mnt/gfs" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence
     ```
 
+* **How to extend my GFS2 volume?**
+
+    The GFS2 volume was configured using LVM ([Logical Volume Manager](https://en.wikipedia.org/wiki/Logical_volume_management)) that enhance the management and flexibility of physical storage.
+
+    a. Create a new multiattach volume and attach it to your instances. Check that the volume is well attached by running the command `sudo parted -l`
+
+    b. On one node, add the new volume in the Volume Group:
+
+    ```sh
+    sudo vgextend VolumeGroupName /dev/vdX
+    ```
+
+    c. Still on one node, extend the Logical Volume:
+
+    ```sh
+    sudo lvextend -l +100%FREE /dev/VolumeGroupName/LogicalVolumeName
+    ```
+
+    d. Check that the Logical Volume has been extended by running the command `sudo lvs`
+
+    e. Before extending the GFS2 volume, check on the other nodes that you don't have error messages. Run `sudo pvs`. If you see something like:
+
+    ```
+    WARNING: Couldn't find device with uuid JuoyG2-ftdd-U9xm-LLei-VrY7-4GZz-FgC2dr.
+    WARNING: VG shared_vg1 is missing PV JuoyG2-ftdd-U9xm-LLei-VrY7-4GZz-FgC2dr (last written to /dev/vdX)
+    ```
+    You must add the device by running the command:
+
+    ```
+    sudo lvmdevices --adddev /dev/vdX
+    ```
+
+    Check again with the command `sudo pvs`. The warning message shouldn't appear.
+
+    f. If everything's ok, you can grow your GFS2 volume by typing:
+
+    ```sh
+    sudo gfs2_grow <YourGFS2MountVolume>
+    ```
+
+    !!! warning
+        You cannot decrease the size of a GFS2 file system
+    
 * **What happens if a VM gets disconnected?**
 
     This covers two different use cases, a temporal and/or unexpected disconnection, and a permanent one.
@@ -484,6 +527,8 @@ You can enable them if you wish with `pcs`:
 - [Getting start with Pacamaker](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_getting-started-with-pacemaker-configuring-and-managing-high-availability-clusters#proc_learning-to-use-pacemaker-getting-started-with-pacemaker)
 - [Configuring a Red Hat High Availability cluster on Red Hat OpenStack Platform](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_a_red_hat_high_availability_cluster_on_red_hat_openstack_platform/index)
 - [GFS2 file systems in a cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_gfs2_file_systems/assembly_configuring-gfs2-in-a-cluster-configuring-gfs2-file-systems#proc_configuring-gfs2-in-a-cluster.adoc-configuring-gfs2-cluster)
+- [Growing a GFS2 file system](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_gfs2_file_systems/assembly_creating-mounting-gfs2-configuring-gfs2-file-systems#proc_growing-gfs2-filesystem-creating-mounting-gfs2)
+- [Managing LVM volume groups](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/managing-lvm-volume-groups_configuring-and-managing-logical-volumes#managing-lvm-volume-groups_configuring-and-managing-logical-volumes)
 
 ## OCFS2 as a second example
 

docs/cloud/rahti/images/Using_Rahti_integrated_registry.md

@@ -11,29 +11,29 @@ The process is simple:
 
 1. With a terminal, connect to the Rahti registry:
     ```sh
-    docker login -p $(oc whoami -t ) -u unused image-registry.apps.2.rahti.csc.fi
+    sudo docker login -p $(oc whoami -t ) -u unused image-registry.apps.2.rahti.csc.fi
     ```
- 
+
     _Alternatively, you can access to this address: <https://oauth-openshift.apps.2.rahti.csc.fi/oauth/token/display> to request
     a token. Once connected, display and copy the token. The command will be:_
 
     ```sh
-    docker login -p <YOUR_TOKEN> -u unused image-registry.apps.2.rahti.csc.fi
+    sudo docker login -p <YOUR_TOKEN> -u unused image-registry.apps.2.rahti.csc.fi
     ```
 
     !!! info
         If you get any error, make sure you are logged in. If you run `oc whoami`, the command should return your username.
 
 2. Tag the image you want to push:
    ```sh
-   docker tag centos:7 image-registry.apps.2.rahti.csc.fi/{YOUR_RAHTI_PROJECT_NAME}/centos:<tag>
+   sudo docker tag centos:7 image-registry.apps.2.rahti.csc.fi/{YOUR_RAHTI_PROJECT_NAME}/centos:<tag>
    ```
    _Replace {YOUR_RAHTI_PROJECT_NAME} by the name of your project._
    _Please note that YOUR_RAHTI_PROJECT_NAME here is the Rahti project name (AKA namespace name), and does not refer to CSC project._
 
 4. Push your image:
    ```sh
-   docker push image-registry.apps.2.rahti.csc.fi/{YOUR_RAHTI_PROJECT_NAME}/centos:<tag>
+   sudo docker push image-registry.apps.2.rahti.csc.fi/{YOUR_RAHTI_PROJECT_NAME}/centos:<tag>
    ```
 
 You should be able to see your images in your project:

docs/cloud/rahti/usage/cli.md

@@ -80,8 +80,8 @@ Rahti also offers the opportunity of using an internal service account to intera
 
 ```sh
 oc create serviceaccount pusher
-oc policy add-role-to-user system:image-pusher pusher
-docker login -p $(oc sa get-token pusher) -u unused image-registry.apps.2.rahti.csc.fi
+oc policy add-role-to-user system:image-pusher -z pusher
+docker login -p $(oc create token pusher) -u unused image-registry.apps.2.rahti.csc.fi
 ```
 
 This service account token, the one you get with `oc sa get-token pusher` does not expire.

docs/data/sensitive-data/sd-connect-command-line-interface.md

@@ -1,18 +1,17 @@
 # Command Line Interface and automated key management
 
-The new SD Connect command line tools, available from February 2025, support file upload, download (with a-commands) and automated key management (with lock-unlock) during encryption and decryption. After programmatic encryption and upload, data can be viewed through the SD Connect user interface and SD Desktop. Coding skills are required to use the tools effectively, below is a step by step guide to get started. In contrast, files have been uploaded before February 2025, were manually encrypted using your encryption key and will need to be decrypted manually after download. 
+The new SD Connect command line tools, available from February 2025, support file upload, download (with a-commands) and automated key management (with lock-unlock) during encryption and decryption. After programmatic encryption and upload, data can be viewed through the SD Connect user interface and SD Desktop. Coding skills are required to use the tools effectively, below is a step by step guide to get started. In contrast, files have been uploaded before February 2025, were manually encrypted using your encryption key and will need to be decrypted manually after download.
 
 - [Background information](#background-information)
 - [Command line tools and automated key management](#command-line-tools-and-automated-key-management)
 - [Command line tools and manual encryption](#command-line-tools-and-manual-encryption)
 - [Tutorials](#tutorials)
 
-
 ## Background information
 
 SD Connect is part of CSC's Sensitive Data Services, offering a free and secure data processing environment for academic research projects at Finnish universities and research institutes. SD Connect enhances the Allas object storage system by adding an automatic encryption layer, enabling secure storage of sensitive data. Data stored in SD Connect can also be accessed through SD Desktop for secure virtual desktops. While SD Connect is typically accessed via the SD Connect Web interface, command-line tools may offer a more efficient way to manage data in certain situations.
 
-This document provides instructions on how you can install on your local environment (Linux, Mac) and how you can use the a-commands from the allas-cli-utils package to upload and download with automated key management via command line with SD Connect. 
+This document provides instructions on how you can install on your local environment (Linux, Mac) and how you can use the a-commands from the allas-cli-utils package to upload and download with automated key management via command line with SD Connect.
 
 !!! Note
     Allas itself does not differentiate between data uploaded via SD Connect (user interface or commandline tools) and data uploaded to Allas using different methods. Data buckets may contain a mix of SD Connect data, other encrypted data, and regular data. It is the user's responsibility to manage data types within the buckets. However, it is recommended to store SD Connect data in separate buckets and folders to avoid mixing different data types.
@@ -23,7 +22,7 @@ This document provides instructions on how you can install on your local environ
 
 To upload and automatically encrypt sensitive data to SD Connect programmatically, you need to install the command-line tools, which require root access to your laptop or local environment (Mac or Linux). For this reason, you might need support from your organization’s IT unit.
 
-Here you can find step-by-step instructions: https://github.com/CSCfi/allas-cli-utils. This guide provides installation instructions for the a-commands (used to upload and download files) as well as the lock and unlock commands (used to automatically encrypt and decrypt files via automated key management).
+[Here you can find step-by-step instructions](https://github.com/CSCfi/allas-cli-utils). This guide provides installation instructions for the a-commands (used to upload and download files) as well as the lock and unlock commands (used to automatically encrypt and decrypt files via automated key management).
 
 !!! Note
     If you need to upload non-sensitive data (such as scripts, containers, or software for use in SD Desktop), note that these tools are also available on CSC's supercomputers (Puhti, Mahti, and Lumi). However, these systems are restricted to non-sensitive data only. Sensitive data must be uploaded to SD Connect through the appropriate channels.
@@ -36,27 +35,29 @@ To open SD Connect compatible Allas connection you must add option *--sdc* the c
 module load allas
 allas-conf --sdc
 ```
+
 In local installations the connection is typically opened with commands like
 
 ```bash
 export PATH=/some-local-path/allas-cli-utils:$PATH
 source /some-local-path/allas-cli-utils/allas_conf -u your-csc-account --sdc
 ```
 
-The set up process asks first your CSC passwords (Haka or Virtu passwords can't be used here).
-After that you will select the CSC project to be used. This is the normal login process for Allas.
-However, when SD Connect is enabled, the process asks you to give the *SD Connect API token*. This
-token must be retrieved from the [SD Connect web interface](https://sd-connect.csc.fi). Note that the tokens
-are project specific. Make sure you have selected the same SD Connect project in both command line and in web 
-interface.
+- The set up process asks first your CSC passwords (Haka or Virtu passwords can't be used here). After that you will select the CSC project to be used. This is the normal login process for Allas.
+- However, when SD Connect is enabled, the process asks you to give the *SD Connect API token*.
+
+To retrieve the temporary SD Connect API token:
 
-In the web interface the token can be created using dialog that opens by selecting *Create API tokens* from the *Support* menu.
+- Login to the [SD Connect web interface](https://sd-connect.csc.fi). If you have multiple CSC projects, make sure you have selected the same SD Connect project in both the command line and the web interface (top left corner).  
+- In the top right corner of the web interface, click on Support, then select Select API Token from the dropdown menu.
+- In the new dialog, enter a name for your temporary token. Note: Tokens are project-specific, so the name must be unique. Avoid using special characters in the name.
+- Click on Create Token. The token will be displayed only once. Once you see the token, copy it (click the icon to the left of the token). Important: make sure to store it securely, as it will not be retrievable later.
 
-Copy the token, paste into to command line and press enter.
+    ![API token](https://a3s.fi/docs-files/sensitive-data/SD_Connect/SDConnect_APItoken.png)
 
-The SD Connect compatible Allas connection is now valid for next eight hours. And you can use commands like
-*a-list* and *a-delete* to manage both normal Allas objects and SD Connect objects.
+- The token will be valid for 24 hours and will be automatically deleted after this period. Paste the token into the command line and press Enter to use it.
 
+The SD Connect compatible Allas connection is now valid for next eight hours. And you can use commands like *a-list* and *a-delete* to manage both normal Allas objects and SD Connect objects.
 
 ### Step 3: Data upload and automated encryption
 
@@ -72,7 +73,7 @@ This will produce SD Connect object: 2000123-sens/dataset2/my-secret-table.csv.c
 All other a-put options and features can be used too. For example directories are
 stored as tar files, if --asis option is not used.
 
-Command: 
+Command:
 
 ```bash
 a-put --sdc my-secret-directory -b 2000123-sens/dataset2
@@ -94,13 +95,11 @@ The command above will copy all the files from directory my-secret-directory to
 !!! Note
     Since SD Connect was updated in October 2024, it is no longer straightforward to determine which encryption method was used for an encrypted .c4gh file stored in Allas/SD Connect. If you are now using a new encryption method to upload files to an existing CSC project, please ensure you add a note to your folders indicating that the encryption protocol has changed. You can either share this information with your colleagues or clearly include it in the folder name. As a good practice, we advise creating a new folder and avoiding mixing files encrypted with different methods.
 
-
-
 ### Step 4: Data download and automated decryption
 
 Data can be downloaded from Allas with command a-get. If SD Connect connection is enabled, a-get will automatically try to decrypt objects with suffix *.c4gh*.
 
-So for example command: 
+So for example command:
 
 ```bash
 a-get 2000123-sens/dataset2/my-secret-table.csv.c4gh
@@ -114,7 +113,7 @@ And similarly command:
 a-get 2000123-sens/dataset2/my-secret-directory.tar.c4gh
 ```
 
-Will produce local directory: my-secret-directory 
+Will produce local directory: my-secret-directory
 
 Note that this automatic decryption works only for the files that have
 been stored using the new SD Connect that was taken in use in October 2024.
@@ -127,8 +126,7 @@ a-get --sk my-key.sec  2000123-sens/old-date/sample1.txt.c4gh
 ```
 
 Unfortunately there is no easy way to know, which encryption method has been used in
-a .c4gh file stored in Allas. 
-
+a .c4gh file stored in Allas.
 
 ## Command line tools and manual encryption
 
@@ -197,7 +195,7 @@ In this example, we first generate your key pair (a password-protected private k
          C4GH_SECRET_KEY  If defined, it will be used as the default secret key (ie --sk ${C4GH_SECRET_KEY})
       ```
 
-      You may notice that crypt4gh uses `--sk` option for the private key. This might seem odd but apparently, crypt4gh uses term _secure key_ for private key, hence `sk`, and consequently `pk` refers to public key instead of the private key.
+      You may notice that crypt4gh uses `--sk` option for the private key. This might seem odd but apparently, crypt4gh uses term *secure key* for private key, hence `sk`, and consequently `pk` refers to public key instead of the private key.
 
 ### 2.2 Decrypt a file
 
@@ -216,22 +214,16 @@ The command will ask the user to enter the password (passphrase) of your private
 
 Additional information about [data encryption](./sd-connect-introduction-to-data-encryption.md).
 
-
-
 ## Tutorials
 
 - [Tools for client side encryption for Allas](../Allas/allas_encryption.md)
-
 - [Decrypting all files in a directory](../sensitive-data/tutorials/decrypt-directory.md)
-
 - [Using Allas storage service to receive sensitive research data](../sensitive-data/sequencing_center_tutorial.md)
 
-
-
 ## Features in SD Connect
 
-* [Upload](./sd-connect-upload.md)
-* [Share](./sd-connect-share.md)
-* [Download](./sd-connect-download.md)
-* [Delete](./sd-connect-delete.md)
-* [Troubleshooting](./sd-connect-troubleshooting.md)
+- [Upload](./sd-connect-upload.md)
+- [Share](./sd-connect-share.md)
+- [Download](./sd-connect-download.md)
+- [Delete](./sd-connect-delete.md)
+- [Troubleshooting](./sd-connect-troubleshooting.md)