rucio-ams: Fermilab Rucio Deployment Framework

Scientific Data Management

Brandon White

bjwhite@fnal.gov

Dennis Lee

dylee@fnal.gov

Introduction

This repository contains scripts and configurations to deploy Rucio to Kubernetes clusters at Fermilab.

This framework uses kustomize (kubectl apply -k) to allow modification of Helm template outputs from the official Rucio Helm charts. This is used in particular to patch the environment of containers such that they are configured to have the database connection string passed in via the value of a Secret. This allows us to keep the Kubernetes resource manifests generated by the Helm templates under version control, without exposing a base64 encoded password as compared with a --values injection to the Helm template process. Official Rucio containers are used in the deployment.

TODO: The secrets and credentials needed for the application will need to be pushed to Vault. Vault-resident secrets will be downloaded when make apply is run and then loaded into the cluster via SecretGenerator definitions in the kustomization.yaml files.

At Fermilab, we use centrally managed PostgreSQL databases.

Requirements

• kubectl
• Permission to access the Kubernetes cluster
• (For OKD Clusters) OpenShift CLI (oc)
• (Optional) k9s for easier management
• Secrets
  • hostcert.pem: Host server certificate
  • hostkey.pem: Host server key
  • ca.pem: CA Certificate bundle
  • db-connstr: Database Connection String; in the format of postgresql+psycopg://<user>:<password>@<database-url>:<database-port>/<rucio-db-name>
    • Using only postgresql:// defaults to using psycopg2 in SQLAlchemy, which is not installed in the Rucio containers. Rucio containers are now installing psycopg, which is actuall psycopg3. Thus, we use posgtresql+psycopg:// to switch to the proper psycopg.
    • See: https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#module-sqlalchemy.dialects.postgresql.psycopg
  • Note: These secrets should be stored onto a secrets management service (i.e. Vault), then pulled when apply configurations
• (Optional) Custom Policy Package
  • __init__.py
  • permission.py
  • schema.py

Usage

1. Have the following secrets stored in overlays/<experiment>/rucio/etc/.secrets/

   These secrets should optimally be downloaded from a secrets management service (i.e. Vault) at deployment time for centralized distribution, rather than being stored locally.

   If using a custom policy package:

   • Add __init__.py, permission.py, and schema.py to <rucio-ams>/overlays/<experiment>/etc/policy-package
   • No other configuration is required, as the files are mounted as a secret. All policy packages will be referred to in the containers/config files as "fermilab", regardless of the experiment-specific policy package implementation files mounted.

2. Enter your desired experiment directory

   $ cd overlays/int

3. Use the proper Kubernetes cluster configuration

   $ oc project <experiment>
   
   # OR
   
   $ export KUBE_CONFIG=<path to kubernetes configuration>

4. Deploy Rucio for a given overlay using the Makefile in the experiment directory.

   $ make apply

Mechanism in make apply

make apply runs multiple steps, which includes downloading Secrets from Vault (TODO), building K8s resource manifests with Helm templates, and ending by using Kustomize to create or update the Rucio application resources.

Project Structure

• images: Contains container images for use in our Rucio deployments
  • Most, if not all, of these are considered legacy, since we are now using the official images provided by the Rucio team https://github.com/rucio/containers
  • The client image is what we use for running custom scripts
• overlays: Contains configuration and scripts to deploy to experiments

Setting Up a New Experiment

1. Copy the int directory to the overlays directory

   $ cp -r overlays/int overlays/<new-experiment>

2. Get database credentials and add to overlays/<new-experiment>/rucio/etc/.secrets/db-connstr

3. Get the required secrets (certificates and database uri) and store in overlays/<new-experiment>/rucio/etc/.secrets

4. If using a custom policy package, add those files to overlays/<new-experiment>/rucio/etc/policy-package

5. Modify overlays/<new-experiment>/Makefile

   EXPERIMENT=<new-experiment-name>
   SERVER_CHART_VERSION := <desired Rucio chart version>
   DAEMON_CHART_VERSION := <desired Rucio chart version>
   UI_CHART_VERSION := <desired Rucio chart version>

6. Modify overlays/<new-experiment>/rucio/values-rucio-*.yaml to the new experiment's configuration

   Some common changes

   • Image tags

   • URLs to services (i.e. messaging)

   • Enabling/Disabling certain Rucio components

7. On the OKD cluster, make sure the useroot service account is created

8. Enter the overlays/<new-experiment> directory and deploy with Makefile

   $ cd overlays/<new-experiment>
   $ make apply

Upgrading an Experiment's Rucio Deployment to a New Release

1. View release notes for desired release at https://github.com/rucio/rucio/releases

2. Enter the experiment's directory

   $ cd overlays/<experiment>

3. Edit overlays/<experiment>/Makefile lines related to Rucio version

   EXPERIMENT=int
   SERVER_CHART_VERSION := <desired Rucio chart version>
   DAEMON_CHART_VERSION := <desired Rucio chart version>
   UI_CHART_VERSION := <desired Rucio chart version>

4. Update image tags in each of the overlays/<experiment>/rucio/values-rucio-*.yaml

5. If necessary, upgrade databases

6. When ready, deploy with overlays/<experiment>/Makefile

   $ make apply

Upgrading Rucio Database

1. Edit util/upgrade-db-container.yaml

   - name: RUCIO_CFG_DATABASE_DEFAULT
     valueFrom:
       secretKeyRef:
         name: <secret name in cluster>
         key: db-connstr.txt

   To get the secret, run the following:

   $ kubectl get secrets | grep db-connstr
   fnal-db-connstr-<pod hash>                             Opaque                                1      16d

2. Apply util/upgrade-db-container.yaml to the experiment's cluster

   $ kubectl apply -f util/upgrade-db-container.yaml
   pod/rucio-db-upgrade created

3. Enter the pod with k9s, or

   $ kubectl exec -it rucio-db-upgrade -- /bin/bash

4. Generate the rucio.cfg and alembic.ini file and set ALEMBIC_CONFIG (these are the first few commands in docker-entrypoint.sh)

   $ python3 /usr/local/rucio/tools/merge_rucio_configs.py \
           -s /tmp/rucio.config.default.cfg $RUCIO_OVERRIDE_CONFIGS \
           --use-env \
           -d /opt/rucio/etc/rucio.cfg
   
   $ j2 /tmp/alembic.ini.j2 | sed '/^\s*$/d' > /opt/rucio/etc/alembic.ini
   
   $ export ALEMBIC_CONFIG=/opt/rucio/etc/alembic.ini

5. Modify script_location in /opt/rucio/etc/alembic.ini to the Rucio package's migrate repo

   script_location = /usr/local/lib/python3.9/site-packages/rucio/db/sqla/migrate_repo
   

6. Check the current alembic migration version

   alembic current

7. Follow instructions at https://rucio.github.io/documentation/operator/database#upgrading-and-downgrading-the-database-schema

   Ensure that in etc/alembic.ini the database connection string is is set to the same database connection string as the etc/rucio.cfg and issue the following command to verify the changes to the upgrade of the schema:

   alembic upgrade --sql $(alembic current | cut -d' ' -f1):head

   You can edit and then apply the SQL directly on your database.

   alembic upgrade head

   Rucio developers do no advise running upgrades using alembic. Another method is to generate the sql file, then apply it with a psql command

   # In pod
   alembic upgrade --sql $(alembic current | cut -d' ' -f1):head > upgrade.sql
   
   # On host with psql command
   psql> source upgrade.sql

8. Delete the pod with

   $ kubectl delete pod/rucio-db-upgrade
   pod "rucio-db-upgrade" deleted