diff --git a/docs/testbed.mdx b/docs/testbed.mdx deleted file mode 100644 index 5bf3f517b3..0000000000 --- a/docs/testbed.mdx +++ /dev/null @@ -1,845 +0,0 @@ ---- -sidebar_label: Testbed -sidebar_position: 80 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - -# Testbed - -With the OSISM Testbed, it is possible to run a full Sovereign Cloud Stack -deployment on an existing OpenStack environment such as Cleura or [REGIO.cloud](https://regio.digital). - -OSISM is the reference implementation for the Infrastructure as a Service (IaaS) layer in the -[Sovereign Cloud Stack](https://www.sovereigncloudstack.org) (SCS) project. The OSISM Testbed is therefore -used in the SCS project to test and work on the Instrastructure as a Service layer. - -The OSISM Testbed is intended as a playground. Further services and integration will -be added over time. A increasing number of best practices and experiences from the productive -deployments will be included here in the future. It will become more production-like -over time. However, at no point does it claim to represent a production setup exactly. - -## Requirements - -### Cloud access - -The usual prerequisite is to have an account on one of the supported OpenStack cloud providers. -As the OSISM Testbed also virtualizes systems itself, the OpenStack cluster should provide -the capabilities for nested virtualization. - -It is not part of this guide to describe the registration with the individual cloud -providers. Please contact the respective cloud provider for this. - -| Product | Provider | Profile name | Note | -|:--------------------|:---------------|:-----------------|:----------------------------------| -| Cleura | Cleura | `cleura` | | -| Fuga Cloud | FUGA | `fuga` | | -| HuaweiCloud | HuaweiCloud | `huaweicloud` | | -| OVH | OVH | `ovh` | | -| OpenTelekomCloud | T-Systems | `otc` | | -| pluscloud open | plusserver | `pluscloudopen` | | -| pluscloud SCS Test | plusserver | `gx-scs` | | -| REGIO.cloud | OSISM | `regiocloud` | | -| REGIO.cloud | OSISM | `regio-fast` | boot from NVMe SSD backed volumes | -| Wavestack | noris network | `wavestack` | | -| CNDS Cloud | artcodix | `artcodix` | | - -For each cloud provider listed in the table, a predefined profile is available in the -`terraform/environments` directory. This profile contains the name of the public -network, which flavors to use, etc. - -Here is an example from the profile for [REGIO.cloud](https://regio.digital). - -```text -flavor_manager = "SCS-4V-16-50" -flavor_node = "SCS-8V-32-50" -volume_type = "ssd" -image = "Ubuntu 22.04" -image_node = "Ubuntu 22.04" -public = "public" -availability_zone = "nova" -volume_availability_zone = "nova" -network_availability_zone = "nova" -``` - -### Cloud resources - -The OSISM Testbed requires at least the following project quota when using the default flavors: - -| Quantity | Resource | Note | -|:---------|:---------------------|:----------------------------------------------| -| 7 | Instances | 52 VCPUs + 208 GByte RAM (6 modes, 1 manager) | -| 18 | Volumes | 360 GByte volume storage | -| 1 | Floating IP | | -| 1 | Keypair | | -| 3 | Security group | | -| 16 | Security group rules | | -| 1 | Network | | -| 1 | Subetwork | | -| 6 | Ports | | -| 1 | Router | | - -### Software - -* `make` must be installed on the system -* Wireguard or `sshuttle` must be installed on your system for VPN access -* Python must be installed, the Python version used must be at least 3.10, otherwise - the current Ansible release cannot be used (details in the - [Ansible support matrix](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix)) -* `python3-venv` must be installed for managing Python dependencies like Ansible - -## Deployment - -This section describes step by step how to deploy the OSISM Testbed. - -1. Request access from the administrator of the respective cloud or get access to an OpenStack cloud. - -2. Clone the [osism/testbed](https://github.com/osism/testbed) repository. - - The repository can also be cloned to any other location. - - ```bash - mkdir -p ~/src/github.com/osism - git clone https://github.com/osism/testbed ~/src/github.com/osism/testbed - cd ~/src/github.com/osism/testbed - ``` - -3. Configure your cloud access profile - - The access data for the cloud provider used is stored in `terraform/clouds.yaml` and (optionally) - in `terraform/secure.yaml` (same structure, if you want to store credentials on a separate place). - - In file [terraform/clouds.yaml.sample](https://github.com/osism/testbed/blob/main/terraform/clouds.yaml.sample) - you will find examples of typical setups. Settings that are identical for all users of a cloud can be defined - centrally via the profiles of the file - [terraform/clouds-public.yaml](https://github.com/osism/testbed/blob/main/terraform/clouds-public.yaml). - You can reference these settings by using the `profile` parameter in cloud-specific - definition in `terraform/clouds.yaml`. - - The user specific settings of the `clouds.yaml` file are provided by the cloud provider. Please check the - documentation of the cloud provider you are using or their support for details. - - [REGIO.cloud](https://regio.digital) is used as an example here. The cloud name in `clouds.yaml` - and the environment name (value of `ENVIRONMENT`) are `regiocloud` in this case. It is important that - the name of the cloud in `clouds.yaml` matches the name of the environment to be used. The names must - be identical. It is currently not possible to name the cloud `regiocloud-123` in `clouds.yaml` if the - environment is `regiocloud`. - - If another cloud is used, replace `regiocloud` with the respective profile name [from the table above](#cloud-access). - - - - The use of application credentials is preferred. This way it is not necessary to store - details like username, project name or sensitive information like the password in the - `clouds.yaml` file. - - The application credentials can be found in Horizon under **Identity**. Use `OSISM Testbed` as - name and click `Create Application Credential`. - - ```yaml title="terraform/clouds.yaml" - clouds: - regiocloud: - profile: regiocloud - auth: - application_credential_id: ID - application_credential_secret: SECRET - auth_type: "v3applicationcredential" - ``` - - If you want to make use of `terraform/secure.yaml` add your application credential secret there - instead of `terraform/clouds.yaml`. - - ```yaml title="terraform/secure.yaml" - clouds: - regiocloud: - auth: - application_credential_secret: SECRET - ``` - - - - - ```yaml title="terraform/clouds.yaml" - clouds: - regiocloud: - profile: regiocloud - auth: - project_name: PROJECT - username: USERNAME - project_domain_name: DOMAIN - user_domain_name: DOMAIN - ``` - - If you want to make use of `terraform/secure.yaml` add your password there instead of `terraform/clouds.yaml`. - - ```yaml title="terraform/secure.yaml" - clouds: - regiocloud: - auth: - password: PASSWORD - ``` - - - - -4. Prepare the deployment. - - The versions of Ansible and [OpenTofu](https://opentofu.org) are managed - automatically and necessary dependencies are cloned. - - ```bash - make prepare - ``` - - If any error occurs during preparation and you want to run the preparation - again, it is important to run `make wipe-local-install` first. Otherwise the - preparation will not be redone completely and necessary parts will be missing - later on. - -5. Create the infrastructure with OpenTofu. - - ```bash - make ENVIRONMENT=regiocloud create - ``` - -6. Deploy the OSISM manager and bootstrap all nodes. - - - - ```bash - make ENVIRONMENT=regiocloud manager - ``` - - - ```bash - make ENVIRONMENT=regiocloud VERSION_MANAGER=8.0.1 manager - ``` - - - Replace the version with the version you prefer. - Check the [OSISM release notes](https://osism.tech/docs/release-notes/) - to find out what's available. - -7. After the bootstrap, you can log in to the manager via SSH. - - ```bash - make ENVIRONMENT=regiocloud login - ``` - - Yo can log in to the nodes of the cluster via the manager. - - ```bash - osism console testbed-node-0 - ``` - -8. Deploy all services. - - - - It is also possible to deploy the services step by step on the - manager. To do this, first log in to the manager with `make ENVIRONMENT=regiocloud login` - and then execute the deploy scripts one after the other. It is recommended to do this - within a screen session. - - Deploying the services takes some time and depends on how much bandwidth is available, - how the instances are equipped, etc. 90-120 minutes is not unusual when Ceph and OpenStack - are fully deployed. - - To speed up the Ansible playbooks, [ARA](https://ara.recordsansible.org) can be disabled. This - is done by executing `/opt/configuration/scripts/disable-ara.sh`. Run this script **before** the deployment scripts. - Afterwards no more logs are available in the ARA web - interface. To re-enable ARA use `/opt/configuration/scripts/enable-ara.sh`. - - There is also the option of pre-population of images with `/opt/configuration/scripts/pull-images.sh` - so that deployments do not have to be lengthy. Run this script **before** the deployment scripts. - - ```bash - /opt/configuration/scripts/deploy/001-helpers.sh - /opt/configuration/scripts/deploy/100-ceph-with-ansible.sh - /opt/configuration/scripts/deploy/200-infrastructure.sh - /opt/configuration/scripts/deploy/300-openstack.sh - /opt/configuration/scripts/deploy/400-monitoring.sh - /opt/configuration/scripts/deploy/500-kubernetes.sh - ``` - - Prepare OpenStack resources like public network, flavors and images by running - `/opt/configuration/scripts/bootstrap.sh`. Run this script **after** the deployment scripts. - - :::info - - If you only want to deploy the monitoring services with `/opt/configuration/scripts/deploy/400-monitoring.sh`, - a few dependencies must be deployed first. You can then use the monitoring services without having to install a - complete OpenStack & Ceph environment. - - ```bash - osism apply common - osism apply loadbalancer - osism apply opensearch - osism apply mariadb - ``` - - ::: - - - In this single step deployment, Ceph, OpenStack and all necessary - infrastructure services (MariaDB, RabbitMQ, ...) are deployed. - Depending on the cloud, the deployment will take some time. Up to two hours is not unusual. - - ```bash - make ENVIRONMENT=regiocloud deploy - ``` - - - -9. If you want to verify the deployment with [refstack](https://opendev.org/openinfra/refstack) run - `/opt/configuration/scripts/check.sh`. This step will take some time and is optional. - -10. The machine images required for the use of Kubernetes Cluster API and the amphora driver of OpenStack Octavia - service are not provided by default to save resources on the OSISM Testbed and improve deployment time. - These can be provisioned if required. - - ```bash - /opt/configuration/scripts/bootstrap/301-openstack-octavia-amhpora-image.sh - /opt/configuration/scripts/bootstrap/302-openstack-k8s-clusterapi-images.sh - ``` - -11. If you want you can create a test project with a test user after login. It also - creates an instance with a volume attached to a network with a router. This step is optional. - - ```bash - osism apply --environment openstack test - ``` - -12. When the OSISM Testbed is no longer needed, it can be deleted. - - ```bash - make ENVIRONMENT=regiocloud clean - ``` - -## Usage - -Deployment must be completed at this point. - -### Custom CA - -The OSISM Testbed deployment currently uses hostnames in the domain `testbed.osism.xyz`. This is a real domain -and we provide the DNS records matching the addresses used in the OSISM Testbed, so that once you connect to your testbed via a direct -link or Wireguard, you can access hosts and servers by their hostname (e.g. `ssh testbed-manager.testbed.osism.xyz`). - -We also provide a wildcard TLS certificate signed by a custom CA for `testbed.osism.xyz` and `*.testbed.osism.xyz`. -This CA is always used for each testbed. The CA is not regenerated and it is not planned to change this for the next 10 years. - -In order for these certificates to be recognized locally as valid, the CA -[environments/kolla/certificates/ca/testbed.crt](https://raw.githubusercontent.com/osism/testbed/main/environments/kolla/certificates/ca/testbed.crt) -must be imported locally. - -### VPN access - -#### Wireguard - -Install wireguard on your workstation, if you have not done this before. For instructions how to do -it on your workstation, please have a look on the documentation of your used distribution. The -wireguard documentation you will find [here](https://www.wireguard.com). - -Start the wireguard tunnel. -(Press CTRL+c to keep the tunnel running forever. The make target also launches a browser tab with references to all services) - -```bash -make vpn-wireguard ENVIRONMENT=regiocloud -``` - -If you want to connect to the OSISM Testbed from multiple clients, change the client IP -address in the downloaded configuration file to be different on each client. - -If you only want to download the Wireguard configuration, you can use the `vpn-wireguard-config` -target. The configuration is then available in the file `wg-testbed-regiocloud.conf`, for example. - -```bash -make vpn-wireguard-config ENVIRONMENT=regiocloud -``` - -#### sshuttle - -If you do not want to use Wireguard you can also work with [sshuttle](https://github.com/sshuttle/sshuttle). - -```bash -make vpn-sshuttle ENVIRONMENT=regiocloud -killall sshuttle -``` - -### Static entries in `/etc/hosts` - -If you are unable to access the following domains, you can customize your local `/etc/hosts` -with the following static entries. This may be necessary, for example, if you use Pi-hole and -all DNS entries from a public DNS with a non-public IP address are filtered. - -```bash -# OSISM Testbed hosts -192.168.16.5 ara.testbed.osism.xyz ara -192.168.16.5 cgit.testbed.osism.xyz cgit -192.168.16.5 flower.testbed.osism.xyz flower -192.168.16.5 homer.testbed.osism.xyz homer -192.168.16.5 netbox.testbed.osism.xyz netbox -192.168.16.5 testbed-manager.testbed.osism.xyz testbed-manager -192.168.16.5 nexus.testbed.osism.xyz nexus -192.168.16.5 phpmyadmin.testbed.osism.xyz phpmyadmin -192.168.16.9 api-int.testbed.osism.xyz api-int -192.168.16.10 testbed-node-0.testbed.osism.xyz testbed-node-0 -192.168.16.11 testbed-node-1.testbed.osism.xyz testbed-node-1 -192.168.16.12 testbed-node-2.testbed.osism.xyz testbed-node-2 -192.168.16.13 testbed-node-3.testbed.osism.xyz testbed-node-3 -192.168.16.14 testbed-node-4.testbed.osism.xyz testbed-node-4 -192.168.16.15 testbed-node-5.testbed.osism.xyz testbed-node-5 -192.168.16.16 testbed-node-6.testbed.osism.xyz testbed-node-6 -192.168.16.17 testbed-node-7.testbed.osism.xyz testbed-node-7 -192.168.16.18 testbed-node-8.testbed.osism.xyz testbed-node-8 -192.168.16.19 testbed-node-9.testbed.osism.xyz testbed-node-9 -192.168.16.100 keycloak.testbed.osism.xyz keycloak -192.168.16.254 api.testbed.osism.xyz api -``` - -### Webinterfaces - -All SSL enabled services within the OSISM Testbed use certs which are signed by the self-signed -[OSISM Testbed CA](https://raw.githubusercontent.com/osism/testbed/main/environments/kolla/certificates/ca/testbed.crt) -(Download the file and import it as certification authority to your browser). - -If you want to access the services please choose the URL from the following table. - -| Name | URL | Username | Password | Note | -|:-------------------------|:-----------------------------------------------|:-------------|:-------------|:----------------| -| ARA | https://ara.testbed.osism.xyz | ara | password | | -| Ceph | https://api-int.testbed.osism.xyz:8140 | admin | password | | -| Flower | https://flower.testbed.osism.xyz | | | | -| Grafana | https://api-int.testbed.osism.xyz:3000 | admin | password | | -| HAProxy (testbed-node-0) | http://testbed-node-0.testbed.osism.xyz:1984 | openstack | password | | -| HAProxy (testbed-node-1) | http://testbed-node-1.testbed.osism.xyz:1984 | openstack | password | | -| HAProxy (testbed-node-2) | http://testbed-node-2.testbed.osism.xyz:1984 | openstack | password | | -| Homer | https://homer.testbed.osism.xyz | | | | -| Horizon (via Keystone) | https://api.testbed.osism.xyz | admin | password | domain: default | -| Horizon (via Keystone) | https://api.testbed.osism.xyz | test | test | domain: test | -| Netbox | https://netbox.testbed.osism.xyz | admin | password | | -| Netdata | http://testbed-manager.testbed.osism.xyz:19999 | | | | -| Nexus | https://nexus.testbed.osism.xyz | admin | password | | -| OpenSearch Dashboards | https://api.testbed.osism.xyz:5601 | opensearch | password | | -| Prometheus | https://api-int.testbed.osism.xyz:9091 | admin | password | | -| RabbitMQ | https://api-int.testbed.osism.xyz:15672 | openstack | password | | -| phpMyAdmin | https://phpmyadmin.testbed.osism.xyz | root | password | | - -#### Usage of the OpenStack CLI - -The `environments/openstack` folder contains the needed files for the openstack client: - -```bash -cd environments/openstack -export OS_CLOUD= # i.e. admin -openstack floating ip list -``` - -## Advanced Usage - -### External API - -It is possible to provide the OpenStack APIs and the OpenStack Dashboard via the manager's public IP address. -This is not enabled by default, with the exception of the OTC profile. To provide the OpenStack APIs and the -OpenStack dashboard via the public IP address of the manager, the following changes are necessary in the -`terraform/environments/regiocloud.tfvars` file. If a cloud other than the REGIO.cloud is used, the profile -of the other cloud is changed accordingly. - -1. Add the customisation `external_api`. This customisation makes sure that the required security group rules - are created for the various OpenStack APIs and the OpenStack dashboard. - - ``` - # customisation:external_api - ``` - -2. Set parameter `external_api` to `true`. This makes sure that all necessary changes are made in the configuration - repository when the manager service is deployed. It is correct that this is added as a comment. - - ``` - external_api = true - ``` - -3. After the deployment of the manager service and the OpenStack services, the OpenStack APIs and the OpenStack - dashboard can be reached via a DNS name. The service [traefik.me](https://traefik.me) is used for the DNS record. - Run the following two commands on the manager node to get the DNS record. - - ``` - $ source /opt/manager-vars.sh - $ echo "api-${MANAGER_PUBLIC_IP_ADDRESS//./-}.traefik.me" - api-80-158-46-219.traefik.me - ``` - -### Change versions - -1. Go to `/opt/configuration` on `testbed-manager` -2. Run `./scripts/set-openstack-version.sh 2024.2` to set the OpenStack version to `2024.2` -3. Run `./scripts/set-ceph-version.sh reef` to set the Ceph version to `reef` -4. Run `osism update manager` to update the manager service - -### Deploy services - -| Script | Description | -|:------------------------------------------------------------------|:----------------------------| -| `/opt/configuration/scripts/deploy/000-manager.sh` | | -| `/opt/configuration/scripts/deploy/001-helpers.sh` | | -| `/opt/configuration/scripts/deploy/100-ceph-with-ansible.sh` | Ceph with Ansible | -| `/opt/configuration/scripts/deploy/100-ceph-with-rook.sh` | Ceph with Rook | -| `/opt/configuration/scripts/deploy/200-infrastructure.sh` | | -| `/opt/configuration/scripts/deploy/300-openstack.sh` | | -| `/opt/configuration/scripts/deploy/310-openstack-extended.sh` | | -| `/opt/configuration/scripts/deploy/400-monitoring.sh` | | -| `/opt/configuration/scripts/deploy/500-kubernetes.sh` | | -| `/opt/configuration/scripts/deploy/510-clusterapi.sh` | | - -### Upgrade services - -| Script | Description | -|:--------------------------------------------------------------------------|:----------------------------| -| `/opt/configuration/scripts/upgrade/100-ceph-with-ansible.sh` | Ceph with Ansible | -| `/opt/configuration/scripts/upgrade/100-ceph-with-rook.sh` | Ceph with Rook | -| `/opt/configuration/scripts/upgrade/200-infrastructure.sh` | | -| `/opt/configuration/scripts/upgrade/300-openstack.sh` | | -| `/opt/configuration/scripts/upgrade/310-openstack-extended.sh` | | -| `/opt/configuration/scripts/upgrade/400-monitoring.sh` | | -| `/opt/configuration/scripts/upgrade/500-kubernetes.sh` | | -| `/opt/configuration/scripts/upgrade/510-clusterapi.sh` | | - -### Add new OSD in Ceph - -In the testbed, three volumes per node are provided for use by Ceph by default. Two of -these devices are used as OSDs during the initial deployment. The third device is intended -for testing the addition of a further OSD to the Ceph cluster. - -1. Add `sdd` to `ceph_osd_devices` in `/opt/configuration/inventory/host_vars/testbed-node-0/ceph-lvm-configuration.yml`. - The following content is an example, the IDs look different everywhere. Do not copy 1:1 - but only add `sdd` to the file. - -```yaml ---- -# -# This is Ceph LVM configuration for testbed-node-0 -# generated by ceph-configure-lvm-volumes playbook. -# -ceph_osd_devices: - sdb: - osd_lvm_uuid: 95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a - sdc: - osd_lvm_uuid: 29899765-42bf-557b-ae9c-5c7c984b2243 - sdd: -lvm_volumes: -- data: osd-block-95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a - data_vg: ceph-95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a -- data: osd-block-29899765-42bf-557b-ae9c-5c7c984b2243 - data_vg: ceph-29899765-42bf-557b-ae9c-5c7c984b2243 -``` - -2. Run `osism apply ceph-configure-lvm-volumes -l testbed-node-0` - -3. Run `cp /tmp/testbed-node-0-ceph-lvm-configuration.yml /opt/configuration/inventory/host_vars/testbed-node-0/ceph-lvm-configuration.yml`. - -4. Run `osism reconciler sync` - -5. Run `osism apply ceph-create-lvm-devices -l testbed-node-0` - -6. Run `osism apply ceph-osds -l testbed-node-0 -e ceph_handler_osds_restart=false` - -7. Check the OSD tree - - ``` - $ ceph osd tree - ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF - -1 0.13640 root default - -3 0.05846 host testbed-node-0 - 2 hdd 0.01949 osd.2 up 1.00000 1.00000 - 4 hdd 0.01949 osd.4 up 1.00000 1.00000 - 6 hdd 0.01949 osd.6 up 1.00000 1.00000 - -5 0.03897 host testbed-node-1 - 0 hdd 0.01949 osd.0 up 1.00000 1.00000 - 5 hdd 0.01949 osd.5 up 1.00000 1.00000 - -7 0.03897 host testbed-node-2 - 1 hdd 0.01949 osd.1 up 1.00000 1.00000 - 3 hdd 0.01949 osd.3 up 1.00000 1.00000 - ``` - -### Ceph via Rook (technical preview) - -Please have a look at [Deploy Guide - Services - Rook](guides/deploy-guide/services/ceph/rook.md) and [Configuration Guide - Rook](guides/configuration-guide/ceph/rook.md) for details on how to configure Rook. - -To deploy this in the testbed, you can use an environment variable in your make target. - -``` -make CEPH_STACK=rook manager -make CEPH_STACK=rook ceph -``` - -This will make sure `/opt/manager-vars.sh` gets `CEPH_STACK=rook` set which is later being used by: - -``` -/opt/configuration/scripts/deploy-services.sh -/opt/configuration/scripts/deploy-ceph.sh -/opt/configuration/scripts/upgrade/100-ceph-with-rook.sh -``` - -## Using testbed for OpenStack development - -Testbed may be used for doing development on OpenStack services through [`kolla_dev_mode`](https://docs.openstack.org/kolla-ansible/latest/contributor/kolla-for-openstack-development.html). -`kolla_dev_mode` may be activated for all supported service by adding - -``` -kolla_dev_mode: true -``` - -to `environments/kolla/configuration.yml`. This will check out the `git` repositories of all supported and enabled OpenStack services under `/opt/stack` and bind mount them into the appropriate containers. -Since this will fetch a lot of repositories it is advisable to enable this only selectively for the services you are going to work on. You can do so by adding a boolean variable to `environments/kolla/configuration.yml` consisting of the service name and the suffix `_dev_mode`, e.g.: - -``` -cinder_dev_mode: true -``` - -You may customise the used `git` repository by adding `kolla_repos_git` to `environments/kolla/configuration.yml` to specify a common root for the repositories of all services, which will be completed by the service's name, e.g.: - -``` -kolla_repos_git: "https://github.com/openstack" -``` - -will pull services from their `github` mirror. E.g. pull nova from https://github.com/openstack/nova -The complete repository of a single service may be changed by adding a variable consisting of the service name and the suffix `_git_repository`, e.g. - -``` -cinder_git_repository: "https://github.com/myorg/my_custom_cinder_fork" -``` - -You may Specify the `git` reference globally by setting - -``` -kolla_source_version: my_feature_branch -``` - -in `environments/kolla/configuration.yml` or for specific services via service name and suffix `_source_version`, e.g. - -``` -cinder_source_version: my_cinder_feature -``` - -In order to update the `git` repositories on `kolla-ansible` deployments set either - -``` -kolla_dev_repos_pull: true -``` - -or, e.g. - -``` -cinder_dev_repos_pull: true -``` - -in `environments/kolla/configuration.yml` to set the setting for all repositories or a service specific one. - -Thus, as an example for a development workflow: -* Create a `git` mirror of the OpenStack service you want to work on -* In `environments/kolla/configuration.yml` - * Set the corresponding `_dev_mode` variable to `true` - * Set the corresponding `_git_repository` variable to your development mirror created above - * Set the corresponding `_source_version` variable to a branch name you intend to push your changes to - * Set the corresponding `_dev_repos_pull` variable to `true` -* Commit your changes and push them to the specified branch in the specified mirror -* Run `osism apply ` on the manager to checkout the code on every testbed node -* Restart the service containers to actually pick up the new code - -## Troubleshooting - -### Ansible errors - -Ansible errors that have something to do with undefined variables (e.g. `AnsibleUndefined`) are most likely due to cached -facts that are no longer valid. The facts can be updated by running `osism apply facts`. - -### Unsupported locale setting - -```bash -$ make prepare -ansible-playbook -i localhost, ansible/check-local-versions.yml -ERROR: Ansible could not initialize the preferred locale: unsupported locale setting -make: *** [prepare] Error 1 -``` - -To solve the problem you have to modify the `Makefile`. Change the 1st line as follows. - -```bash -export LC_ALL=en_US.UTF-8 -``` - -To find out the locale used on the system `printenv` can be used. - -```bash -$ printenv | grep -i lang|locale -LANG="en_US.UTF-8" -LC_COLLATE="en_US.UTF-8" -LC_CTYPE="UTF-8" -LC_MESSAGES="en_US.UTF-8" -LC_MONETARY="en_US.UTF-8" -LC_NUMERIC="en_US.UTF-8" -LC_TIME="en_US.UTF-8" -LC_ALL= -``` - -## Appendix - -### Configuration - -This section describes how to configure and customise the OSISM Testbed. - -#### Variables - -The defaults for the OpenTofu variables are intended for [REGIO.cloud](https://regio.digital). - -| Variable | Default | Note | -|:--------------------------|:-------------------------|:-----------------------------------------| -| availability_zone | `nova` | | -| ceph_version | `quincy` | | -| cloud_provider | `regiocloud` | | -| configuration_version | `main` | | -| deploy_monitoring | `false` | | -| dns_nameservers | `["8.8.8.8", "9.9.9.9"]` | | -| enable_config_drive | `true` | | -| external_api | `false` | | -| flavor_manager | `SCS-4V-16-50` | | -| flavor_node | `SCS-8V-32-50` | | -| image | `Ubuntu 22.04` | Only Ubuntu 22.04 is currently supported | -| image_node | `Ubuntu 22.04` | Only Ubuntu 22.04 is currently supported | -| keypair | `testbed` | | -| manager_version | `latest` | | -| network_availability_zone | `nova` | | -| number_of_nodes | `3` | | -| number_of_volumes | `3` | | -| openstack_version | `2023.2` | | -| prefix | `testbed` | | -| public | `external` | | -| refstack | `false` | | -| volume_availability_zone | `nova` | | -| volume_size_base | `30` | | -| volume_size_storage | `10` | | -| volume_type | `__DEFAULT__` | | - -#### Overrides - -| Name | Description | -|:-------------------------------------------|:-----------------| -| `manager_boot_from_image` | | -| `manager_boot_from_volume` | | -| `neutron_availability_zone_hints_network` | | -| `neutron_availability_zone_hints_router` | | -| `neutron_router_enable_snat` | | -| `nodes_boot_from_image` | | -| `nodes_boot_from_volume` | | -| `nodes_use_ephemeral_storage` | | - -#### Customisations - -| Name | Description | -|:----------------------|:-----------------| -| `access_floatingip` | | -| `access_ipv4` | | -| `access_ipv6` | | -| `default` | | -| `external_api` | | -| `neutron_floatingip` | | - - -### Notes - -* The configuration is intentionally kept quite static. Please create no PRs to make the configuration more flexible/dynamic. -* The OSISM documentation uses hostnames, examples, addresses etc. from OSISM Testbed. -* The third volume (`/dev/sdd`) is not enabled for Ceph by default. This is to test the scaling of Ceph. -* The manager is used as pull through cache for Docker images and Ubuntu packages. This reduces the amount of traffic consumed. - -### Supported releases - -The following stable Ceph and OpenStack releases are supported. - -The deployment of Ceph is based on [ceph-ansible](https://docs.ceph.com/ceph-ansible/). - -* Ceph Quincy (**default**) -* Ceph Reef -* Ceph Squid - -The deployment of OpenStack is based on [kolla-ansible](https://docs.openstack.org/kolla-ansible/latest/). - -* OpenStack 2023.1 -* OpenStack 2023.2 -* OpenStack 2024.1 (**default**) - -The deployment of Kubernetes is based on [k3s-ansible](https://github.com/techno-tim/k3s-ansible). - -* Kubernetes v1.30 (**default**) - -### Included services - -The following services can currently be used with the OSISM Testbed without further adjustments. - -#### Infrastructure - -* Ceph -* Cluster API Management Cluster -* Fluentd -* Gnocchi -* Grafana -* Haproxy -* Keepalived -* Kubernetes -* Mariadb -* Memcached -* Netbox -* Netdata -* Opensearch -* Openvswitch -* Prometheus exporters -* Rabbitmq -* Redis - -#### OpenStack - -* Barbican -* Ceilometer -* Cinder -* Designate -* Glance -* Horizon -* Ironic -* Keystone -* Magnum -* Manila -* Neutron -* Nova (with Libvirt/KVM) -* Octavia -* Skyline - -### Makefile reference - -```bash -$ make help - -Usage: - make - help Display this help. - clean Destroy infrastructure with OpenTofu. - wipe-local-install Wipe the software dependencies in `venv`. - create Create required infrastructure with OpenTofu. - login Log in on the manager. - vpn-wireguard Establish a wireguard vpn tunnel. - vpn-sshuttle Establish a sshuttle vpn tunnel. - bootstrap Bootstrap everything. - manager Deploy only the manager service. - identity Deploy only identity services. - ceph Deploy only ceph services. - deploy Deploy everything and then check it. - prepare Run local preperations. - deps Install software preconditions to `venv`. - -$ make -``` - -### CI jobs - -You can inspect the [results of the daily zuul jobs](https://zuul.services.betacloud.xyz/t/osism/builds?project=osism%2Ftestbed&skip=0). diff --git a/docs/testbed/appendix.mdx b/docs/testbed/appendix.mdx new file mode 100644 index 0000000000..5b14f2f41a --- /dev/null +++ b/docs/testbed/appendix.mdx @@ -0,0 +1,164 @@ +--- +sidebar_label: Appendix +sidebar_position: 5 +--- + +# Appendix + +## Configuration + +This section describes how to configure and customise the OSISM Testbed. + +### Variables + +The defaults for the OpenTofu variables are intended for [REGIO.cloud](https://regio.digital). + +| Variable | Default | Note | +|:--------------------------|:-------------------------|:-----------------------------------------| +| availability_zone | `nova` | | +| ceph_version | `quincy` | | +| cloud_provider | `regiocloud` | | +| configuration_version | `main` | | +| deploy_monitoring | `false` | | +| dns_nameservers | `["8.8.8.8", "9.9.9.9"]` | | +| enable_config_drive | `true` | | +| external_api | `false` | | +| flavor_manager | `SCS-4V-16-50` | | +| flavor_node | `SCS-8V-32-50` | | +| image | `Ubuntu 22.04` | Only Ubuntu 22.04 is currently supported | +| image_node | `Ubuntu 22.04` | Only Ubuntu 22.04 is currently supported | +| keypair | `testbed` | | +| manager_version | `latest` | | +| network_availability_zone | `nova` | | +| number_of_nodes | `3` | | +| number_of_volumes | `3` | | +| openstack_version | `2023.2` | | +| prefix | `testbed` | | +| public | `external` | | +| refstack | `false` | | +| volume_availability_zone | `nova` | | +| volume_size_base | `30` | | +| volume_size_storage | `10` | | +| volume_type | `__DEFAULT__` | | + +### Overrides + +| Name | Description | +|:-------------------------------------------|:-----------------| +| `manager_boot_from_image` | | +| `manager_boot_from_volume` | | +| `neutron_availability_zone_hints_network` | | +| `neutron_availability_zone_hints_router` | | +| `neutron_router_enable_snat` | | +| `nodes_boot_from_image` | | +| `nodes_boot_from_volume` | | +| `nodes_use_ephemeral_storage` | | + +### Customisations + +| Name | Description | +|:----------------------|:-----------------| +| `access_floatingip` | | +| `access_ipv4` | | +| `access_ipv6` | | +| `default` | | +| `external_api` | | +| `neutron_floatingip` | | + + +## Notes + +* The configuration is intentionally kept quite static. Please create no PRs to make the configuration more flexible/dynamic. +* The OSISM documentation uses hostnames, examples, addresses etc. from OSISM Testbed. +* The third volume (`/dev/sdd`) is not enabled for Ceph by default. This is to test the scaling of Ceph. +* The manager is used as pull through cache for Docker images and Ubuntu packages. This reduces the amount of traffic consumed. + +## Supported releases + +The following stable Ceph and OpenStack releases are supported. + +The deployment of Ceph is based on [ceph-ansible](https://docs.ceph.com/ceph-ansible/). + +* Ceph Quincy (**default**) +* Ceph Reef +* Ceph Squid + +The deployment of OpenStack is based on [kolla-ansible](https://docs.openstack.org/kolla-ansible/latest/). + +* OpenStack 2023.1 +* OpenStack 2023.2 +* OpenStack 2024.1 (**default**) + +The deployment of Kubernetes is based on [k3s-ansible](https://github.com/techno-tim/k3s-ansible). + +* Kubernetes v1.30 (**default**) + +## Included services + +The following services can currently be used with the OSISM Testbed without further adjustments. + +### Infrastructure + +* Ceph +* Cluster API Management Cluster +* Fluentd +* Gnocchi +* Grafana +* Haproxy +* Keepalived +* Kubernetes +* Mariadb +* Memcached +* Netbox +* Netdata +* Opensearch +* Openvswitch +* Prometheus exporters +* Rabbitmq +* Redis + +### OpenStack + +* Barbican +* Ceilometer +* Cinder +* Designate +* Glance +* Horizon +* Ironic +* Keystone +* Magnum +* Manila +* Neutron +* Nova (with Libvirt/KVM) +* Octavia +* Skyline + +## Makefile reference + +```bash +$ make help + +Usage: + make + help Display this help. + clean Destroy infrastructure with OpenTofu. + wipe-local-install Wipe the software dependencies in `venv`. + create Create required infrastructure with OpenTofu. + login Log in on the manager. + vpn-wireguard Establish a wireguard vpn tunnel. + vpn-sshuttle Establish a sshuttle vpn tunnel. + bootstrap Bootstrap everything. + manager Deploy only the manager service. + identity Deploy only identity services. + ceph Deploy only ceph services. + deploy Deploy everything and then check it. + prepare Run local preperations. + deps Install software preconditions to `venv`. + +$ make +``` + +## CI jobs + +You can inspect the [results of the daily zuul jobs](https://zuul.services.betacloud.xyz/t/osism/builds?project=osism%2Ftestbed&skip=0). \ No newline at end of file diff --git a/docs/testbed/deployment.mdx b/docs/testbed/deployment.mdx new file mode 100644 index 0000000000..8e5a5d62f2 --- /dev/null +++ b/docs/testbed/deployment.mdx @@ -0,0 +1,239 @@ +--- +sidebar_label: Deployement +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Deployment + +This section describes step by step how to deploy the OSISM Testbed. + +## Initializing + +1. Request access from the administrator of the respective cloud or get access to an OpenStack cloud. + +2. Clone the [osism/testbed](https://github.com/osism/testbed) repository. + + The repository can also be cloned to any other location. + + ```bash + mkdir -p ~/src/github.com/osism + git clone https://github.com/osism/testbed ~/src/github.com/osism/testbed + cd ~/src/github.com/osism/testbed + ``` + +3. Configure your cloud access profile + + The access data for the cloud provider used is stored in `terraform/clouds.yaml` and (optionally) in `terraform/secure.yaml` (same structure, if you want to store credentials in a separate place). + + In file [terraform/clouds.yaml.sample](https://github.com/osism/testbed/blob/main/terraform/clouds.yaml.sample) you will find examples of typical setups. Settings that are identical for all users of a cloud can be defined centrally via the profiles of the file [terraform/clouds-public.yaml](https://github.com/osism/testbed/blob/main/terraform/clouds-public.yaml). You can reference these settings by using the `profile` parameter in the cloud-specific definition in `terraform/clouds.yaml`. + + The user-specific settings of the `clouds.yaml` file are provided by the cloud provider. Please check the + documentation of the cloud provider you are using or their support for details. + + [REGIO.cloud](https://regio.digital) is used as an example here. The cloud name in `clouds.yaml` + and the environment name (value of `ENVIRONMENT`) are `regiocloud` in this case. It is important that + the name of the cloud in `clouds.yaml` matches the name of the environment to be used. The names must + be identical. It is currently not possible to name the cloud `regiocloud-123` in `clouds.yaml` if the + environment is `regiocloud`. + + If another cloud is used, replace `regiocloud` with the respective profile name [from the table above](#cloud-access). + + + + The use of application credentials is preferred. This way it is not necessary to store + details like username, project name or sensitive information like the password in the + `clouds.yaml` file. + + The application credentials can be found in Horizon under **Identity**. Use `OSISM Testbed` as + name and click `Create Application Credential`. + + ```yaml title="terraform/clouds.yaml" + clouds: + regiocloud: + profile: regiocloud + auth: + application_credential_id: ID + application_credential_secret: SECRET + auth_type: "v3applicationcredential" + ``` + + If you want to make use of `terraform/secure.yaml` add your application credential secret there + instead of `terraform/clouds.yaml`. + + ```yaml title="terraform/secure.yaml" + clouds: + regiocloud: + auth: + application_credential_secret: SECRET + ``` + + + + + ```yaml title="terraform/clouds.yaml" + clouds: + regiocloud: + profile: regiocloud + auth: + project_name: PROJECT + username: USERNAME + project_domain_name: DOMAIN + user_domain_name: DOMAIN + ``` + + If you want to make use of `terraform/secure.yaml` add your password there instead of `terraform/clouds.yaml`. + + ```yaml title="terraform/secure.yaml" + clouds: + regiocloud: + auth: + password: PASSWORD + ``` + + + + +4. Prepare for the deployment. + + The versions of Ansible and [OpenTofu](https://opentofu.org) are managed + automatically, and necessary dependencies are cloned. + + ```bash + make prepare + ``` + + :::tip + + If any error occurs during preparation, and you want to run the preparation + again, it is important to run `make wipe-local-install` first. Otherwise, the + preparation will not be redone completely, and necessary parts will be missing + later on. + + ::: + +## Start deployement + +1. Create the infrastructure with OpenTofu. + + ```bash + make ENVIRONMENT=regiocloud create + ``` + +2. Deploy the OSISM manager and bootstrap all nodes. + + + + ```bash + make ENVIRONMENT=regiocloud manager + ``` + + + ```bash + make ENVIRONMENT=regiocloud VERSION_MANAGER=8.0.1 manager + ``` + + + Replace the version with the version you prefer. + Check the [OSISM release notes](https://osism.tech/docs/release-notes/) + to find out what's available. + +3. After the bootstrap, you can log in to the manager via SSH. + + ```bash + make ENVIRONMENT=regiocloud login + ``` + + Yo can log in to the nodes of the cluster via the manager. + + ```bash + osism console testbed-node-0 + ``` + +4. Deploy all services. + + + + It is also possible to deploy the services step by step on the + manager. To do this, first log in to the manager with `make ENVIRONMENT=regiocloud login` + and then execute the deployment scripts one after the other. It is recommended to do this + within a screen session. + + Deploying the services takes some time and depends on how much bandwidth is available, + how the instances are equipped, etc. 90–120 minutes is not unusual when Ceph and OpenStack + are fully deployed. + + To speed up the Ansible playbooks, [ARA](https://ara.recordsansible.org) can be disabled. This + is done by executing `/opt/configuration/scripts/disable-ara.sh`. Run this script **before** the deployment scripts. + Afterwards, no more logs are available in the ARA web + interface. To re-enable ARA use `/opt/configuration/scripts/enable-ara.sh`. + + There is also the option of pre-population of images with `/opt/configuration/scripts/pull-images.sh` + so that deployments do not have to be lengthy. Run this script **before** the deployment scripts. + + ```bash + /opt/configuration/scripts/deploy/001-helpers.sh + /opt/configuration/scripts/deploy/100-ceph-with-ansible.sh + /opt/configuration/scripts/deploy/200-infrastructure.sh + /opt/configuration/scripts/deploy/300-openstack.sh + /opt/configuration/scripts/deploy/400-monitoring.sh + /opt/configuration/scripts/deploy/500-kubernetes.sh + ``` + + Prepare OpenStack resources like public network, flavors and images by running + `/opt/configuration/scripts/bootstrap.sh`. Run this script **after** the deployment scripts. + + :::info + + If you only want to deploy the monitoring services with `/opt/configuration/scripts/deploy/400-monitoring.sh`, + a few dependencies must be deployed first. You can then use the monitoring services without having to install a + complete OpenStack & Ceph environment. + + ```bash + osism apply common + osism apply loadbalancer + osism apply opensearch + osism apply mariadb + ``` + + ::: + + + In this single-step deployment, Ceph, OpenStack and all necessary + infrastructure services (MariaDB, RabbitMQ, ...) are deployed. + Depending on the cloud, the deployment will take some time. Up to two hours is not unusual. + + ```bash + make ENVIRONMENT=regiocloud deploy + ``` + + + +## Verify and adjust deployment + +1. If you want to verify the deployment with [refstack](https://opendev.org/openinfra/refstack) run + `/opt/configuration/scripts/check.sh`. This step will take some time and is optional. + +2. The machine images required for the use of Kubernetes Cluster API and the amphora driver of OpenStack Octavia + service are not provided by default to save resources on the OSISM Testbed and improve deployment time. + These can be provisioned if required. + + ```bash + /opt/configuration/scripts/bootstrap/301-openstack-octavia-amhpora-image.sh + /opt/configuration/scripts/bootstrap/302-openstack-k8s-clusterapi-images.sh + ``` + +3. If you want, you can create a test project with a test user after login. It also + creates an instance with a volume attached to a network with a router. This step is optional. + + ```bash + osism apply --environment openstack test + ``` + +4. When the OSISM Testbed is no longer needed, it can be deleted. + + ```bash + make ENVIRONMENT=regiocloud clean + ``` \ No newline at end of file diff --git a/docs/testbed/index.md b/docs/testbed/index.md new file mode 100644 index 0000000000..ac9843b568 --- /dev/null +++ b/docs/testbed/index.md @@ -0,0 +1,24 @@ +--- +sidebar_label: Testbed +sidebar_position: 80 +--- + +# Testbed + +With the OSISM Testbed, it is possible to run a full Sovereign Cloud Stack +deployment on an existing OpenStack environment such as Cleura or [REGIO.cloud](https://regio.digital). + +OSISM is the reference implementation for the Infrastructure as a Service (IaaS) layer in the +[Sovereign Cloud Stack](https://www.sovereigncloudstack.org) (SCS) project. The OSISM Testbed is therefore +used in the SCS project to test and work on the Infrastructure as a Service layer. + +To get started with Testbed we first need to [prepare our environment](prerequisites.mdx). + +:::info + +The OSISM Testbed is intended as a playground. Further services and integration will +be added over time. An increasing number of best practices and experiences from the productive +deployments will be included here in the future. It will become more production-like +over time. However, at no point does it claim to represent a production setup exactly. + +::: \ No newline at end of file diff --git a/docs/testbed/prerequisites.mdx b/docs/testbed/prerequisites.mdx new file mode 100644 index 0000000000..dd64fb726c --- /dev/null +++ b/docs/testbed/prerequisites.mdx @@ -0,0 +1,73 @@ +--- +sidebar_label: Getting started +sidebar_position: 1 +--- + +# Prerequisites + +## Cloud access + +The usual prerequisite is to have an account on one of the supported OpenStack cloud providers. +As the OSISM Testbed also virtualizes systems itself, the OpenStack cluster should provide +the capabilities for nested virtualization. + +It is not part of this guide to describe the registration with the individual cloud +providers. Please contact the respective cloud provider for this. + +| Provider | Product | Profile name | Note | +|:----------------|:--------------------|:-----------------|:-----------------------------------| +| OSISM | REGIO.cloud | `regiocloud` | | +| OSISM | REGIO.cloud | `regio-fast` | boot from NVMe SSD backed volumes | +| Cleura | Cleura | `cleura` | | +| FUGA | Fuga Cloud | `fuga` | | +| HuaweiCloud | HuaweiCloud | `huaweicloud` | | +| OVH | OVH | `ovh` | | +| T-Systems | OpenTelekomCloud | `otc` | | +| plusserver | pluscloud open | `pluscloudopen` | | +| plusserver | pluscloud SCS Test | `gx-scs` | | +| noris network | Wavestack | `wavestack` | | +| artcodix | CNDS Cloud | `artcodix` | | + +For each cloud provider listed in the table, a predefined profile is available in the +`terraform/environments` directory. This profile contains the name of the public +network, which flavors to use, etc. + +Here is an example from the profile for [REGIO.cloud](https://regio.digital). + +```text +flavor_manager = "SCS-4V-16-50" +flavor_node = "SCS-8V-32-50" +volume_type = "ssd" +image = "Ubuntu 22.04" +image_node = "Ubuntu 22.04" +public = "public" +availability_zone = "nova" +volume_availability_zone = "nova" +network_availability_zone = "nova" +``` + +## Cloud resources + +The OSISM Testbed requires at least the following project quota when using the default flavors: + +| Resource | Quantity | Note | +|:----------------------|:----------|:----------------------------------------------| +| Instances | 7 | 52 VCPUs + 208 GByte RAM (6 modes, 1 manager) | +| Volumes | 18 | 360 GByte volume storage | +| Floating IP | 1 | | +| Keypair | 1 | | +| Security group | 3 | | +| Security group rules | 16 | | +| Network | 1 | | +| Subnetwork | 1 | | +| Ports | 6 | | +| Router | 1 | | + +## Software + +* `make` must be installed on the system +* [Wireguard](https://wireguard.com) or [`sshuttle`](https://github.com/sshuttle/sshuttle) must be installed on your system for VPN access +* Python must be installed, the Python version used must be at least 3.10, otherwise + the current Ansible release cannot be used (more details in the + [Ansible support matrix](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix)) +* `python3-venv` must be installed for managing Python dependencies like Ansible \ No newline at end of file diff --git a/docs/testbed/troubleshooting.mdx b/docs/testbed/troubleshooting.mdx new file mode 100644 index 0000000000..02dcec2222 --- /dev/null +++ b/docs/testbed/troubleshooting.mdx @@ -0,0 +1,70 @@ +--- +sidebar_label: Troubleshooting +sidebar_position: 4 +--- + +# Troubleshooting + +## Ansible errors + +Ansible errors that have something to do with undefined variables (e.g. `AnsibleUndefined`) are most likely due to cached facts that are no longer valid. The facts can be updated by running `osism apply facts`. + +## Unsupported locale setting + +```bash +$ make prepare +ansible-playbook -i localhost, ansible/check-local-versions.yml +ERROR: Ansible could not initialize the preferred locale: unsupported locale setting +make: *** [prepare] Error 1 +``` + +To solve the problem you have to modify the `Makefile`. Change the 1st line as follows. + +```bash +export LC_ALL=en_US.UTF-8 +``` + +To find out the locale used on the system `printenv` can be used. + +```bash +$ printenv | grep -i lang|locale +LANG="en_US.UTF-8" +LC_COLLATE="en_US.UTF-8" +LC_CTYPE="UTF-8" +LC_MESSAGES="en_US.UTF-8" +LC_MONETARY="en_US.UTF-8" +LC_NUMERIC="en_US.UTF-8" +LC_TIME="en_US.UTF-8" +LC_ALL= +``` + +## Static entries in `/etc/hosts` + +If you are unable to access the following domains, you can customize your local `/etc/hosts` +with the following static entries. This may be necessary, for example, if you use Pi-hole and +all DNS entries from a public DNS with a non-public IP address are filtered. + +```bash +# OSISM Testbed hosts +192.168.16.5 ara.testbed.osism.xyz ara +192.168.16.5 cgit.testbed.osism.xyz cgit +192.168.16.5 flower.testbed.osism.xyz flower +192.168.16.5 homer.testbed.osism.xyz homer +192.168.16.5 netbox.testbed.osism.xyz netbox +192.168.16.5 testbed-manager.testbed.osism.xyz testbed-manager +192.168.16.5 nexus.testbed.osism.xyz nexus +192.168.16.5 phpmyadmin.testbed.osism.xyz phpmyadmin +192.168.16.9 api-int.testbed.osism.xyz api-int +192.168.16.10 testbed-node-0.testbed.osism.xyz testbed-node-0 +192.168.16.11 testbed-node-1.testbed.osism.xyz testbed-node-1 +192.168.16.12 testbed-node-2.testbed.osism.xyz testbed-node-2 +192.168.16.13 testbed-node-3.testbed.osism.xyz testbed-node-3 +192.168.16.14 testbed-node-4.testbed.osism.xyz testbed-node-4 +192.168.16.15 testbed-node-5.testbed.osism.xyz testbed-node-5 +192.168.16.16 testbed-node-6.testbed.osism.xyz testbed-node-6 +192.168.16.17 testbed-node-7.testbed.osism.xyz testbed-node-7 +192.168.16.18 testbed-node-8.testbed.osism.xyz testbed-node-8 +192.168.16.19 testbed-node-9.testbed.osism.xyz testbed-node-9 +192.168.16.100 keycloak.testbed.osism.xyz keycloak +192.168.16.254 api.testbed.osism.xyz api +``` \ No newline at end of file diff --git a/docs/testbed/usage.mdx b/docs/testbed/usage.mdx new file mode 100644 index 0000000000..b4a3601871 --- /dev/null +++ b/docs/testbed/usage.mdx @@ -0,0 +1,305 @@ +--- +sidebar_label: Usage +sidebar_position: 3 +--- + +# Usage + +Deployment must be completed at this point. + +## Custom CA + +The OSISM Testbed deployment currently uses hostnames in the domain `testbed.osism.xyz`. This is a real domain, and we provide the DNS records matching the addresses used in the OSISM Testbed, so that once you connect to your testbed via a direct link or Wireguard, you can access hosts and servers by their hostname (e.g. `ssh testbed-manager.testbed.osism.xyz`). + +We also provide a wildcard TLS certificate signed by a custom CA for `testbed.osism.xyz` and `*.testbed.osism.xyz`. +This CA is always used for each testbed. The CA is not regenerated, and it is not planned to change this for the next 10 years. + +In order for these certificates to be recognized locally as valid, the CA +[environments/kolla/certificates/ca/testbed.crt](https://raw.githubusercontent.com/osism/testbed/main/environments/kolla/certificates/ca/testbed.crt) +must be imported locally. + +## VPN access + +You can connect to [REGIO.cloud](https://regio.digital) either by using Wireguard, or sshuttle. + +### Wireguard + +Install [Wireguard](https://www.wireguard.com) on your workstation. For instructions on how to do +it on your workstation, please have a look at the documentation of your used distribution. + +Start the wireguard tunnel. (Press `CTRL+C` to keep the tunnel running forever. The `make target` also launches a browser tab with references to all services.) + +```bash +make vpn-wireguard ENVIRONMENT=regiocloud +``` + +If you want to connect to the OSISM Testbed from multiple clients, change the client IP +address in the downloaded configuration file to be different on each client. + +If you only want to download the Wireguard configuration, you can use the `vpn-wireguard-config` +target. The configuration is then available in the file `wg-testbed-regiocloud.conf`, for example. + +```bash +make vpn-wireguard-config ENVIRONMENT=regiocloud +``` + +### sshuttle + +Install [sshuttle](https://github.com/sshuttle/sshuttle) on your workstation, then start the sshuttle tunnel. + +```bash +make vpn-sshuttle ENVIRONMENT=regiocloud +killall sshuttle +``` + +## Webinterfaces + +All SSL-enabled services within the OSISM Testbed use certs which are signed by the self-signed +[OSISM Testbed CA](https://raw.githubusercontent.com/osism/testbed/main/environments/kolla/certificates/ca/testbed.crt) +(Download the file and import it as certification authority to your browser). + +If you want to access the services, please choose the URL from the following table: + +| Name | URL | Username | Password | Note | +|:-------------------------|:-----------------------------------------------|:-------------|:-------------|:----------------| +| ARA | https://ara.testbed.osism.xyz | ara | password | | +| Ceph | https://api-int.testbed.osism.xyz:8140 | admin | password | | +| Flower | https://flower.testbed.osism.xyz | | | | +| Grafana | https://api-int.testbed.osism.xyz:3000 | admin | password | | +| HAProxy (testbed-node-0) | http://testbed-node-0.testbed.osism.xyz:1984 | openstack | password | | +| HAProxy (testbed-node-1) | http://testbed-node-1.testbed.osism.xyz:1984 | openstack | password | | +| HAProxy (testbed-node-2) | http://testbed-node-2.testbed.osism.xyz:1984 | openstack | password | | +| Homer | https://homer.testbed.osism.xyz | | | | +| Horizon (via Keystone) | https://api.testbed.osism.xyz | admin | password | domain: default | +| Horizon (via Keystone) | https://api.testbed.osism.xyz | test | test | domain: test | +| Netbox | https://netbox.testbed.osism.xyz | admin | password | | +| Netdata | http://testbed-manager.testbed.osism.xyz:19999 | | | | +| Nexus | https://nexus.testbed.osism.xyz | admin | password | | +| OpenSearch Dashboards | https://api.testbed.osism.xyz:5601 | opensearch | password | | +| Prometheus | https://api-int.testbed.osism.xyz:9091 | admin | password | | +| RabbitMQ | https://api-int.testbed.osism.xyz:15672 | openstack | password | | +| phpMyAdmin | https://phpmyadmin.testbed.osism.xyz | root | password | | + +### Usage of the OpenStack CLI + +The `environments/openstack` folder contains the needed files for the openstack client: + +```bash +cd environments/openstack +export OS_CLOUD= # i.e. admin +openstack floating ip list +``` + +## Advanced Usage + +### External API + +It is possible to provide the OpenStack APIs and the OpenStack Dashboard via the manager's public IP address. +This is not enabled by default, except for the OTC profile. To provide the OpenStack APIs and the +OpenStack dashboard via the public IP address of the manager, the following changes are necessary in the +`terraform/environments/regiocloud.tfvars` file. If a cloud other than the REGIO.cloud is used, the profile +of the other cloud is changed accordingly. + +1. Add the customization `external_api`. This customization makes sure that the required security group rules + are created for the various OpenStack APIs and the OpenStack dashboard. + + ``` + # customisation:external_api + ``` + +2. Set parameter `external_api` to `true`. This makes sure that all necessary changes are made in the configuration repository when the manager service is deployed. It is correct that this is added as a comment. + + ``` + external_api = true + ``` + +3. After the deployment of the manager service and the OpenStack services, the OpenStack APIs and the OpenStack dashboard can be reached via a DNS name. The service [traefik.me](https://traefik.me) is used for the DNS record. Run the following two commands on the manager node to get the DNS record. + + ``` + $ source /opt/manager-vars.sh + $ echo "api-${MANAGER_PUBLIC_IP_ADDRESS//./-}.traefik.me" + api-80-158-46-219.traefik.me + ``` + +### Change versions + +1. Go to `/opt/configuration` on `testbed-manager` +2. Run `./scripts/set-openstack-version.sh 2024.2` to set the OpenStack version to `2024.2` +3. Run `./scripts/set-ceph-version.sh reef` to set the Ceph version to `reef` +4. Run `osism update manager` to update the manager service + +### Deploy services + +| Script | Description | +|:------------------------------------------------------------------|:----------------------------| +| `/opt/configuration/scripts/deploy/000-manager.sh` | | +| `/opt/configuration/scripts/deploy/001-helpers.sh` | | +| `/opt/configuration/scripts/deploy/100-ceph-with-ansible.sh` | Ceph with Ansible | +| `/opt/configuration/scripts/deploy/100-ceph-with-rook.sh` | Ceph with Rook | +| `/opt/configuration/scripts/deploy/200-infrastructure.sh` | | +| `/opt/configuration/scripts/deploy/300-openstack.sh` | | +| `/opt/configuration/scripts/deploy/310-openstack-extended.sh` | | +| `/opt/configuration/scripts/deploy/400-monitoring.sh` | | +| `/opt/configuration/scripts/deploy/500-kubernetes.sh` | | +| `/opt/configuration/scripts/deploy/510-clusterapi.sh` | | + +### Upgrade services + +| Script | Description | +|:--------------------------------------------------------------------------|:----------------------------| +| `/opt/configuration/scripts/upgrade/100-ceph-with-ansible.sh` | Ceph with Ansible | +| `/opt/configuration/scripts/upgrade/100-ceph-with-rook.sh` | Ceph with Rook | +| `/opt/configuration/scripts/upgrade/200-infrastructure.sh` | | +| `/opt/configuration/scripts/upgrade/300-openstack.sh` | | +| `/opt/configuration/scripts/upgrade/310-openstack-extended.sh` | | +| `/opt/configuration/scripts/upgrade/400-monitoring.sh` | | +| `/opt/configuration/scripts/upgrade/500-kubernetes.sh` | | +| `/opt/configuration/scripts/upgrade/510-clusterapi.sh` | | + +### Add new OSD in Ceph + +In the testbed, three volumes per node are provided for use by Ceph by default. Two of +these devices are used as OSDs during the initial deployment. The third device is intended +for testing the addition of a further OSD to the Ceph cluster. + +1. Add `sdd` to `ceph_osd_devices` in `/opt/configuration/inventory/host_vars/testbed-node-0/ceph-lvm-configuration.yml`. + +:::info + +The following content is an example, the IDs look different everywhere. Do not copy 1:1 but only add `sdd` to the file. + +::: + +```yaml +--- +# +# This is Ceph LVM configuration for testbed-node-0 +# generated by ceph-configure-lvm-volumes playbook. +# +ceph_osd_devices: + sdb: + osd_lvm_uuid: 95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a + sdc: + osd_lvm_uuid: 29899765-42bf-557b-ae9c-5c7c984b2243 + sdd: +lvm_volumes: +- data: osd-block-95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a + data_vg: ceph-95a9a2e0-b23f-55b2-a04f-e02ddfc0e82a +- data: osd-block-29899765-42bf-557b-ae9c-5c7c984b2243 + data_vg: ceph-29899765-42bf-557b-ae9c-5c7c984b2243 +``` + +2. Run `osism apply ceph-configure-lvm-volumes -l testbed-node-0` + +3. Run `cp /tmp/testbed-node-0-ceph-lvm-configuration.yml /opt/configuration/inventory/host_vars/testbed-node-0/ceph-lvm-configuration.yml`. + +4. Run `osism reconciler sync` + +5. Run `osism apply ceph-create-lvm-devices -l testbed-node-0` + +6. Run `osism apply ceph-osds -l testbed-node-0 -e ceph_handler_osds_restart=false` + +7. Check the OSD tree + + ``` + $ ceph osd tree + ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF + -1 0.13640 root default + -3 0.05846 host testbed-node-0 + 2 hdd 0.01949 osd.2 up 1.00000 1.00000 + 4 hdd 0.01949 osd.4 up 1.00000 1.00000 + 6 hdd 0.01949 osd.6 up 1.00000 1.00000 + -5 0.03897 host testbed-node-1 + 0 hdd 0.01949 osd.0 up 1.00000 1.00000 + 5 hdd 0.01949 osd.5 up 1.00000 1.00000 + -7 0.03897 host testbed-node-2 + 1 hdd 0.01949 osd.1 up 1.00000 1.00000 + 3 hdd 0.01949 osd.3 up 1.00000 1.00000 + ``` + +### Ceph via Rook (technical preview) + +Please have a look at [Deploy Guide - Services - Rook](guides/deploy-guide/services/ceph/rook.md) and [Configuration Guide - Rook](guides/configuration-guide/ceph/rook.md) for details on how to configure Rook. + +To deploy this in the testbed, you can use an environment variable in your make target. + +``` +make CEPH_STACK=rook manager +make CEPH_STACK=rook ceph +``` + +This will make sure `/opt/manager-vars.sh` gets `CEPH_STACK=rook` set which is later being used by: + +``` +/opt/configuration/scripts/deploy-services.sh +/opt/configuration/scripts/deploy-ceph.sh +/opt/configuration/scripts/upgrade/100-ceph-with-rook.sh +``` + +## Using testbed for OpenStack development + +Testbed may be used for doing development on OpenStack services through [`kolla_dev_mode`](https://docs.openstack.org/kolla-ansible/latest/contributor/kolla-for-openstack-development.html). +`kolla_dev_mode` may be activated for all supported service by adding + +``` +kolla_dev_mode: true +``` + +to `environments/kolla/configuration.yml`. This will check out the `git` repositories of all supported and enabled OpenStack services under `/opt/stack`, and bind mount them into the appropriate containers. +Since this will fetch a lot of repositories it is advisable to enable this only selectively for the services you are going to work on. You can do so by adding a boolean variable to `environments/kolla/configuration.yml` consisting of the service name and the suffix `_dev_mode`, e.g.: + +``` +cinder_dev_mode: true +``` + +You may customise the used `git` repository by adding `kolla_repos_git` to `environments/kolla/configuration.yml` to specify a common root for the repositories of all services, which will be completed by the service's name, e.g.: + +``` +kolla_repos_git: "https://github.com/openstack" +``` + +This will pull services from their `github` mirror. E.g. pull nova from https://github.com/openstack/nova +The complete repository of a single service may be changed by adding a variable consisting of the service name and the suffix `_git_repository`, e.g.: + +``` +cinder_git_repository: "https://github.com/myorg/my_custom_cinder_fork" +``` + +You may specify the `git` reference globally by setting + +``` +kolla_source_version: my_feature_branch +``` + +in `environments/kolla/configuration.yml`, or for specific services, via service name and suffix `_source_version`, e.g.: + +``` +cinder_source_version: my_cinder_feature +``` + +In order to update the `git` repositories on `kolla-ansible` deployments set either + +``` +kolla_dev_repos_pull: true +``` + +or + +``` +cinder_dev_repos_pull: true +``` + +in `environments/kolla/configuration.yml` to set the setting for all repositories, or a service specific one. + +### Example workflow + +Thus, as an example for a development workflow: +* Create a `git` mirror of the OpenStack service you want to work on +* In `environments/kolla/configuration.yml`: + * Set the corresponding `_dev_mode` variable to `true` + * Set the corresponding `_git_repository` variable to your development mirror created above + * Set the corresponding `_source_version` variable to a branch name you intend to push your changes to + * Set the corresponding `_dev_repos_pull` variable to `true` +* Commit your changes and push them to the specified branch in the specified mirror +* Run `osism apply ` on the manager to checkout the code on every testbed node +* Restart the service containers to actually pick up the new code \ No newline at end of file