Skip to content

Commit 3fb0f0c

Browse files
softwarefactory-project-zuul[bot]softwarefactory-project-zuul[bot]
authored
Merge pull request #634 from rajeshP524/ceph-scalelab
Add Optional Ceph support to Scalelab DT Reviewed-by: Andrew Bays <[email protected]> Reviewed-by: rajeshP524
2 parents d66b86f + 5b99bfc commit 3fb0f0c

File tree

12 files changed

+473
-7
lines changed

12 files changed

+473
-7
lines changed

‎dt/perfscale/scalelab/ceph/ceph-secret.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
data:
4+
ceph.client.openstack.keyring: _replaced_
5+
ceph.conf: _replaced_
6+
kind: Secret
7+
metadata:
8+
name: ceph-conf-files
9+
namespace: openstack
10+
type: Opaque

‎dt/perfscale/scalelab/ceph/edpm-nodeset/kustomization.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1alpha1
3+
kind: Component
4+
5+
resources:
6+
- nova-ceph.yaml
7+
8+
replacements:
9+
# Nodeset services override for Ceph
10+
- source:
11+
kind: ConfigMap
12+
name: edpm-nodeset-values-ceph
13+
fieldPath: data.nodeset.services
14+
targets:
15+
- select:
16+
kind: OpenStackDataPlaneNodeSet
17+
fieldPaths:
18+
- spec.services
19+
options:
20+
create: true
21+
# Nova Ceph configuration
22+
- source:
23+
kind: ConfigMap
24+
name: edpm-nodeset-values-ceph
25+
fieldPath: data.nova.ceph.conf
26+
targets:
27+
- select:
28+
kind: ConfigMap
29+
name: ceph-nova
30+
fieldPaths:
31+
- data.03-ceph-nova\.conf
32+
options:
33+
create: true
34+
# EDPM nodeset extraMounts
35+
- source:
36+
kind: ConfigMap
37+
name: edpm-nodeset-values-ceph
38+
fieldPath: data.extraMounts
39+
targets:
40+
- select:
41+
kind: OpenStackDataPlaneNodeSet
42+
fieldPaths:
43+
- spec.nodeTemplate.extraMounts
44+
options:
45+
create: true

‎dt/perfscale/scalelab/ceph/edpm-nodeset/nova-ceph.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: ceph-nova
6+
namespace: openstack
7+
data:
8+
03-ceph-nova.conf: _replaced_
9+
---
10+
apiVersion: dataplane.openstack.org/v1beta1
11+
kind: OpenStackDataPlaneService
12+
metadata:
13+
name: nova-custom-ceph
14+
namespace: openstack
15+
spec:
16+
dataSources:
17+
- configMapRef:
18+
name: ceph-nova
19+
- secretRef:
20+
name: nova-cell1-compute-config
21+
- secretRef:
22+
name: nova-migration-ssh-key
23+
playbook: osp.edpm.nova
24+
caCerts: combined-ca-bundle
25+
edpmServiceType: nova

‎dt/perfscale/scalelab/ceph/kustomization.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1alpha1
3+
kind: Component
4+
5+
resources:
6+
- ceph-secret.yaml
7+
8+
replacements:
9+
# Ceph secret replacements
10+
- source:
11+
kind: ConfigMap
12+
name: service-values-ceph
13+
fieldPath: data.ceph.conf
14+
targets:
15+
- select:
16+
kind: Secret
17+
name: ceph-conf-files
18+
fieldPaths:
19+
- data.ceph\.conf
20+
options:
21+
create: true
22+
- source:
23+
kind: ConfigMap
24+
name: service-values-ceph
25+
fieldPath: data.ceph.keyring
26+
targets:
27+
- select:
28+
kind: Secret
29+
name: ceph-conf-files
30+
fieldPaths:
31+
- data.ceph\.client\.openstack\.keyring
32+
options:
33+
create: true
34+
# Ceph-specific control plane replacements
35+
- source:
36+
kind: ConfigMap
37+
name: service-values-ceph
38+
fieldPath: data.cinderVolumes.ceph.customServiceConfig
39+
targets:
40+
- select:
41+
kind: OpenStackControlPlane
42+
fieldPaths:
43+
- spec.cinder.template.cinderVolumes.ceph.customServiceConfig
44+
options:
45+
create: true
46+
- source:
47+
kind: ConfigMap
48+
name: service-values-ceph
49+
fieldPath: data.cinderVolumes.ceph.networkAttachments
50+
targets:
51+
- select:
52+
kind: OpenStackControlPlane
53+
fieldPaths:
54+
- spec.cinder.template.cinderVolumes.ceph.networkAttachments
55+
options:
56+
create: true
57+
- source:
58+
kind: ConfigMap
59+
name: service-values-ceph
60+
fieldPath: data.glance.customServiceConfig
61+
targets:
62+
- select:
63+
kind: OpenStackControlPlane
64+
fieldPaths:
65+
- spec.glance.template.customServiceConfig
66+
options:
67+
create: true
68+
- source:
69+
kind: ConfigMap
70+
name: service-values-ceph
71+
fieldPath: data.extraMounts
72+
targets:
73+
- select:
74+
kind: OpenStackControlPlane
75+
fieldPaths:
76+
- spec.extraMounts
77+
options:
78+
create: true

‎examples/dt/perfscale/scalelab/README.md‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@ This is a collection of CR templates that represent a validated Red Hat OpenStac
88
- OVN networking
99
- Network isolation over three NIC
1010
- this example only uses two compute nodes in one node set, though it is assumed the number will be scaled
11-
- Swift enabled and used as Glance backend
11+
- Swift enabled and used as Glance backend (default)
12+
- Ceph backend support (optional)
13+
- Cinder Volume using RBD backend
14+
- Glance using RBD backend
15+
- Nova using RBD for ephemeral storage
1216

1317
## Considerations
1418

@@ -22,11 +26,17 @@ This is a collection of CR templates that represent a validated Red Hat OpenStac
2226

2327
5. In stage 4 [kustomize](https://kustomize.io/) is used to generate the dataplane CRs dynamically. The `edpm/values.yaml` file must be updated to fit your environment. kustomize version 5 or newer required.
2428

29+
6. Ceph backend is optional. To enable Ceph, see [ceph.md](ceph.md) for configuration requirements. Ceph must be deployed and configured before applying the CRs.
30+
2531
## Stages
2632

2733
All stages must be executed in the order listed below. Everything is required unless otherwise indicated.
2834

2935
1. [Create lvm cluster and subscription on worker nodes](lvm-cluster.md)
3036
1. [Install the OpenStack K8S operators and their dependencies](../../../common/)
3137
2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
32-
3. [Configure and deploy the dataplane](dataplane.md)
38+
3. [Configure and deploy the dataplane](dataplane.md)
39+
40+
## Optional: Ceph Backend
41+
42+
To deploy with Ceph backend for Cinder, Glance, and Nova, see [Ceph Configuration Guide](ceph.md).

‎examples/dt/perfscale/scalelab/ceph.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# Ceph Backend Configuration
2+
3+
This guide describes how to enable Ceph backend for OpenStack services. Ceph support is **optional** and provides storage for Cinder, Glance, and Nova.
4+
5+
## Overview
6+
7+
When enabled, Ceph provides:
8+
- **Cinder Volume**: RBD backend for block storage
9+
- **Glance**: RBD backend for image storage
10+
- **Nova Compute**: RBD backend for ephemeral storage
11+
12+
## Prerequisites
13+
14+
1. A Ceph cluster must be deployed and accessible via storage network.
15+
2. The following Ceph configuration must be available:
16+
- Ceph configuration file (`ceph.conf`)
17+
- Ceph client keyring (`ceph.client.openstack.keyring`)
18+
- Ceph FSID (from `ceph.conf`)
19+
3. Ceph pools must be created:
20+
- `volumes` - for Cinder volumes
21+
- `images` - for Glance images
22+
- `vms` - for Nova ephemeral storage
23+
24+
## Configuration Steps
25+
26+
### 1. Prepare Ceph Configuration Values
27+
28+
Retrieve and encode the Ceph configuration from your Ceph cluster:
29+
30+
```bash
31+
# Get base64-encoded Ceph configuration
32+
CEPH_CONF=$(cat /etc/ceph/ceph.conf | base64 -w 0)
33+
CEPH_KEYRING=$(cat /etc/ceph/ceph.client.openstack.keyring | base64 -w 0)
34+
35+
# Extract FSID from secret/ alternatively can be fetched from ceph.conf
36+
FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //')
37+
```
38+
39+
### 2. Enable Ceph in Control Plane
40+
41+
Edit `examples/dt/perfscale/scalelab/kustomization.yaml` and uncomment:
42+
43+
```yaml
44+
components:
45+
- ../../../../dt/perfscale/scalelab/
46+
# To enable Ceph, uncomment the following line:
47+
- ../../../../dt/perfscale/scalelab/ceph/ # ← Uncomment this
48+
49+
resources:
50+
- networking/nncp/values.yaml
51+
- service-values.yaml
52+
# To enable Ceph, uncomment the following line:
53+
- service-values-ceph.yaml # ← Uncomment this
54+
```
55+
56+
Edit `service-values-ceph.yaml` and replace the following values:
57+
58+
```yaml
59+
data:
60+
# Ceph conf and keyring
61+
ceph:
62+
conf: CHANGEME_CEPH_CONF # Replace with $CEPH_CONF
63+
keyring: CHANGEME_CEPH_KEYRING # Replace with $CEPH_KEYRING
64+
65+
# Cinder RBD backend
66+
cinderVolumes:
67+
ceph:
68+
customServiceConfig: |
69+
[DEFAULT]
70+
enabled_backends = ceph
71+
[ceph]
72+
volume_backend_name = ceph
73+
volume_driver = cinder.volume.drivers.rbd.RBDDriver
74+
rbd_ceph_conf = /etc/ceph/ceph.conf
75+
rbd_user = openstack
76+
rbd_pool = volumes
77+
rbd_flatten_volume_from_snapshot = False
78+
rbd_secret_uuid = CHANGEME # Replace with $CEPH_FSID
79+
```
80+
81+
### 3. Enable Ceph in Data Plane
82+
83+
Edit `examples/dt/perfscale/scalelab/edpm/nodeset/kustomization.yaml` and uncomment:
84+
85+
```yaml
86+
components:
87+
- ../../../../../../dt/perfscale/scalelab/edpm/nodeset
88+
# To enable Ceph, uncomment the following line:
89+
- ../../../../../../dt/perfscale/scalelab/ceph/edpm-nodeset # ← Uncomment this
90+
91+
resources:
92+
- values.yaml
93+
# To enable Ceph, uncomment the following line:
94+
- values-ceph.yaml # ← Uncomment this
95+
```
96+
97+
Edit `edpm/nodeset/values-ceph.yaml` and configure:
98+
99+
```yaml
100+
data:
101+
nova:
102+
ceph:
103+
conf: | # Replace CHANGEME_NOVA_CEPH_CONF with:
104+
[libvirt]
105+
images_type=rbd
106+
images_rbd_pool=vms
107+
images_rbd_ceph_conf=/etc/ceph/ceph.conf
108+
images_rbd_glance_store_name=default_backend
109+
images_rbd_glance_copy_poll_interval=15
110+
images_rbd_glance_copy_timeout=600
111+
rbd_user=openstack
112+
rbd_secret_uuid=<FSID> # Replace with $CEPH_FSID
113+
```
114+
115+
## Deployment
116+
117+
After enabling and configuring Ceph, deploy as normal following the main [README.md](README.md):
118+
119+
1. Deploy control plane: [control-plane.md](control-plane.md)
120+
2. Deploy data plane: [dataplane.md](dataplane.md)
121+
122+
The Ceph configuration will be automatically integrated into the generated CRs.
123+
124+
## What Gets Configured
125+
126+
### Control Plane
127+
128+
- **Ceph Secret**: `ceph-conf-files` secret with Ceph configuration and keyring
129+
- **Cinder**:
130+
- Ceph volume backend configured
131+
- **Glance**: RBD backend for image storage
132+
- **Extra Mounts**: Ceph configuration files mounted in CinderVolume and GlanceAPI pods
133+
134+
### Data Plane
135+
136+
- **Ceph Client**: Installed and configured on compute nodes
137+
- **Nova**:
138+
- Custom service `nova-custom-ceph` replaces standard `nova` service
139+
- RBD backend for ephemeral storage
140+
- Ceph configuration mounted in Nova compute containers
141+
142+
## Disabling Ceph
143+
144+
To deploy without Ceph (default Swift backend for Glance):
145+
146+
1. Ensure Ceph lines remain commented in both kustomization.yaml files
147+
2. Do not include `service-values-ceph.yaml` or `values-ceph.yaml` resources
148+
3. Deploy normally - Swift will be used as Glance backend
149+
150+
## Architecture
151+
152+
The Ceph integration uses a modular kustomize component structure:
153+
154+
```
155+
dt/perfscale/scalelab/ceph/ # Control plane Ceph component
156+
├── kustomization.yaml # Ceph replacements for control plane
157+
├── ceph-secret.yaml # Ceph secret template
158+
└── edpm-nodeset/ # Data plane Ceph component
159+
├── kustomization.yaml # Ceph replacements for EDPM
160+
└── nova-ceph.yaml # Nova Ceph service definition
161+
162+
examples/dt/perfscale/scalelab/
163+
├── service-values-ceph.yaml # Control plane Ceph values
164+
└── edpm/nodeset/values-ceph.yaml # Data plane Ceph values
165+
```
166+
167+
This modular design allows Ceph to be enabled/disabled by simply commenting/uncommenting two lines in the kustomization files.
168+

‎examples/dt/perfscale/scalelab/control-plane.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,16 @@ vi networking/nncp/values.yaml
2222
vi service-values.yaml
2323
```
2424

25+
### Optional: Enable Ceph backend
26+
27+
To enable Ceph backend for Cinder, Glance, and Nova, see the [Ceph Configuration Guide](ceph.md).
28+
If enabling Ceph, uncomment the Ceph lines in [kustomization.yaml](kustomization.yaml) and edit
29+
[service-values-ceph.yaml](service-values-ceph.yaml).
30+
```
31+
vi kustomization.yaml # Uncomment Ceph component and resource
32+
vi service-values-ceph.yaml
33+
```
34+
2535
## Apply node network configuration
2636

2737
Generate the node network configuration
@@ -63,4 +73,6 @@ oc apply -f control-plane.yaml
6373
Wait for control plane to be available
6474
```
6575
oc wait osctlplane controlplane --for condition=Ready --timeout=600s
66-
```
76+
```
77+
78+
**Note:** If Ceph is enabled, the generated CRs will include Ceph configuration for Cinder, Glance, and extraMounts.

0 commit comments

Comments
 (0)