Skip to content

Commit 6b3ab2e

Browse files
softwarefactory-project-zuul[bot]softwarefactory-project-zuul[bot]
authored
Merge pull request #232 from openstack-k8s-operators/zuul/validate-all-scenarios
Generate build jobs for all automations This patch provides a script generating all of the zuul.d content, based on the automation files. Each scenario will get its own job definition, and will be triggered only when its files are changed. Note that common files, such as "lib", will trigger all the jobs. This patch also introduces a new github workflow, to ensure all of the automation scenarios have a job, by re-running the script and ensuring no chance happened in zuul.d directory. Depends-On: openstack-k8s-operators/ci-framework#1742 Reviewed-by: Cédric Jeanneret Reviewed-by: Pragadeeswaran Sathyanarayanan <psathyan@redhat.com> Reviewed-by: Andrew Bays <andrew.bays@gmail.com>
2 parents 2d5ab81 + b24f345 commit 6b3ab2e

File tree

9 files changed

+240
-27
lines changed

9 files changed

+240
-27
lines changed

‎.github/workflows/linter.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,23 @@ jobs:
4141
run: |
4242
kustomize version
4343
./test-kustomizations.sh examples/
44+
45+
test-zuul:
46+
runs-on: ubuntu-latest
47+
steps:
48+
- name: Checkout
49+
uses: actions/checkout@v4
50+
51+
- name: Setup python
52+
uses: actions/setup-python@v5
53+
with:
54+
python-version: '3.10'
55+
56+
- name: Install yaml
57+
run: pip install pyyaml
58+
59+
- name: Run create-zuul-jobs
60+
run: |
61+
sha256sum zuul.d/* > checksum;
62+
./create-zuul-jobs.py;
63+
sha256sum -c checksum || (git diff && exit 1) && exit 0;

‎.yamllint.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
extends: default
33
ignore:
44
- '*.md'
5+
- 'zuul.d/'
56

67
rules:
78
line-length:

‎automation/mocks/ovs-dpdk.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ovs-dpdk-sriov.yaml

‎automation/mocks/sriov.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ovs-dpdk-sriov.yaml

‎automation/vars/README.md‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,47 @@ project if you add a new key, or modify a type!
77

88
More information about [yamale](https://github.com/23andMe/Yamale).
99

10-
## Manually check schema
10+
### Manually check schema
1111
```Bash
1212
$ pip install yamale
1313
$ yamale -s .ci/automation-schema.yaml automation/vars
1414
```
15+
16+
## Adding a new automation scenario
17+
18+
You have to ensure new CI job is create for your new scenario.
19+
20+
You can generate the new job running the `create-zuul-jobs.py`
21+
script located at the root of this repository.
22+
23+
The script requires `pyyaml`.
24+
25+
The Zuul job supports `Depends-On` in case your new scenario needs specific
26+
data from the CI Framework (usually in the
27+
[ci_gen_kustomize_values](https://ci-framework.readthedocs.io/en/latest/roles/ci_gen_kustomize_values.html)
28+
role).
29+
30+
### Manually validating your scenario: CI Framework
31+
32+
The [CI Framework](https://github.com/openstack-k8s-operators/ci-framework) project is
33+
able to consume the automation files. Therefore, it can be used in order to ensure
34+
the various pieces are working together.
35+
36+
#### Read-only build
37+
38+
The Framework provides a playbook able to run against the automation, and build the CRs.
39+
This is reflected in the Zuul CI tests associated to this project, like
40+
`rhoso-architecture-validate-hci`.
41+
42+
If you want to manually test using the CI Framework before proposing a new scenario automation,
43+
you can run the following:
44+
```Bash
45+
$ cd ci-framework
46+
$ make run_ctx_architecture_test \
47+
SCENARIO_NAME=your_scenario \
48+
ARCH_REPO=../architecture \
49+
NET_ENV_FILE=./ci/playbooks/files/networking-env-definition.yml
50+
```
51+
You have to install `podman` to run this - everything will run in a container.
52+
53+
You can find the built CRs in `~/ci-framework-data/your_scenario/artifacts/`.

‎create-zuul-jobs.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env python3
2+
3+
import os
4+
import yaml
5+
6+
_AUTO_PATH = 'automation/vars/'
7+
_JOBS = []
8+
_PROJECTS = ['noop']
9+
_BASE_TRIGGER_FILES = [
10+
'lib',
11+
]
12+
13+
def create_job(name, data):
14+
j_name = 'rhoso-architecture-validate-{}'.format(name)
15+
_paths = [p['path'] for p in data['stages']] + _BASE_TRIGGER_FILES
16+
for i in ['va', 'dt']:
17+
if os.path.exists(os.path.join(i, name)):
18+
_paths.append(os.path.join(i, name))
19+
_mock = os.path.join('automation', 'mocks', '{}.yaml'.format(name))
20+
if os.path.exists(_mock):
21+
_paths.append(_mock)
22+
_paths.sort()
23+
job = {
24+
'job': {
25+
'name': j_name,
26+
'parent': 'rhoso-architecture-base-job',
27+
'vars': {
28+
'cifmw_architecture_scenario': name,
29+
},
30+
'files': _paths,
31+
}
32+
}
33+
_JOBS.append(job)
34+
_PROJECTS.append(j_name)
35+
36+
37+
for fname in [f for f in os.listdir(_AUTO_PATH) if f.endswith('.yaml')]:
38+
with open(os.path.join(_AUTO_PATH, fname), 'r') as y_file:
39+
scenarios = yaml.safe_load(y_file)
40+
for scenario, stages in scenarios['vas'].items():
41+
create_job(scenario, stages)
42+
43+
_sorted_jobs = sorted(_JOBS, key= lambda d: d['job']['name'])
44+
with open('./zuul.d/validations.yaml', 'w+') as zuul_jobs:
45+
yaml.dump(_sorted_jobs, zuul_jobs)
46+
47+
_PROJECTS.sort()
48+
with open('./zuul.d/projects.yaml', 'w+') as zuul_projects:
49+
struct = [{'project': {'github-check': {'jobs': _PROJECTS},
50+
'github-gate': {'jobs': _PROJECTS}
51+
}
52+
}
53+
]
54+
yaml.dump(struct, zuul_projects)

zuul.d/validate-jobs.yaml renamed to zuul.d/base-jobs.yaml

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
# How to create a new job:
3-
# - check the syntax defined for the existing jobs
3+
# - run .ci/create-zuul-jobs.py script to regenerate based on automations
44
# - if your scenario needs mocked data:
55
# - push a proper YAML file in automation/mocks/SCENARIO_NAME.yaml
66
# - that's all!
@@ -14,19 +14,3 @@
1414
parent: cifmw-architecture-validate-base
1515
required-projects:
1616
- openstack-k8s-operators/ci-framework
17-
18-
- job:
19-
name: rhoso-architecture-validate-hci
20-
parent: rhoso-architecture-base-job
21-
vars:
22-
cifmw_architecture_scenario: hci
23-
files:
24-
- examples/va/hci
25-
26-
- job:
27-
name: rhoso-architecture-validate-ovs-dpdk-sriov
28-
parent: rhoso-architecture-base-job
29-
vars:
30-
cifmw_architecture_scenario: ovs-dpdk-sriov
31-
files:
32-
- examples/va/nfv

‎zuul.d/projects.yaml‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
1-
---
21
- project:
32
github-check:
4-
jobs:
5-
- noop
6-
- rhoso-architecture-validate-hci
7-
- rhoso-architecture-validate-ovs-dpdk-sriov
3+
jobs: &id001
4+
- noop
5+
- rhoso-architecture-validate-bgp
6+
- rhoso-architecture-validate-hci
7+
- rhoso-architecture-validate-ovs-dpdk
8+
- rhoso-architecture-validate-ovs-dpdk-sriov
9+
- rhoso-architecture-validate-sriov
10+
- rhoso-architecture-validate-uni01alpha
11+
- rhoso-architecture-validate-uni02beta
12+
- rhoso-architecture-validate-uni04delta
13+
- rhoso-architecture-validate-uni06zeta
814
github-gate:
9-
jobs:
10-
- noop
11-
- rhoso-architecture-validate-hci
12-
- rhoso-architecture-validate-ovs-dpdk-sriov
15+
jobs: *id001

‎zuul.d/validations.yaml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
- job:
2+
files:
3+
- dt/bgp
4+
- examples/dt/bgp/control-plane
5+
- examples/dt/bgp/control-plane/nncp
6+
- examples/dt/bgp/edpm/deployment
7+
- examples/dt/bgp/edpm/nodeset
8+
- lib
9+
name: rhoso-architecture-validate-bgp
10+
parent: rhoso-architecture-base-job
11+
vars:
12+
cifmw_architecture_scenario: bgp
13+
- job:
14+
files:
15+
- examples/va/hci
16+
- examples/va/hci/control-plane
17+
- examples/va/hci/control-plane/nncp
18+
- examples/va/hci/deployment
19+
- examples/va/hci/edpm-pre-ceph/deployment
20+
- examples/va/hci/edpm-pre-ceph/nodeset
21+
- lib
22+
- va/hci
23+
name: rhoso-architecture-validate-hci
24+
parent: rhoso-architecture-base-job
25+
vars:
26+
cifmw_architecture_scenario: hci
27+
- job:
28+
files:
29+
- automation/mocks/ovs-dpdk.yaml
30+
- examples/va/nfv/ovs-dpdk
31+
- examples/va/nfv/ovs-dpdk/edpm/deployment
32+
- examples/va/nfv/ovs-dpdk/edpm/nodeset
33+
- examples/va/nfv/ovs-dpdk/nncp
34+
- lib
35+
name: rhoso-architecture-validate-ovs-dpdk
36+
parent: rhoso-architecture-base-job
37+
vars:
38+
cifmw_architecture_scenario: ovs-dpdk
39+
- job:
40+
files:
41+
- automation/mocks/ovs-dpdk-sriov.yaml
42+
- examples/va/nfv/ovs-dpdk-sriov
43+
- examples/va/nfv/ovs-dpdk-sriov/edpm/deployment
44+
- examples/va/nfv/ovs-dpdk-sriov/edpm/nodeset
45+
- examples/va/nfv/ovs-dpdk-sriov/nncp
46+
- lib
47+
name: rhoso-architecture-validate-ovs-dpdk-sriov
48+
parent: rhoso-architecture-base-job
49+
vars:
50+
cifmw_architecture_scenario: ovs-dpdk-sriov
51+
- job:
52+
files:
53+
- automation/mocks/sriov.yaml
54+
- examples/va/nfv/sriov
55+
- examples/va/nfv/sriov/edpm/deployment
56+
- examples/va/nfv/sriov/edpm/nodeset
57+
- examples/va/nfv/sriov/nncp
58+
- lib
59+
name: rhoso-architecture-validate-sriov
60+
parent: rhoso-architecture-base-job
61+
vars:
62+
cifmw_architecture_scenario: sriov
63+
- job:
64+
files:
65+
- dt/uni01alpha
66+
- examples/dt/uni01alpha
67+
- examples/dt/uni01alpha/control-plane
68+
- examples/dt/uni01alpha/control-plane/nncp
69+
- examples/dt/uni01alpha/networker
70+
- lib
71+
name: rhoso-architecture-validate-uni01alpha
72+
parent: rhoso-architecture-base-job
73+
vars:
74+
cifmw_architecture_scenario: uni01alpha
75+
- job:
76+
files:
77+
- dt/uni02beta
78+
- examples/dt/uni02beta
79+
- examples/dt/uni02beta/control-plane
80+
- examples/dt/uni02beta/control-plane/nncp
81+
- lib
82+
name: rhoso-architecture-validate-uni02beta
83+
parent: rhoso-architecture-base-job
84+
vars:
85+
cifmw_architecture_scenario: uni02beta
86+
- job:
87+
files:
88+
- dt/uni04delta
89+
- examples/dt/uni04delta
90+
- examples/dt/uni04delta/control-plane
91+
- examples/dt/uni04delta/control-plane/nncp
92+
- examples/dt/uni04delta/deployment
93+
- examples/dt/uni04delta/edpm-pre-ceph
94+
- examples/dt/uni04delta/edpm-pre-ceph/nodeset
95+
- lib
96+
name: rhoso-architecture-validate-uni04delta
97+
parent: rhoso-architecture-base-job
98+
vars:
99+
cifmw_architecture_scenario: uni04delta
100+
- job:
101+
files:
102+
- dt/uni06zeta
103+
- examples/dt/uni06zeta
104+
- examples/dt/uni06zeta/control-plane
105+
- examples/dt/uni06zeta/control-plane/nncp
106+
- lib
107+
name: rhoso-architecture-validate-uni06zeta
108+
parent: rhoso-architecture-base-job
109+
vars:
110+
cifmw_architecture_scenario: uni06zeta

0 commit comments

Comments
 (0)