Skip to content

Commit c18ac85

Browse files
mangelajocursoragent
mangelajo
and
cursoragent
authored
feat: exporter-as-main QEMU pods with ExitAndReplace recycle e2e (#929)
## Summary Make QEMU ExporterSet pods follow an exporter-as-main / runtime-as-sidecar model so Pod lifecycle, logs, and ExitAndReplace recycling behave correctly, and cover that path in the main e2e suite. ### Pod layout (`qemu.jumpstarter.dev`) - **exporter** is the main container (`jmp run`, UID 65532) — default `kubectl logs`, and exporter exit completes the Pod (`restartPolicy: Never`). - **target-runtime** is a native sidecar (`restartPolicy: Always`, runs as root) — starts first so `launcher.sock` is ready; runs QEMU via `jumpstarter-exec`. - **copy-jumpstarter-exec** init container stages the binary from the **exporter** image onto `/shared` (not baked into each runtime image). - Shared `emptyDir` holds Unix sockets (QMP, serial, launcher), disk, and cloud-init cidata (world-traversable so the runtime can mount vvfat across the UID boundary). - Runtime creates sockets with permissive mode so the non-root exporter can connect. - Normalize Kubernetes quantity suffixes (`10Gi` → `10G`) for QEMU driver config. ### Lifecycle / recycle - Add `jumpstarter-exec shutdown` and best-effort runtime teardown from the exporter on `exitOnLeaseEnd`. - ExitAndReplace: delete unleased exporters whose Pods are `Succeeded`/`Failed`, then refill `minAvailableReplicas` (avoids Completed/Offline zombies and `maxReplicas` stalls). ### E2E / CI - Fold ExporterSet QEMU coverage into the main `e2e-tests` job; add `make e2e-exporterset-qemu` for focused local runs. - Build/load exporter + qemu-runtime images in CI; prefetch Alpine guest image during `e2e-setup` (CI and local). - Specs: Online/Ready Pod; power cycle + ExitAndReplace recycle stays responsive. - Flash/boot remains skipped while shared emptyDir SizeLimit is `100Mi` (needs #924). - Share OIDC/legacy client–exporter helpers in `e2e/test/utils.go`; cover `shutdown_runtime_sidecar` paths for diff-cover. - Update JEP-0014 to match the exporter-main / runtime-sidecar model. ## Test plan - [x] `go test ./internal/exporterset/...` (ExitAndReplace cleanup + RenderPod layout) - [x] `make e2e-exporterset-qemu` — Online/Ready + power cycle/recycle; flash/boot skipped on 100Mi SizeLimit - [x] `make -C python pkg-test-jumpstarter` — includes `shutdown_runtime_sidecar` coverage - [x] CI `e2e-tests` on this PR - [x] Manual: apply `controller/hack/sample-x86_64-kind.yaml`, lease/release, confirm Completed Pod is deleted and a new Ready instance appears --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 282c423 commit c18ac85

40 files changed

Lines changed: 2159 additions & 231 deletions

‎.dockerignore‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Build context for python/Containerfile is the repo root.
2+
# Keep .git — hatch-vcs metadata hooks call `git rev-parse` for URLs/version.
3+
rust/target/
4+
**/target/
5+
python/.venv/
6+
**/.venv/
7+
**/__pycache__/
8+
**/.pytest_cache/
9+
**/.mypy_cache/
10+
**/.ruff_cache/
11+
e2e/testdata/
12+
.github/
13+
docs/_build/
14+
controller/bin/
15+
*.qcow2
16+
*.img
17+
*.raw

‎.github/actions/load-e2e-artifacts/action.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,18 @@ runs:
2727
name: exporterset-controller-image-${{ inputs.arch }}
2828
path: /tmp/artifacts
2929

30+
- name: Download exporter image
31+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
32+
with:
33+
name: exporter-image-${{ inputs.arch }}
34+
path: /tmp/artifacts
35+
36+
- name: Download qemu-runtime image
37+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
38+
with:
39+
name: qemu-runtime-image-${{ inputs.arch }}
40+
path: /tmp/artifacts
41+
3042
- name: Download python wheels
3143
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
3244
with:
@@ -39,5 +51,7 @@ runs:
3951
docker load < /tmp/artifacts/controller-image.tar
4052
docker load < /tmp/artifacts/operator-image.tar
4153
docker load < /tmp/artifacts/exporterset-controller-image.tar
54+
docker load < /tmp/artifacts/exporter-image.tar
55+
docker load < /tmp/artifacts/qemu-runtime-image.tar
4256
mkdir -p controller/deploy/operator/dist
4357
cp /tmp/artifacts/operator-install.yaml controller/deploy/operator/dist/install.yaml

‎.github/workflows/e2e.yaml‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,77 @@ jobs:
172172
path: /tmp/exporterset-controller-image.tar
173173
retention-days: 1
174174

175+
build-exporter-image:
176+
needs: changes
177+
if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch'
178+
strategy:
179+
matrix:
180+
include: ${{ fromJson(needs.changes.outputs.e2e-matrix) }}
181+
runs-on: ${{ matrix.os }}
182+
timeout-minutes: 45
183+
steps:
184+
- name: Checkout repository
185+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
186+
with:
187+
fetch-depth: 0
188+
189+
- name: Cache exporter image
190+
id: cache
191+
uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5
192+
with:
193+
path: /tmp/exporter-image.tar
194+
# Bust when exporter image inputs change: Containerfile, Python sources,
195+
# lockfiles, or embedded jumpstarter-exec (rust/).
196+
key: exporter-image-${{ matrix.arch }}-${{ hashFiles('python/Containerfile', 'python/packages/**/*.py', 'python/**/pyproject.toml', 'python/uv.lock', 'rust/jumpstarter-exec/**', 'controller/Makefile') }}
197+
198+
- name: Build exporter image
199+
if: steps.cache.outputs.cache-hit != 'true'
200+
run: |
201+
make -C controller docker-build-exporter
202+
docker save quay.io/jumpstarter-dev/jumpstarter:latest -o /tmp/exporter-image.tar
203+
204+
- name: Upload exporter image
205+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
206+
with:
207+
name: exporter-image-${{ matrix.arch }}
208+
path: /tmp/exporter-image.tar
209+
retention-days: 1
210+
211+
build-qemu-runtime-image:
212+
needs: changes
213+
if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch'
214+
strategy:
215+
matrix:
216+
include: ${{ fromJson(needs.changes.outputs.e2e-matrix) }}
217+
runs-on: ${{ matrix.os }}
218+
timeout-minutes: 30
219+
steps:
220+
- name: Checkout repository
221+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
222+
with:
223+
fetch-depth: 0
224+
225+
- name: Cache qemu-runtime image
226+
id: cache
227+
uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5
228+
with:
229+
path: /tmp/qemu-runtime-image.tar
230+
# Bust when QEMU runtime inputs change.
231+
key: qemu-runtime-image-${{ matrix.arch }}-${{ hashFiles('controller/Makefile', 'controller/Containerfile.qemu-runtime') }}
232+
233+
- name: Build qemu-runtime image
234+
if: steps.cache.outputs.cache-hit != 'true'
235+
run: |
236+
make -C controller docker-build-qemu-runtime
237+
docker save quay.io/jumpstarter-dev/virtual/qemu-runtime:latest -o /tmp/qemu-runtime-image.tar
238+
239+
- name: Upload qemu-runtime image
240+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
241+
with:
242+
name: qemu-runtime-image-${{ matrix.arch }}
243+
path: /tmp/qemu-runtime-image.tar
244+
retention-days: 1
245+
175246
build-python-wheels:
176247
needs: changes
177248
if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch'
@@ -215,12 +286,14 @@ jobs:
215286
# ===========================================================================
216287

217288
e2e-tests:
218-
needs: [changes, build-controller-image, build-operator-image, build-exporterset-controller-image, build-python-wheels]
289+
needs: [changes, build-controller-image, build-operator-image, build-exporterset-controller-image, build-exporter-image, build-qemu-runtime-image, build-python-wheels]
219290
strategy:
220291
matrix:
221292
include: ${{ fromJson(needs.changes.outputs.e2e-matrix) }}
222293
runs-on: ${{ matrix.os }}
223-
timeout-minutes: 60
294+
# Includes ExporterSet QEMU coverage (TCG); flash/boot still skipped until #924.
295+
# Job > Ginkgo suite timeout (60m in e2e/lib/common.sh) for setup/log upload.
296+
timeout-minutes: 70
224297
steps:
225298
- name: Checkout repository
226299
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -320,7 +393,7 @@ jobs:
320393
artifact-name: e2e-logs-compat-old-controller
321394

322395
e2e-compat-old-client:
323-
needs: [changes, build-controller-image, build-operator-image, build-exporterset-controller-image, build-python-wheels]
396+
needs: [changes, build-controller-image, build-operator-image, build-exporterset-controller-image, build-exporter-image, build-qemu-runtime-image, build-python-wheels]
324397
# Skip on PRs — compat tests run in merge queue and workflow_dispatch.
325398
if: >-
326399
github.event_name != 'pull_request'

‎Makefile‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,12 @@ help:
3636
@echo " make docs-test - Run documentation tests"
3737
@echo ""
3838
@echo "End-to-end testing:"
39-
@echo " make e2e-setup - Setup e2e test environment (one-time)"
40-
@echo " make e2e-run - Run e2e tests (requires e2e-setup first)"
41-
@echo " make e2e - Same as e2e-run"
42-
@echo " make e2e-full - Full setup + run (for CI or first time)"
43-
@echo " make e2e-clean - Clean up e2e test environment (delete cluster, certs, etc.)"
39+
@echo " make e2e-setup - Setup e2e test environment (one-time)"
40+
@echo " make e2e-run - Run full e2e suite (includes ExporterSet QEMU)"
41+
@echo " make e2e - Same as e2e-run"
42+
@echo " make e2e-exporterset-qemu - Run ExporterSet QEMU e2e only"
43+
@echo " make e2e-full - Full setup + run (for CI or first time)"
44+
@echo " make e2e-clean - Clean up e2e test environment (delete cluster, certs, etc.)"
4445
@echo ""
4546
@echo "Per-project targets:"
4647
@echo " make build-<project> - Build specific project"
@@ -198,6 +199,12 @@ e2e-run:
198199
@echo "Running e2e tests..."
199200
@bash e2e/run-e2e.sh
200201

202+
# Focused local run of ExporterSet QEMU e2e (also covered by make e2e-run / CI).
203+
.PHONY: e2e-exporterset-qemu
204+
e2e-exporterset-qemu:
205+
@echo "Running ExporterSet QEMU e2e tests..."
206+
@GINKGO_LABEL_FILTER=exporterset-qemu bash e2e/run-e2e.sh
207+
201208
# Convenience alias for running e2e tests
202209
.PHONY: e2e
203210
e2e: e2e-run

‎controller/Containerfile.qemu-runtime‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ RUN dnf install --setopt=install_weak_deps=False -y \
2626
virtiofsd && \
2727
dnf clean all
2828

29+
# Default image USER is non-root; ExporterSet QEMU pods override target-runtime
30+
# to runAsUser 0 via Pod securityContext so QEMU can access devices and shared
31+
# volume paths owned by the exporter container.
2932
USER 65532:65532
3033

3134
ENTRYPOINT ["/shared/jumpstarter-exec"]

‎controller/hack/deploy_vars‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ BASEDOMAIN=${BASEDOMAIN:-"jumpstarter.${IP}.nip.io"}
88
IMG=${IMG:-quay.io/jumpstarter-dev/jumpstarter-controller:latest}
99
OPERATOR_IMG=${OPERATOR_IMG:-$(make -C deploy/operator --no-print-directory -s print-img 2>/dev/null || echo "quay.io/jumpstarter-dev/jumpstarter-operator:latest")}
1010
EXPORTER_SET_CONTROLLER_IMG=${EXPORTER_SET_CONTROLLER_IMG:-quay.io/jumpstarter-dev/jumpstarter-exporterset-controller:latest}
11+
EXPORTER_IMG=${EXPORTER_IMG:-quay.io/jumpstarter-dev/jumpstarter:latest}
12+
QEMU_RUNTIME_IMG=${QEMU_RUNTIME_IMG:-quay.io/jumpstarter-dev/virtual/qemu-runtime:latest}
1113

1214
# Determine endpoints based on NETWORKING_MODE and CLUSTER_TYPE
1315
if [ "${NETWORKING_MODE}" == "ingress" ]; then
@@ -46,4 +48,6 @@ export IMAGE_TAG
4648
export IMG
4749
export OPERATOR_IMG
4850
export EXPORTER_SET_CONTROLLER_IMG
51+
export EXPORTER_IMG
52+
export QEMU_RUNTIME_IMG
4953

‎controller/hack/deploy_with_operator.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,18 @@ fi
3333
load_image "${IMG}"
3434
load_image "${OPERATOR_IMG}"
3535
load_image "${EXPORTER_SET_CONTROLLER_IMG}"
36+
# Exporter + QEMU runtime images are required for ExporterSet QEMU e2e / samples.
37+
# Missing images are skipped so plain controller deploys still work.
38+
if container_image_exists "${EXPORTER_IMG}"; then
39+
load_image "${EXPORTER_IMG}"
40+
else
41+
echo -e "${YELLOW}Skipping load of exporter image (not present locally): ${EXPORTER_IMG}${NC}"
42+
fi
43+
if container_image_exists "${QEMU_RUNTIME_IMG}"; then
44+
load_image "${QEMU_RUNTIME_IMG}"
45+
else
46+
echo -e "${YELLOW}Skipping load of qemu-runtime image (not present locally): ${QEMU_RUNTIME_IMG}${NC}"
47+
fi
3648

3749
# Deploy the operator
3850
echo -e "${GREEN}Deploying Jumpstarter operator ...${NC}"

‎controller/hack/sample-x86_64-kind.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,3 +72,4 @@ spec:
7272
arch: x86_64
7373
smp: 1
7474
mem: 1G
75+
disk_size: 2G

‎controller/hack/utils‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ export K3S_KUBECONFIG=${K3S_KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}
1919

2020
# Color codes for terminal output
2121
export GREEN='\033[0;32m'
22+
export YELLOW='\033[1;33m'
2223
export NC='\033[0m' # No Color
2324

2425
# Get external IP address
@@ -76,6 +77,20 @@ load_image() {
7677
esac
7778
}
7879

80+
# Return 0 if the image exists in the local container tool store.
81+
container_image_exists() {
82+
local image=$1
83+
local tool=${CONTAINER_TOOL:-}
84+
if [ -z "${tool}" ]; then
85+
if command -v podman >/dev/null 2>&1; then
86+
tool=podman
87+
else
88+
tool=docker
89+
fi
90+
fi
91+
${tool} image inspect "${image}" >/dev/null 2>&1
92+
}
93+
7994
create_cluster() {
8095
_require_valid_cluster_type
8196
case "${CLUSTER_TYPE}" in

‎controller/internal/exporterset/exporterconfig.go‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,14 @@ const (
4444
// configVolumeName is the volume name for the ExporterConfig Secret.
4545
configVolumeName = "exporter-config"
4646

47-
// configMountPath is where the ExporterConfig Secret is mounted.
48-
configMountPath = "/etc/jumpstarter/exporters"
47+
// ExporterConfigMountPath is where the ExporterConfig Secret is mounted
48+
// inside the exporter container (QEMU provisioner and injectConfigVolume).
49+
ExporterConfigMountPath = "/etc/jumpstarter/exporters"
4950

50-
// exporterContainerName is the init-container name in the sidecar Pod.
51+
// configMountPath is the unexported alias used within this package.
52+
configMountPath = ExporterConfigMountPath
53+
54+
// exporterContainerName is the main container that runs jmp run.
5155
exporterContainerName = "exporter"
5256
)
5357

0 commit comments

Comments
 (0)