CI Workflow

Reference documentation for the GitHub Actions CI workflow.

Repeated E2E logic is factored into reusable shell scripts (hack/ci-*.sh) and a composite GitHub Action (.github/actions/setup-e2e-infra/), reducing duplication across every e2e-* job and the tempest job.

The build-e2e-images job centralises E2E image builds. It builds the images whose sources the pull request changed and pushes those to GHCR under run-scoped tags (e2e-${run_id}-<orig_tag>), then resolves every other image to the digest behind the tag main last published. Both kinds go into the job's image-map output. The tempest job and every e2e-* job except e2e-infra docker pull whatever the map names via the load-e2e-images composite action and re-tag the images to their canonical local references, so a pull request that touches one operator spends about four minutes here instead of the 39 the full build took. The cleanup-e2e-tags job prunes the run-scoped tags at the end of the workflow, with a nightly safety net in cleanup-images.yaml for cancelled runs (GH-310).

File Location

.github/workflows/ci.yaml

The file uses the .yaml extension (matching build-images.yaml and deploy-docs.yaml) and quotes the trigger key as "on" to prevent YAML boolean interpretation.

Trigger Events

The workflow triggers on two event types in three scopes:

Event	Scope	Description
push	branches: [main]	Runs on every push to the main branch
push	tags: ["v*"]	Runs on every v-prefixed tag push (triggers publish and release jobs)
pull_request	branches: [main], types: [opened, synchronize, reopened, labeled]	Runs on every pull request targeting main; the labeled type lets a ci:* label schedule jobs the paths alone did not

Most gate, test, and E2E jobs run only on pull_request events — they each carry a github.event_name == 'pull_request' guard. test-shell is the exception. It carries no guard and also runs on pushes to main and on tag pushes. A breakage that lands on main therefore turns main's own run red at the commit that caused it, instead of surfacing on the next PR branch. Otherwise, pushes to main and tag pushes (v*) run only the publish and release jobs (build-and-push, merge-operator-images, helm-push, helm-push-target-cluster, github-release): the merged commit's PR was already green, so the E2E suite is not re-run on push ("publish-only-on-merge"). On tag pushes the changes job forces all areas and all operators active, so every operator's images and charts are published regardless of which files the tagged commit touched.

Change classes and labels

A pull request runs the jobs whose inputs it changes. The changes job classifies the changed paths with dorny/paths-filter, and hack/ci-resolve-changes.sh turns those classes into one flag per job. A job's inputs are the code it tests, the images it loads, the suite it runs, and the scripts it calls. Nothing runs because Go code changed somewhere else.

Change class	Paths	What it schedules
<op> (one per operator)	operators/<op>/**	that operator's Go gates, its test and test-integration legs, its e2e-operator leg. keystone adds e2e-operator-upgrade; c5c3 adds the four ControlPlane jobs and e2e-orders-multicluster
nova_chart_egress	operators/nova/helm/nova-operator/templates/_helpers.tpl	the keystone and placement test and test-integration legs, whose unit tests check the nova chart's Keystone and Placement egress rules
image_<svc>	images/<svc>/**, nix/locks/*/<svc>/**, patches/<svc>/**; image_nova also images/nova-compute/**	that service's e2e-operator leg
image_ovn, image_tempest	the OVN and Tempest image sources (the Tempest locks included)	the ovn e2e-operator job; the Tempest image rebuild
image_proxy	images/keystone-federation-proxy/**, flake.nix, flake.lock, nix/lib/**, hack/ci-nix-build-e2e-images.sh	the keystone e2e-operator job and the federation proxy image rebuild
image_libvirt	images/libvirt/**, nix/pkgs/libvirt.nix, flake.nix, flake.lock, nix/lib/**, hack/ci-nix-build-e2e-images.sh	e2e-nova-libvirt and the libvirt image rebuild
go_common	internal/**, go.work, .golangci.yml, flake.nix, flake.lock, nix/lib/operator-image.nix, nix/lib/static-image.nix, nix/lib/image-defaults.nix, hack/ci-nix-build-e2e-images.sh	every operator's Go gates and operator image, every e2e-operator leg, e2e-operator-upgrade; a push publishes every operator image
images_base	flake.nix, flake.lock, nix/lib/**, nix/pkgs/**, releases/**, scripts/**, overrides/**, hack/ci-nix-build-e2e-images.sh, hack/ci-service-image-releases.sh	every service image, every Tempest image (one per release), every service operator's e2e leg
tempest_src and tempest_<svc>	images/tempest/**, tests/tempest/**, the Tempest scripts, the Tempest and plugin pins in releases/*/test-refs.yaml	tempest, narrowed to the services whose configuration changed; a pin bump runs every service
tests_e2e_<op>	tests/e2e/<op>/**, tests/e2e/<op>-operator/**, and tests/e2e/lib/** for the seven service operators that run a namespace-scoped-rbac suite (keystone, horizon, glance, placement, barbican, cinder, nova)	that operator's e2e-operator leg
tests_controlplane, tests_controlplane_sso, tests_external_keystone	the three ControlPlane suites; tests_controlplane also covers deploy/kind/hypervisor-operator-fixtures/**, whose fixtures.yaml the full-chain suite applies	the job that runs that suite
tests_chaos, tests_multicluster, tests_orders_multicluster, tests_operator_upgrade, tests_prometheus	each suite's own tree; the two multicluster classes also cover deploy/target-cluster/**	the job that runs it
e2e_infra	tests/e2e/infrastructure/**	e2e-infra
e2e_shared, e2e_openbao, ci_plumbing, makefile	the deploy stack, the infra scripts, the shared composite actions, the workflow file, the remaining CI helpers	the canary: e2e-infra plus the keystone e2e-operator leg. deploy/openbao/** adds the barbican leg
actionlint	.github/**	actionlint
docs, helm, target_cluster_chart	as before	docs, helm-validate, helm-push-target-cluster
publish_legacy	the wide paths a push publishes from	push events only, ignored on pull requests

The canary is the deliberate gap. A change to the substrate every e2e job stands on could break any of them, and running all of them costs about 450 runner minutes. Instead it runs the infrastructure suite and one operator leg, and a label asks for the rest.

Labels

Six labels add jobs. None of them ever removes one.

Label	Schedules
ci:full	everything, and builds every image
ci:tempest	the Tempest legs of the services the pull request touches, plus the legs that deploy a changed ovn or placement operator (neither has a leg of its own), or the keystone legs when it touches none
ci:chaos	all four e2e-chaos legs. run-chaos is an alias
ci:controlplane	e2e-controlplane, e2e-controlplane-sso, e2e-external-keystone, e2e-autoscaling, e2e-orders-multicluster
ci:multicluster	e2e-multicluster, e2e-orders-multicluster
ci:measure-sizing	e2e-controlplane, e2e-controlplane-sso and all twelve Tempest legs, each with the sizing measurement. Neither ci:full nor a tag push implies it

The labeled trigger means a label applied after the last push starts a run that evaluates the new label set. A label outside this set resolves to nothing: the changes job emits noop=true, every gated job skips, and the run lands in a concurrency group of its own so it does not cancel the pipeline in flight. See Concurrency.

On a v* tag push every flag is forced on. On a push to main the operator matrix keeps its publish semantics, driven by the publish_legacy class, so what a merge publishes is unchanged.

Sizing measurement

The ci:measure-sizing label makes the changes job emit measure-sizing=true. e2e-controlplane, e2e-controlplane-sso and every tempest leg then record what a VerticalPodAutoscaler recommender recommends for the workloads in openstack, using four additions to the job:

Addition	What it does
WITH_VPA on Setup E2E infrastructure	true under the label, empty otherwise. hack/deploy-infra.sh installs metrics-server and the recommender of deploy/kind/vpa, and removes the MariaDB CRD's scale subresource so the database StatefulSet can carry a VPA
Start the sizing measurement	Right after the setup: starts hack/ci-vpa-recommendations.sh watch _output/sizing openstack in the background. The watch applies an updateMode: "Off" VPA for every workload and saves each recommendation once a minute, so a workload a suite deletes keeps its last one
Collect the sizing measurement	After the last suite step, always(): stops the watch, takes a last snapshot and writes recommendations.tsv and recommendations.md, printing the table to the log. It fails the job when nothing was recorded; gh run rerun --failed repeats such a leg. The tempest legs set MEASURE_LEG=<service>-<release> for the TSV's comment line
Upload the sizing measurement	always(): uploads _output/sizing/ as sizing-e2e-controlplane, sizing-e2e-controlplane-sso or sizing-tempest-<service>-<release>, kept 14 days

Without the label the three steps skip and the banner of the setup step prints VPA recommender : false. Sizing Calibration describes how the artifacts turn into the sizing figures. The node budget of Link 6z leaves the metrics-server and recommender pods out, since the devstack it measures installs neither.

Environment Variables

Top-level environment variables centralise registry configuration and pin tool versions for CI reproducibility:

env:
  REGISTRY: ghcr.io
  IMAGE_PREFIX: ghcr.io/c5c3
  KIND_CLUSTER: cobaltcore
  KIND_VERSION: v0.33.0
  CONTROLLER_GEN_VERSION: v0.22.0
  GOFUMPT_VERSION: v0.12.0
  GOLANGCI_LINT_VERSION: v2.14.0

REGISTRY and IMAGE_PREFIX are referenced by build-e2e-images, by the jobs that load its images (the packages: read row under Permissions), and by the publish jobs (build-and-push, merge-operator-images, helm-push, helm-push-target-cluster) to construct image names and registry URLs. KIND_CLUSTER is the single source of truth for every E2E job's kind cluster name (mirroring the CLUSTER_NAME default in hack/deploy-infra.sh). KIND_VERSION is the kind binary every E2E job creates its cluster with, passed to the create-kind-cluster composite action, which hands it to helm/kind-action's version input. That action defaults to v0.31.0, whose kindnetd does not enforce NetworkPolicy egress against the post-DNAT destination; pinning it keeps CI on the enforcing build and in lockstep with the KIND_VERSION in hack/install-test-deps.sh that local development installs. Renovate groups the two pins so they bump in one pull request, and tests/unit/renovate/workflow_pins_custommanager_test.sh asserts they agree. CONTROLLER_GEN_VERSION is used by verify-codegen to pin controller-gen to a specific version. GOFUMPT_VERSION is used by format-check to pin gofumpt to a specific version; the same version is mirrored in the Makefile (GOFUMPT_VERSION ?= v0.12.0) so that make fmt and make format-check use a consistent version locally. GOLANGCI_LINT_VERSION pins the golangci-lint binary installed by the lint job and keys its analysis cache. setup-envtest is installed via @release-0.23 because the sub-module does not publish its own release tags.

Permissions

Top-level permissions are restricted to least privilege:

permissions:
  contents: read

Jobs that need elevated access declare per-job permissions: blocks:

Job	Additional Permissions	Reason
build-e2e-images	packages: write	Push the E2E images to GHCR under run-scoped tags
e2e-operator, e2e-operator-upgrade, e2e-chaos, e2e-prometheus, e2e-controlplane, e2e-controlplane-sso, e2e-autoscaling, e2e-external-keystone, e2e-multicluster, e2e-orders-multicluster, e2e-ovn-overlay, e2e-nova-libvirt, tempest	packages: read	Pull the E2E images from GHCR
cleanup-e2e-tags	packages: write	Delete the run-scoped tags from GHCR
build-and-push	packages: write	Push per-platform operator image digests to GHCR
merge-operator-images	packages: write	Push final multi-arch operator image manifest list
helm-push	packages: write	Push Helm charts to GHCR OCI registry
helm-push-target-cluster	packages: write	Push the target-cluster-access chart to GHCR OCI registry
github-release	contents: write	Create GitHub Releases

Job Dependency DAG

The workflow defines 38 jobs. changes and test-shell run on every event. The gate, test and E2E jobs carry a github.event_name == 'pull_request' guard, and cleanup-e2e-tags follows build-e2e-images, which runs on a pull request only. The publish and release jobs run on push events only.

The figure groups the jobs into stages and draws the order of the stages. The table lists every job with its stage and its needs: list, and links the section that describes the job.

Job	Stage	needs
changes	Change detection	none
lint	Gates	changes
format-check	Checks	changes
shellcheck	Gates	changes
feature-ids	Checks	changes
review-markers	Checks	changes
test-shell	Every event	none
verify-invalid-cr-fixtures	Gates	changes
chainsaw-lint	Gates	changes
actionlint	Checks	changes
test	Gates	changes
test-integration	Gates	changes
test-race	Checks	changes
govulncheck	Checks	changes
verify-codegen	Gates	changes
docs	Checks	changes
helm-validate	Checks	changes
e2e-infra	Infrastructure suite	changes
build-e2e-images	Image build	changes, lint, shellcheck, test, test-integration, verify-codegen, verify-invalid-cr-fixtures, chainsaw-lint
e2e-operator	E2E fan-out	changes, build-e2e-images
e2e-operator-upgrade	E2E fan-out	changes, build-e2e-images
e2e-chaos	Chaos	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images, e2e-operator
e2e-prometheus	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-controlplane	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-controlplane-sso	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-autoscaling	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-external-keystone	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-multicluster	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-orders-multicluster	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-ovn-overlay	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
e2e-nova-libvirt	E2E fan-out	changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images
tempest	Tempest	changes, build-e2e-images, e2e-infra, e2e-operator, e2e-chaos, e2e-prometheus
cleanup-e2e-tags	Cleanup	changes, build-e2e-images, e2e-operator, e2e-operator-upgrade, e2e-chaos, e2e-prometheus, e2e-controlplane, e2e-controlplane-sso, e2e-autoscaling, e2e-external-keystone, e2e-multicluster, e2e-orders-multicluster, e2e-ovn-overlay, e2e-nova-libvirt, tempest
build-and-push	Publish images	changes
merge-operator-images	Publish images	changes, build-and-push
helm-push	Publish charts	changes
helm-push-target-cluster	Publish charts	changes
github-release	Release	changes, merge-operator-images, helm-push

e2e-infra, the jobs of the E2E fan-out, e2e-chaos and tempest bring up a kind cluster through the create-kind-cluster and setup-e2e-infra composite actions and dump diagnostics through hack/ci-dump-diagnostics.sh. They run on the self-hosted runners, as do test-integration and build-e2e-images.

Jobs

changes

Classifies the paths an event changed and resolves them into the outputs the other jobs read: one flag per job, the change sets build-e2e-images builds from, and the matrices of the test, Tempest, publish and cleanup jobs. Every other job except test-shell lists it in needs.

Dependencies: none (the job has no needs:) Condition: none. The job has no if: and runs on every event that starts the workflow: a push to main, a v* tag push, and a pull_request event of type opened, synchronize, reopened or labeled.

The job runs on ubuntu-latest with timeout-minutes: 8.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	dorny/paths-filter@v4	Step id filter. Matches the changed paths against the filters: block and writes one output per change class, true when a changed path matches
3	Resolve effective changes	Step id result. Runs hack/ci-resolve-changes.sh. The step env passes each class as FILTER_<class>, the pull request's label names as the JSON array PR_LABELS, the event as EVENT_NAME, EVENT_ACTION and EVENT_LABEL, and the operator lists ALL_OPERATORS, SERVICE_OPERATORS and CANARY_OPERATOR: keystone. The script writes noop, the job flags, the change sets, tempest-services, test-targets and e2e-operators
4	Generate Tempest release matrix	Step id tempest-matrix. Runs hack/ci-generate-tempest-matrix.sh with TEMPEST_SERVICES set to the tempest-services list of step 3, joined with spaces. The script scans releases/*/ and writes tempest-releases with one entry per selected service and release. An empty selection emits every service. The script fails when the tests/tempest/<service>-<slug>/ directory of any service is missing for any release, whatever the selection
5	Generate cleanup package matrix	Step id cleanup-matrix. Runs hack/ci-generate-cleanup-matrix.sh, which derives the GHCR package names from images/*/ and from every operators/*/ directory that holds a go.mod. It writes cleanup-packages and cleanup-e2e-packages. The job exports the second
6	Shard the e2e-operator matrix	Step id e2e-operator-legs. A jq filter turns e2e-operators into an include list and splits nova into two entries, shard: "1" and shard: "2". It writes legs

Step 3 has three special cases. A labeled event whose label is neither a ci:* label nor run-chaos resolves to noop=true: every other flag is false, the change sets are [], and test-targets and e2e-operators hold the placeholder __none__. The ci:full label and a v* tag push force every job flag on and select every operator. measure-sizing is the exception: only the ci:measure-sizing label sets it. On a push to main the script sets build-e2e-images to false, and e2e-operators holds every operator when the go_common or publish_legacy class changed, otherwise the operators whose own class or image_<op> class changed. Change classes and labels describes the classes and the labels.

The outputs: block of the job lists its keys in the order below. Step 3 writes all of them except e2e-operator-legs, tempest-releases and cleanup-e2e-packages.

The flags come first. Each one is true or false:

Output	Meaning
noop	true for a labeled event whose label does not steer CI. shellcheck, feature-ids, review-markers, verify-invalid-cr-fixtures and chainsaw-lint skip on true
go	Gates lint, format-check, test, test-integration, test-race, govulncheck and verify-codegen
docs	Gates docs
helm	Gates helm-validate
target-cluster-chart	Gates helm-push-target-cluster
e2e-infra	Gates e2e-infra
e2e-chaos	Gates e2e-chaos
e2e-prometheus	Gates e2e-prometheus
e2e-controlplane	Gates e2e-controlplane
e2e-controlplane-sso	Gates e2e-controlplane-sso
e2e-external-keystone	Gates e2e-external-keystone
e2e-autoscaling	Gates e2e-autoscaling
e2e-multicluster	Gates e2e-multicluster
e2e-orders-multicluster	Gates e2e-orders-multicluster
e2e-ovn-overlay	Gates e2e-ovn-overlay
e2e-nova-libvirt	Gates e2e-nova-libvirt
e2e-operator-upgrade	Gates e2e-operator-upgrade
tempest	Gates tempest
measure-sizing	true under the ci:measure-sizing label. e2e-controlplane, e2e-controlplane-sso and tempest read it for WITH_VPA and for their three sizing steps (see Sizing measurement)
actionlint	Gates actionlint

The change sets and the build flags follow. build-e2e-images passes the first four to hack/ci-resolve-e2e-images.sh:

Output	Meaning
changed-operators	JSON array of the operators whose Go code is affected. A Makefile change adds the canary operator. Passed as CHANGED_OPERATORS
changed-services	JSON array of the operators whose service image sources changed. Passed as CHANGED_SERVICES
changed-tempest	true when the image_tempest or images_base class changed. Passed as CHANGED_TEMPEST
changed-proxy	true when the image_proxy class changed. Passed as CHANGED_PROXY
changed-ovn	true when the image_ovn class changed: the ovn and backup-shifter image sources, flake.nix, flake.lock, nix/lib/** or hack/ci-nix-build-e2e-images.sh. Passed as CHANGED_OVN
changed-libvirt	true when the image_libvirt class changed: the libvirt image sources, nix/pkgs/libvirt.nix, flake.nix, flake.lock, nix/lib/** or hack/ci-nix-build-e2e-images.sh. Passed as CHANGED_LIBVIRT
build-e2e-images	true when has-e2e-operators, tempest or an e2e-* flag other than e2e-infra is true. Gates build-e2e-images
has-e2e-operators	true when e2e-operators holds at least one operator. Gates e2e-operator, build-and-push and helm-push

The matrices come last:

Output	Meaning
e2e-operators	{"operator":[...]}. The matrix of merge-operator-images and helm-push and the operator dimension of build-and-push. github-release loops over it to package the charts
e2e-operator-legs	Written by step 6: the operators of e2e-operators as include entries, with nova split into shards 1 and 2. The matrix of e2e-operator
test-targets	{"target":[...]} with common and operator names. The matrix of test and test-integration
tempest-releases	Written by step 4: {"include":[...]} with one entry per service and release. The matrix of tempest
cleanup-e2e-packages	Written by step 5: JSON array of the GHCR packages that may receive run-scoped tags, every published package. The package matrix of cleanup-e2e-tags

lint

Runs golangci-lint using the project's .golangci.yml configuration.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	actions/cache@v6	Persists the golangci-lint analysis cache, keyed on the Go and golangci-lint versions
4	golangci/golangci-lint-action@v9	Installs golangci-lint binary (install-only: true); version pinned via GOLANGCI_LINT_VERSION (v2.14.0)
5	make lint	Runs golangci-lint per module via the Makefile

The golangci-lint-action@v9 step is used with install-only: true, which installs the pinned golangci-lint binary (and caches it) without running lint. The actual linting is delegated to make lint, which cds into each module directory and runs golangci-lint run ./... — a necessary pattern for Go multi-module workspaces. The actions/setup-go@v7 step is required because install-only mode does not set up Go internally.

Enabled linters (12 total, configured in .golangci.yml):

Linter	Category	Description
errcheck	correctness	Checks for unchecked errors in Go code
gocritic	style	Provides diagnostics for bugs, performance, and style issues
govet	correctness	Reports suspicious constructs, roughly equivalent to go vet
ineffassign	correctness	Detects assignments to existing variables that are never used
staticcheck	correctness	Comprehensive static analysis rules from the staticcheck suite
unused	correctness	Checks for unused constants, variables, functions, and types
bodyclose	resource-leak	Checks whether HTTP response bodies are closed successfully
errorlint	correctness	Validates Go 1.13+ error wrapping patterns (%w, errors.Is, errors.As)
exhaustive	correctness	Checks exhaustiveness of enum switch statements
gosec	security	Inspects source code for security problems (hardcoded credentials, weak crypto, unsafe operations)
nilerr	correctness	Finds code that returns nil even after checking that an error is not nil
noctx	correctness	Detects HTTP requests and TLS dials missing context.Context propagation

Generated code matching zz_generated.*.go is excluded from all lint checks via the exclusions.paths configuration.

format-check

Verifies all Go files conform to gofumpt formatting. gofumpt is a strict superset of gofmt — it applies all standard gofmt rules plus additional formatting conventions for consistency. Detects non-conforming files and prints a unified diff showing the required changes, so developers can identify and fix formatting issues without guessing.

Only git-tracked Go files are checked (git ls-files '*.go') to avoid unexpected failures on generated, vendored, or tooling code that may not follow gofumpt conventions.

The same version and check logic are available locally via the Makefile: make install-gofumpt installs the pinned version, make format-check mirrors the CI check, and make fmt applies formatting to all tracked Go files. The Makefile targets use xargs without the -r flag (unlike CI) for macOS portability — BSD xargs does not support -r. This is safe because the repository always contains tracked .go files.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.go == 'true'

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	go install mvdan.cc/gofumpt@${{ env.GOFUMPT_VERSION }}	Installs gofumpt at the pinned version (v0.12.0), unless the actions/cache@v6 step that runs before it restored the binary
4	git ls-files '*.go' | xargs -r gofumpt -l	Lists non-conforming tracked Go files; on failure, prints unified diff and exits 1

The check uses git ls-files '*.go' | xargs -r gofumpt -l to collect non-conforming files from tracked sources only. If any are found, their paths are printed along with a unified diff (gofumpt -d), and the job exits 1. The -r flag prevents xargs from running gofumpt when no Go files are piped (GNU coreutils, available on ubuntu-latest).

Timeout: 8 minutes.

shellcheck

Validates shell scripts with shellcheck to catch scripting issues early. The shellcheck binary is pre-installed on ubuntu-latest runners.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	make shellcheck	Runs shellcheck --severity=warning over hack/*.sh and the operator rotation scripts (operators/*/internal/controller/scripts/*.sh)

Timeout: 8 minutes.

feature-ids

Verifies the whole tracked tree (code, tests, CI, scripts, docs) is free of internal feature/requirement IDs. Runs on every pull request except a no-op label run — not path-filtered — so a stray ID added anywhere is caught. This job folds in the former docs-only check.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	make check-feature-ids	Greps the tracked tree for internal feature/requirement ID patterns and fails on any hit

Timeout: 8 minutes.

review-markers

Verifies that no new unresolved reviewer marker enters the tracked tree. A reviewer marker is a review question that was committed in place of its answer, such as a comment that starts with Reviewer: please …. The job tolerates the markers the tree already carries through the per-file baseline in scripts/review-markers-baseline.txt, and it fails only when a file holds more markers than its baseline count. No path filter applies.

Dependencies: needs: [changes]Condition: Runs on pull_request events when the noop output of changes is not true.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	make check-review-markers	Runs scripts/check-no-review-markers.sh, which counts the lines that match please[[:space:]]+verify (case-insensitive) in each git-tracked file and compares each count with the file's baseline entry

The script takes its file list from git ls-files, so git-ignored build output is out of scope. It also skips .planwerk/, .claude/, a path named architecture, the script and the baseline file themselves, and the lockfiles go.sum, go.work.sum, package-lock.json and Chart.lock.

A file with more matches than its baseline entry fails the job with exit code 1, and so does a file with matches and no entry. The script prints each match as file:line:match. A file with fewer matches than its entry only produces an info: line that asks to tighten the baseline with scripts/check-no-review-markers.sh --update-baseline. Exit code 2 means the script ran outside a git work tree or the baseline file is missing. The scan works line by line, so it misses a marker whose two words sit on different lines.

Runs on ubuntu-latest. Timeout: 8 minutes.

verify-invalid-cr-fixtures

Enforces the canonical-scaffold contract for the invalid-CR Chainsaw fixtures. Runs _generate.py --check (drift mode) and the test_generate.py unit suite (FIXTURES count + chainsaw-test.yaml cross-reference) so a hand-edit to any 02-…/03-…/…/12-*.yaml fixture, or a rename or removal that desynchronises FIXTURES from chainsaw-test.yaml, fails the build before the heavy cluster-bound e2e-operator job runs. Not path-filtered because the check is sub-second and python3 is preinstalled on ubuntu-latest runners.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	make verify-invalid-cr-fixtures	Runs _generate.py --check and test_generate.py

Timeout: 8 minutes.

chainsaw-lint

Schema-lints every Chainsaw test (tests/**/chainsaw-test.yaml) and configuration (tests/{e2e,e2e-chaos}/chainsaw-config.yaml) via chainsaw lint so typos, removed fields, or schema drift after a chainsaw version bump fail fast — before the cluster-bound e2e-operator and e2e-chaos jobs spin up a kind cluster. Not path-filtered because no cluster is needed: chainsaw is restored from the shared testdeps cache via the setup-test-deps composite action, the same one consumed internally by setup-e2e-infra. A schema break therefore surfaces in needs.*.result for both build-e2e-images and e2e-chaos.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	./.github/actions/setup-test-deps	Restores the testdeps cache and runs make install-test-deps (puts chainsaw on PATH)
3	make chainsaw-lint	Runs chainsaw lint test -f and chainsaw lint configuration -f over every matching file under tests/

Timeout: 8 minutes.

actionlint

Static-lints every workflow file. actionlint resolves the expression contexts, so a needs.changes.outputs.<name> that no job exports is caught here instead of evaluating to the empty string and leaving a job unscheduled for good. It also runs shellcheck over every run: script.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.actionlint == 'true'Path filter: .github/**

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	Install actionlint	Downloads the pinned release, verifies ACTIONLINT_SHA256 with sha256sum --check --strict, installs to /usr/local/bin
3	actionlint -no-color	Lints every workflow under .github/workflows/

The binary is pinned and checksummed the way verify-container-images.yaml installs yq. Renovate bumps ACTIONLINT_VERSION; the checksum then fails until it is updated alongside, which is the intended signal.

actionlint validates runs-on: against the labels GitHub itself provides plus the built-in self-hosted. The workflows use only those, so no .github/actionlint.yaml exists; a custom runner pool would have to be declared there first or the lint fails on a runner that exists.

Three run: scripts carry inline # shellcheck disable= directives for deliberate word splitting: the release-list loops in e2e-operator (SC2086), the digest expansion in merge-operator-images (SC2012, SC2046), and the version appends in build-images.yaml (SC2129).

Timeout: 8 minutes.

test-shell

Runs every shell unit test under tests/unit/ (hack/, deploy/, renovate/, docs/, ci/, images/). The job carries no event guard, so it runs on pull requests and on the push runs for main and v* tags: a breakage that lands on main fails main's own run. Tests read repo files only (no cluster, no untrusted input) and the job finishes in about a minute on a cold runner. Tests that depend on yq or kustomize are written to skip gracefully when those tools are missing; the job installs kustomize explicitly so the deploy/ overlay assertions run their full check set (yq is preinstalled on ubuntu-latest).

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	Install kustomize	Downloads the pinned kustomize binary into /usr/local/bin
3	make test-shell	Iterates every tests/unit/**/*_test.sh and aggregates exit status

Timeout: 8 minutes.

test

Runs unit tests with a matrix strategy resolved per pull request out of [common, keystone, c5c3, horizon, glance, placement, barbican, ovn, neutron, cinder, nova]. Each matrix leg tests a single target — either internal/common or one operator — producing a single coverage profile uploaded to Codecov under a dedicated flag.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	make test-common or make test-operator	Runs unit tests for the matrix target
4	codecov/codecov-action@v7	Uploads coverage profile with target-specific flag

Matrix strategy:

strategy:
  fail-fast: false
  matrix: ${{ fromJson(needs.changes.outputs.test-targets) }}

The resolver builds that list: common when shared Go code or the Makefile changed, plus every operator whose own code changed. A pull request touching one operator runs one leg.

The common leg runs make test-common (producing cover-unit-common.out). Operator legs run make test-operator OPERATOR=<target> (producing cover-unit-<operator>.out). This deduplicates common coverage into a single leg instead of uploading it under each operator flag.

Coverage upload:

files: cover-unit-${{ matrix.target }}.out
flags: unit-${{ matrix.target }}

The if: always() condition ensures coverage is uploaded even when tests fail, so partial coverage data is not lost.

test-integration

Runs envtest-based integration tests with a matrix strategy resolved per pull request out of [common, keystone, c5c3, horizon, glance, placement, barbican, ovn, neutron, cinder, nova] and coverage uploaded to Codecov. Requires setup-envtest to download kubebuilder assets (kube-apiserver, etcd) for the test API server.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	go install setup-envtest@release-0.23	Installs envtest asset downloader (pinned to release branch)
4	make test-integration-common or make test-integration	Runs integration tests for the matrix target
5	codecov/codecov-action@v7	Uploads coverage with integration-<target> flag

Matrix strategy:

strategy:
  fail-fast: false
  matrix: ${{ fromJson(needs.changes.outputs.test-targets) }}

The resolver builds that list: common when shared Go code or the Makefile changed, plus every operator whose own code changed. A pull request touching one operator runs one leg.

The common leg runs make test-integration-common (producing cover-integration-common.out), which tests ./internal/common/... with -tags=integration. Operator legs run make test-integration OPERATOR=<target> (producing cover-integration-<operator>.out). Both targets set KUBEBUILDER_ASSETS via $(SETUP_ENVTEST) use <pinned-k8s-version> -p path.

Timeout: 35 minutes (longer than unit tests to account for envtest startup).

test-race

Runs all Go unit tests with the race detector enabled to catch data races in concurrent operator code — reconcilers, watches, informer caches. Separate from the main test job because the race detector adds 2–5x overhead. Uses -count=1 to disable test caching, since race conditions are non-deterministic and cached results could mask real races.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.go == 'true'Path filter: Go source files (same filter as test and test-integration)

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	make test-race RACE_FLAGS="-count=1"	Delegates to the Makefile so the module list stays in sync

CI delegates to make test-race so the list of modules under race testing is defined in one place (the Makefile's OPERATORS variable and internal/common). RACE_FLAGS="-count=1" disables test caching — race conditions are non-deterministic, so cached results could mask real races. No continue-on-error or if: always() — a detected data race fails the job immediately.

This job runs independently and does not appear in any other job's needs: array. It is not on the critical path for E2E or publish jobs, so race detector overhead does not slow down the primary feedback loop. The corresponding local command is make test-race (which omits -count=1 via the default empty RACE_FLAGS for developer convenience).

Timeout: 30 minutes (accommodates 2–5x race detector overhead).

govulncheck

Scans all Go modules for reachable vulnerabilities using govulncheck, the official Go vulnerability scanner maintained by the Go team. Unlike dependency-list scanners, govulncheck analyses call graphs to detect only vulnerabilities in code paths that are actually reachable — reducing false positives. Catches supply-chain vulnerabilities at the PR stage, before container images are built.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.go == 'true'Path filter: Go source files (same filter as test, test-integration, and test-race)

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	go install golang.org/x/vuln/cmd/govulncheck@latest	Installs the latest govulncheck binary
4	make govulncheck	Delegates to hack/ci-govulncheck.sh, which scans internal/common and all $(OPERATORS) modules with an explicit allowlist

govulncheck uses @latest intentionally — unlike other pinned tools (controller-gen, gofumpt), pinning govulncheck to an old version defeats the purpose of vulnerability scanning because the vulnerability database is updated frequently. This is a deliberate deviation from the general pinning policy, justified by the security tool's nature.

The CI step delegates to make govulncheck, which runs hack/ci-govulncheck.sh over internal/common and each operator in the $(OPERATORS) Makefile variable. govulncheck has no native suppression flag, so the wrapper runs it in JSON mode per module, keeps only the reachable symbol-level findings (the ones that fail the default text report), and drops any whose advisory ID appears in the script's ALLOWLIST map. The build fails if, and only if, a reachable finding survives the allowlist — matching govulncheck's normal failure semantics while letting the project ride out advisories that have no fix and no real exposure. Every allowlist entry carries a one-line justification, and if an allowlisted advisory is no longer reported, the wrapper prints a notice so the stale entry can be removed. Dependencies with known CVEs whose vulnerable functions are not called in project code are reported as informational but do not fail the job.

This job runs independently and does not appear in any other job's needs: array. It is not on the critical path for E2E or publish jobs, matching the test-race pattern. When a new Go module is added to go.work, the OPERATORS variable in the Makefile must be updated with the new module name. The verification test (tests/ci/verify_govulncheck_modules.sh) catches drift between go.work and the Makefile automatically.

Timeout: 15 minutes.

verify-codegen

Verifies that generated code (CRD, webhook and RBAC manifests, deepcopy functions, the chart RBAC rules templates) is committed and up-to-date, and that every workspace member's go.mod/go.sum is tidy. This is a gate job — it blocks merge alongside lint, test, and shellcheck.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	make verify-go-tidy	Fails if any module's go.mod/go.sum differs from what go mod tidy would write
4	make nix-lock-check-operators	Fails, without Nix, if an operators/<op>/vendor.json is missing or no longer matches the go.mod and go.sum files it derives from (Operator images)
5	go install controller-gen@${{ env.CONTROLLER_GEN_VERSION }}	Installs the pinned code generator
6	make manifests && make generate	Regenerates CRD, webhook and RBAC (config/rbac/role.yaml) manifests and deepcopy functions
7	make verify-crd-sync	Verifies Helm chart CRD copies match controller-gen output
8	make verify-helm-rbac	Verifies each chart's templates/_rbac-rules.tpl matches the regenerated config/rbac/role.yaml
9	git diff --exit-code	Fails if any files changed (stale generated code)

When the diff check fails, the job produces a GitHub Actions ::error:: annotation with instructions to run make manifests && make generate locally and commit the result.

docs

Builds the VitePress documentation site to catch broken links and build errors.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.docs == 'true'Path filter: docs/**, package.json, package-lock.json

Step	Action	Details
1	actions/checkout@v7	Full history (fetch-depth: 0) for git-based features
2	actions/setup-node@v7	Node.js 24, npm cache enabled
3	npm ci	Installs dependencies from lockfile
4	npm run docs:build	Builds the documentation site

helm-validate

Validates Helm chart structure, template rendering, and unit tests for every operator chart and the operator-library testbed without requiring a cluster. Verifies the generated values.schema.json and _rbac-rules.tpl files are in sync with their sources, vendors the shared operator-library subchart, then runs helm lint, helm template with six value override scenarios, and helm unittest for each chart to catch regressions at PR time. The chart list is the operators/*/helm/*-operator and operators/*/helm/*-testbed globs, so a new operator chart in that layout is validated without editing the job.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.helm == 'true'Path filter: operators/*/helm/**, deploy/target-cluster/**, hack/gen-helm-values-schema.py, hack/gen-helm-rbac-rules.py, Makefile (forced true on v* tag pushes)

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	azure/setup-helm@v5	Installs Helm CLI (SHA-pinned)
3	helm plugin install helm-unittest	Installs helm-unittest plugin (pinned to v1.0.3)
4	make verify-helm-schema	Fails if any chart's values.schema.json has drifted from the shared generator
5	make verify-helm-rbac	Fails if any chart's templates/_rbac-rules.tpl or templates/_webhook-rbac-rules.tpl has drifted from its committed config/rbac/role.yaml, or if a webhook rule grants a read the manager rules do not
6	make helm-deps	Vendors the operator-library subchart into each consumer chart's charts/
7	helm lint	Validates chart structure and syntax for every chart
8	helm template (7 scenarios)	Renders each chart with value overrides to catch broken conditionals and invalid YAML
9	helm unittest	Runs the unit test suites under each chart's tests/ directory
10	Validate target-cluster-access chart	Runs helm lint, helm template (3 scenarios) and helm unittest on deploy/target-cluster/target-cluster-access

Template scenarios (step 8), run against each chart:

Scenario	Values	Purpose
1 — default values	(none)	Validates baseline rendering with chart defaults
2 — webhook disabled	webhook.enabled=false	Validates conditional exclusion of webhook resources
3 — external service account	serviceAccount.create=false, serviceAccount.name=existing-sa	Validates ServiceAccount conditional logic
4 — custom resources	resources.limits.cpu=100m, resources.limits.memory=64Mi	Validates resource override wiring
5 — namespace-scoped RBAC	rbac.namespaceScoped=true, webhook.enabled=false	Validates Role/RoleBinding rendering instead of ClusterRole/ClusterRoleBinding. A chart that refuses the mode by design (ovn-operator, neutron-operator, c5c3-operator) fails the render with the documented is not supported by <chart> message, which the job accepts; any other failure fails the job
6 — node placement	priorityClassName=cobaltcore-platform, nodeSelector.role=platform, one tolerations entry	Validates that the generated schema admits the placement keys and that they render
7 — standalone webhook	webhook.standalone=true	Validates the standalone webhook release: the ClusterRole names no secrets, and the Deployment args carry --enable-controllers=false and no --leader-elect

Unit test suites (step 9): the shared templates are tested once, in the operator-library testbed (operators/shared/helm/operator-library-testbed/tests/); each operator chart's own tests/ suites cover what that chart adds.

Chart	Test File	Key Assertions
testbed	deployment_test.yaml	Image, replicas, resources, nodeSelector/tolerations/priorityClassName, securityContext, probes, args (including the webhook.standalone args), extraArgs/extraEnv, conditional webhook volume mount
testbed	networkpolicy_test.yaml, certificate_test.yaml, service_test.yaml, serviceaccount_test.yaml, clusterrolebinding_test.yaml, rolebinding_test.yaml, pdb_test.yaml, servicemonitor_test.yaml, release_namespace_test.yaml	The shared manifests, their conditionals and the release-namespace threading
testbed	clusterrole_test.yaml, role_test.yaml	The shared RBAC templates: rendering per scope, the webhook rule set under webhook.standalone, the webhook guard, the hook-less default
testbed	schema_validation_test.yaml	The shared values schema: type, enum, range, quantity and conditional constraints, including the webhook.standalone rule
operator	clusterrole_test.yaml, role_test.yaml	Wiring of the generated rules, the grants whose restriction is deliberate, the webhook rule set and its rule count under webhook.standalone, a chart's refusal of namespace-scoped mode
operator	deployment_test.yaml	The chart's image default and what the chart adds through its hooks (keystone federation flag, c5c3 barbican-operator identity)
operator	webhook_test.yaml	Mutating/Validating configs for the chart's CR kinds when enabled, absent when disabled
operator	schema_validation_test.yaml	The chart ships an enforced schema; the keys its values.schema.extras.json adds

Timeout: 15 minutes.

e2e-infra

End-to-end infrastructure deployment and Chainsaw test. Deploys the full infrastructure stack (Flux, cert-manager, MariaDB, ESO, OpenBao) to a kind cluster and validates health of all operators, CRs, and ExternalSecrets.

Dependencies: needs: [changes]Condition: if: github.event_name == 'pull_request' && needs.changes.outputs.e2e-infra == 'true'

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
4	setup-e2e-infra composite action	Installs Flux CLI, test deps, and deploys infra stack
5	chainsaw test	Runs E2E tests from tests/e2e/infrastructure/
6	make deploy-infra (re-run)	Unchanged-parameter re-run (no SKIP_KIND_CREATE) — exercises the script's existing-cluster detection
7	chainsaw test --report-name chainsaw-report-rerun	Re-runs the full infrastructure suite to prove the healthy stack is left unchanged
8	make deploy-infra with WITH_METRICS_SERVER=true and WITH_NFS=true	Additive re-run — the script's Phase-3 wait gates the new metrics-server and csi-driver-nfs HelmReleases on Ready, and its Step-3 rollout wait gates Deployment/nfs-server
9	kubectl get deployment nfs-server -n openstack + kubectl get helmrelease csi-driver-nfs -n kube-system	Asserts the additive WITH_NFS opt-in landed. nfs-health skips when the server is absent, so without this step dropping WITH_NFS: "true" from step 8 would leave the job green with the NFS stack untested
10	chainsaw test --report-name chainsaw-report-additive	Scoped run over infra-stack-health, garage-health, flux-web-health, no-prometheus-when-disabled, openbao-instance, and nfs-health; the metrics-server and NFS absence suites are deliberately excluded
11	hack/ci-dump-diagnostics.sh (always)	Dumps HelmReleases, pods, node pressure (capacity and allocated requests, containers with restarts and their last termination reason, per-pod memory working set, kernel OOM lines from the kind node), events, Flux logs
12	Upload JUnit report	Uploads test results as artifact (14-day retention)
13	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Timeout: 50 minutes.

The two re-run legs (steps 6–10) lock the make deploy-infra idempotency contract: the unchanged-parameter re-run must converge against the provisioned cluster, and the additive re-run must install only the newly enabled components while leaving the base stack untouched. Both opt-ins ride that one additive run: make deploy-infra is convergent, so a separate WITH_NFS leg would repeat the whole base install for one kubectl apply -k deploy/kind/nfs and one extra name on the HelmRelease wait list.

build-e2e-images

Centralised image build for E2E test jobs. Builds the images whose sources the pull request changed, pushes them to GHCR under run-scoped tags (e2e-${run_id}-<orig_tag>), and resolves the rest to the digests main published. The tempest job and every e2e-* job except e2e-infra docker pull from GHCR via the load-e2e-images composite action instead of rebuilding.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, verify-invalid-cr-fixtures, chainsaw-lint]

Condition: Runs only when build-e2e-images == 'true' and no gate job failed. The resolver sets that flag when any job downstream of this one is scheduled, so the four-way OR the job used to carry lives in one place. Uses always() so the job runs when upstream Go jobs are skipped (e.g. pure E2E test-definition PRs where go=false). Skipped on PRs from forks (the workflow's GITHUB_TOKEN is read-only on packages: for forked pull_request events, so GHCR push would fail) — see github.event.pull_request.head.repo.fork guard.

Permissions: contents: read, packages: write (required for GHCR push).

Outputs: image-map, the JSON object every consumer resolves its images through.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned) with full history, which the resolver's revision check reads, and with persist-credentials: false, so no token lies in /github/runner_temp for the build users of step 5 to read
2	docker/setup-buildx-action@v4	Provides docker buildx imagetools inspect, which the resolver of step 4 reads digests and revision annotations with; the job runs no BuildKit build
3	registry-login composite action	Authenticates to GHCR with GITHUB_TOKEN
4	Resolve images	Runs hack/ci-resolve-e2e-images.sh with changed-operators, changed-services, changed-tempest, changed-proxy, changed-ovn and changed-libvirt; writes the BUILD_* variables, OVN_VERSION and the image-map output
5	Build Nix images	Only when BUILD_NIX_IMAGES is true. Runs hack/ci-nix-build-e2e-images.sh in the nixos/nix container (docker://nixos/nix, pinned by tag and digest): builds the flake attribute <svc>-<rel_>-image for each <service> <release> line of BUILD_SERVICE_IMAGES (<image>-<rel_>-image for a line with a third field, such as nova 2026.1 nova-compute), tempest-<rel_>-image for each release of BUILD_TEMPEST_RELEASES, ovn-image when BUILD_OVN is true, libvirt-image when BUILD_LIBVIRT is true, keystone-federation-proxy-image when BUILD_PROXY is true and <op>-operator-image for each name in BUILD_OPERATORS in one nix build, as build users of its own, and writes each image stream to .nix-e2e/<name>-<release>.tar (ovn-<OVN_VERSION>.tar for ovn, libvirt-latest.tar for libvirt, keystone-federation-proxy-latest.tar for the federation proxy, <op>-operator-latest.tar for an operator)
6	Load Nix images	Only when BUILD_NIX_IMAGES is true. docker load of each tar listed in .nix-e2e/images.txt, tagged <IMAGE_PREFIX>/<name>:<release> (<IMAGE_PREFIX>/<op>-operator:latest for an operator), then the tar is removed
7	Tag federation proxy image	Only when BUILD_PROXY is true. Tags <IMAGE_PREFIX>/keystone-federation-proxy:latest, which step 6 loaded, as <IMAGE_PREFIX>/keystone-federation-proxy:dev, the tag the keystone federation suites and step 10 address
8	Tag operator images	Tags <IMAGE_PREFIX>/<op>-operator:latest, which step 6 loaded, as <IMAGE_PREFIX>/<op>-operator:dev for each name in BUILD_OPERATORS, the tag the suites and step 10 address
9	Verify operator images	Runs tests/container-images/verify_operator_image.sh on <IMAGE_PREFIX>/<op>-operator:dev for each name in BUILD_OPERATORS: the layout of nix/lib/static-image.nix, /usr/bin/manager as 65532:65532, and the four static labels alone
10	Push E2E images to GHCR	For each image built above, docker tag to <repo>:e2e-${run_id}-<orig_tag> and docker push: the operator images under <op>-operator:dev, <IMAGE_PREFIX>/ovn:<OVN_VERSION> only when BUILD_OVN is true, <IMAGE_PREFIX>/libvirt:latest only when BUILD_LIBVIRT is true

hack/ci-resolve-e2e-images.sh derives the image set from the tree: an operator image per operators/<op>/ with a go.mod, a service image per (operator, release) pair in releases/*/source-refs.yaml, a derived image per release of the service it is built from (nova-compute from nova, built whenever that nova image is), a Tempest image per releases/<release>/, the federation proxy, the OVN daemon image under the version hack/ci-resolve-ovn-version.sh reads from images/ovn/source.json, and the libvirt image as libvirt:latest. An image whose sources this pull request changed is built here and mapped to its run-scoped tag. Every other image is mapped to the index digest behind its published tag (<op>-operator:latest, <svc>:<release>, tempest:<release>, keystone-federation-proxy:latest, ovn:<version>, libvirt:latest), which the run pulls instead of rebuilding. A source that has never been published, the state of a new operator before its first merge, is built instead of failing the run.

An operator image is reused only when every commit in the checkout that touches the operator's sources is in the history of the commit its index annotation org.opencontainers.image.revision, written by merge-operator-images, names. Those sources are the paths of the operator's own change filter plus go_common, the two filters hack/ci-resolve-changes.sh marks an operator changed by. When one build-and-push leg fails on main, merge-operator-images publishes nothing, and every <op>-operator:latest keeps its previous revision. The resolver builds such an image instead. It does the same for an image without a revision annotation and for a revision the checkout does not have, and a notice in the job log names the reason.

The OVN daemon image carries no OpenStack release and no go.mod; its key comes from the pin. The resolver builds it when changed-ovn is set or main has not published the pinned version yet: it writes BUILD_OVN=true, and step 5 builds ovn-image beside the other Nix images. A pin the resolver cannot read fails the step before the map is written. The libvirt image follows the same rule through changed-libvirt and BUILD_LIBVIRT: built when its inputs changed or latest is not published yet, otherwise pulled by the digest latest names.

The service, nova-compute, Tempest, ovn and libvirt images are flake attributes that share their Nix store layers. The nixos/nix container runs as root in the runner's workspace and builds without the Nix sandbox, which an unprivileged container cannot create, so its builders run as build users of their own that can neither open the runner's Docker socket nor write the workspace, and the job's checkout persists no token for them to read. Its script gives .nix-e2e/ back to the owner of the workspace on exit, so the runner can delete it (e2e builds). A reused image is never tagged in the registry: the consumers pull it by digest, which is what keeps the run-scoped tag off the published version that cleanup-e2e-tags would then refuse to prune. tests/unit/hack/ci_resolve_e2e_images_test.sh pins the decisions against a stubbed registry.

GH-310 replaced the previous docker save | zstd | upload-artifact transport with GHCR push/pull because the 355 MB single-blob artifact intermittently timed out at the 5-minute actions/download-artifact window. Layer-level pull retries plus the GHCR CDN dramatically reduce the failure rate.

Timeout: 120 minutes. The job needs 45 without the ovn and libvirt images. A run that builds the ovn image without a cached build of the pin compiles Open vSwitch and OVN from source, and one that builds the libvirt image compiles libvirt and QEMU, which is what the build-ovn and build-libvirt jobs of build-images.yaml budget 60 minutes for; one nix build builds both side by side. Every E2E job gates on build-e2e-images succeeding, so a job killed at its timeout costs the pull request its whole E2E coverage.

e2e-operator

End-to-end operator test using kind cluster and Chainsaw. Pulls pre-built images from GHCR via the load-e2e-images composite action, deploys the infrastructure stack and operator via Helm, and runs Chainsaw E2E test suites.

Dependencies: needs: [changes, build-e2e-images]Condition: Runs only when has-e2e-operators == 'true' and build-e2e-images succeeded. Permissions: contents: read, packages: read (required for GHCR pull).

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
4	load-e2e-images composite action	Pulls run-scoped GHCR tags and re-tags to canonical local refs
5	kind load docker-image	Loads the operator image, one service image per release under releases/ (2026.1 and 2026.2 today) and the 2026.1-upgraded image into kind, plus ovn:<pin> on the ovn and neutron legs; the nova leg also loads the five sibling operator images, the sibling service images for every release (keystone, placement, glance and neutron at 2026.1 and 2026.2) and ovn:<pin>
6	setup-e2e-infra composite action	Installs Flux CLI, test deps, and deploys infra stack; the ovn, neutron and nova legs pass WITH_OVN_KERNEL_MODULES: true, the cinder and nova legs pass WITH_MESSAGING: true, and the cinder leg alone passes WITH_NFS: true
7	hack/ci-deploy-operator.sh (sibling operators)	nova leg: keystone-, placement- and glance-operator; neutron and nova legs: ovn-operator; nova leg: neutron-operator. Each goes into its <op>-system Namespace, ahead of the matrix operator
8	hack/ci-deploy-operator.sh	Installs CRDs and deploys operator via Helm
9	chainsaw test	Runs E2E tests from tests/e2e/<operator>/ and tests/e2e/<operator>-operator/; a nova shard runs its half of the suite directories
10	hack/ci-dump-diagnostics.sh (always)	Dumps operator pods, all pods, node pressure (capacity and allocated requests, containers with restarts and their last termination reason, per-pod memory working set, kernel OOM lines from the kind node), events, operator logs
11	hack/ci-dump-diagnostics.sh (always, sibling operators)	The neutron leg dumps ovn-system; the nova leg dumps keystone-system, placement-system, glance-system, ovn-system and neutron-system, one call per operator under OPERATOR_ONLY=1 so only the three sections that differ per operator are emitted again
12	Upload JUnit report	Uploads test results as the e2e-<operator>-junit-report artifact, e2e-nova-<shard>-junit-report on a nova shard (14-day retention)
13	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Matrix strategy:

strategy:
  fail-fast: false
  matrix: ${{ fromJson(needs.changes.outputs.e2e-operator-legs) }}

The operator matrix is dynamically constructed by the changes job, including only operators whose code (or shared code) changed. Its Shard the e2e-operator matrix step turns the e2e-operators output into e2e-operator-legs: one leg per operator, except nova, which becomes two legs with shard: "1" and shard: "2". The publish jobs keep reading e2e-operators, so they still see nova once. The imagePullPolicy: Never Helm value ensures the kind-loaded image is used instead of attempting a registry pull. The service images the operator renders are loaded into kind under their published tag, so hack/ci-deploy-operator.sh also sets the operator's default image pull policy to IfNotPresent (DEFAULT_IMAGE_PULL_POLICY); without it a tag would resolve to Always and kubelet would pull the published image over the pull request's build. Timeout: 68 minutes, 150 for each nova shard.

The two OVN legs. ovn ships no per-release service image. Its Pods all run ghcr.io/c5c3/ovn:<pin>, where <pin> is what hack/ci-resolve-ovn-version.sh reads out of images/ovn/source.json. The Resolve E2E images step appends that ref and Load images into kind loads it.

The neutron leg loads the same image plus ovn-operator:dev, and deploys the ovn-operator into ovn-system ahead of the neutron-operator. A Neutron stays out of Ready until an OVNCentral publishes its Northbound and Southbound addresses, and a NeutronMetadataAgent resolves an OVNChassis. That second operator gets a diagnostics dump of its own with OPERATOR: ovn, since the first dump derives its Namespace from the matrix operator and never looks at ovn-system.

Both legs set WITH_OVN_KERNEL_MODULES: true on the setup-e2e-infra step, which loads openvswitch and geneve on the kind node for the chassis DaemonSets. See Kernel-module-dependent suites.

The cinder leg. Its setup-e2e-infra step carries WITH_NFS: true and WITH_MESSAGING: true, neither of which hack/deploy-infra.sh installs by default: the suites mount their volumes as inline CSI volumes from the kind NFS export, and each one takes its own vhost on the shared-rabbitmq broker. The chainsaw step runs with --parallel 2, as the neutron and nova legs do. A cinder suite brings up three or four Deployments, a db-sync Job and a probe pod, and four of those at once do not fit on the kind node of the self-hosted runner; the other legs keep the shared config's parallel: 4. Chart-level suites live in tests/e2e/cinder-operator/, which the -operator directory probe appends to tests/e2e/cinder/. The leg runs on a self-hosted runner, like every leg, and one diagnostics dump covers it, because it deploys a single operator. See Cinder E2E Test Suites.

The nova leg. It deploys five sibling operators through hack/ci-deploy-operator.sh before the nova-operator: keystone, placement, glance, ovn and neutron, each into its own <op>-system Namespace. spec.keystoneEndpoint and spec.serviceUser are required on the Nova CRD, and a booted server needs a Placement allocation, a Glance image and a Neutron port. The Neutron waits on a live OVNCentral, so the ovn-operator comes with it, and the compute-node-pool suite runs a single-node OVNChassis beside it for its NovaCompute. A sibling deploy that fails stops the job at helm install --wait --timeout 120s, the chainsaw step is skipped, and the dumps still run. The setup-e2e-infra step passes WITH_MESSAGING: true: every Nova process dials the bus, and the scheduler and the conductor report ready off their broker connection. It passes WITH_OVN_KERNEL_MODULES: true for that chassis. WITH_NFS stays off, since spec.endpoints.cinder is opt-in on the Nova CRD.

On top of nova-operator:dev, nova:2026.1 and nova:2026.2, the leg also resolves the five sibling operator images; keystone, placement, glance and neutron at every release each one ships (2026.1 and 2026.2 today, per hack/ci-service-image-releases.sh); nova-compute at every nova release, which the NovaCompute pool runs; ovn:<pin>; and tempest:2026.1, whose openstack client is what the catalog, seed and verify Jobs of the functional suites run, on every per-release suite, so it is loaded once. It loads the whole list in one kind load docker-image call, so the base layers the service images share go onto the node once. Every release is there so a 2026.1 or 2026.2 Nova suite can pair with siblings of its own release. The suites the leg carries are the eighteen under tests/e2e/nova/ and the chart-level metrics suite under tests/e2e/nova-operator/, described in Nova E2E Test Suites. Chainsaw runs with --parallel 2, because a full-stack Nova suite carries a Keystone, an OVNCentral, a Neutron, a Placement, a Glance and the five Nova workloads. The wall is timeout-minutes: ${{ matrix.operator == 'nova' && 150 || 68 }}, so only this leg pays for its image loads, its sibling deploys and the suites stacked on it.

The leg runs as two shards, e2e-operator (nova, 1) and e2e-operator (nova, 2), each on its own runner and kind cluster with the full bring-up above. As a single leg it took 125 to 137 minutes, until compute-node-pool, 27 minutes that no other suite runs beside, pushed it past its 150-minute wall on 2026-09-24. The Run E2E tests step names the suites of shard 2 (compute-node-pool, invalid-novacompute-cr, basic-deployment-2026-2, release-upgrade, healthcheck, deletion-cleanup, pod-security-restricted and namespace-scoped-rbac) and gives shard 1 every other suite directory of tests/e2e/nova/ and tests/e2e/nova-operator/, so a new suite runs in shard 1 until it is moved. Each shard takes three of the six suites that run 24 to 30 minutes, which puts both at about 80 to 95 minutes. tests/unit/ci/nova_e2e_matrix_test.sh fails when a suite runs in neither shard or in both.

A third step, Dump diagnostic info (nova siblings), calls hack/ci-dump-diagnostics.sh once per sibling under always(), each with OPERATOR_ONLY=1: the dump above it already emitted the infrastructure block and the openstack Namespace's Job and pod logs, which do not change with OPERATOR.

Nova stays out of the two-cluster placed-services suite (tests/e2e-multicluster/placed-services/), decided on 2026-09-17. That suite proves where the children land. It also runs an OVNChassis and a NeutronMetadataAgent on the management cluster against the node ports of the OVNCentral placed on the target, which is why the e2e-multicluster job sets WITH_OVN_KERNEL_MODULES and loads the OVN and Neutron images into the management cluster. Every target-cluster-access grant a Nova would touch (Deployments, Jobs, CronJobs, HTTPRoutes, NetworkPolicies, HorizontalPodAutoscalers) the placed Keystone, Barbican, OVNCentral and Neutron cover already. The placed Neutron runs against a broker address that resolves nowhere (rabbitmq.openstack.invalid), which a Nova cannot do: membership would mean a real broker on the target cluster, a seventh operator on the management cluster and an eighth image inside a 90-minute job. The operator's envtest keeps Nova's remote teardown covered, and #1013 covers it against a real second cluster when compute clusters arrive. #1019 does not reopen it.

e2e-operator-upgrade

Operator helm-upgrade-in-place E2E. Installs the last released keystone-operator chart+image from GHCR as the baseline, brings a Keystone CR to Ready, then helm upgrades the release to the locally built chart+CRDs and asserts the deployed Keystone survives the operator upgrade (Ready persists, status.installedRelease unchanged, no re-bootstrap). See Operator Upgrade E2E Tests for suite details.

Dependencies: needs: [changes, build-e2e-images]Condition: Runs on pull_request when e2e-operator-upgrade == 'true', build-e2e-images succeeded, and no dependency failed or was cancelled. The resolver sets that flag from the suite's own tree or a keystone code change. Permissions: contents: read, packages: read (required for GHCR pull).

Unlike the per-CR e2e-operator matrix, this suite manages the operator Helm release itself, so it runs in its own single job. The job pulls the run-scoped :dev operator and 2026.1 service images, helm registry logins GHCR, fetches the released baseline via hack/ci-fetch-released-operator.sh, deploys the infra stack, installs the baseline via hack/ci-deploy-operator.sh (with CHART_DIR pointing at the pulled chart and IMAGE_TAG=latest), and runs the suite from tests/e2e-operator-upgrade/. Blocking (no continue-on-error). Timeout: 68 minutes.

e2e-chaos

End-to-end chaos tests using kind cluster, Chaos Mesh, and Chainsaw. Pulls the operator and service images its leg needs from GHCR via the load-e2e-images composite action, deploys them alongside Chaos Mesh infrastructure, and runs the chaos test suites of its matrix entry. See Chaos E2E Test Suites for test suite details.

The figure shows the test bed and the six steps a suite takes, and why a partition rule sits on the server pods. Overview describes every step under its number.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images, e2e-operator]Condition: Runs only when e2e-chaos == 'true', build-e2e-images succeeded, and no dependency failed or was cancelled. The resolver sets that flag from the tests_chaos class or the ci:chaos label, so the job's own condition no longer reads the label set. Permissions: contents: read, packages: read (required for GHCR pull).

The e2e-chaos job depends on the standard gate jobs plus e2e-operator, so chaos tests run after the happy-path operator E2E suite has passed. Gating is set per matrix leg via continue-on-error: ${{ matrix.suite == 'network' || matrix.suite == 'ovn' || matrix.suite == 'nova' || matrix.suite == 'controlplane' }}: the pod leg is blocking, so an operator-restart, PDB, or rotation regression fails the build. The network, ovn, nova and controlplane legs stay non-blocking. The expression names those four rather than negating pod, so a leg added to the matrix later gates merges until it is argued out of that. The network leg's ip_set/sch_netem kernel-module dependency keeps it prone to environment flakiness, the ovn leg builds an OVN datapath out of the host's openvswitch and geneve modules, and three of the nova leg's six suites are NetworkChaos partitions on the same sch_netem and ip_set modules the network leg's suites use, while its neutron-metadata-agent-pod-kill needs the openvswitch and geneve modules for an OVNChassis. The controlplane entry's korc-network-partition is a NetworkChaos partition on the sch_netem and ip_set modules as well, which places all four under Kernel-module-dependent suites. On-demand pre-validation of any leg is available via the ci:chaos PR label, with run-chaos kept as an alias.

The job runs as a five-entry matrix, split by chaos type and by the stack each leg needs:

Leg	Operators deployed	Suites
pod	keystone, horizon, glance, placement, barbican	the PodChaos suites of the keystone stack, the Keystone outage suites of Glance, Placement and Barbican, horizon-dashboard-pod-kill, and the eight platform suites that need only the standalone stack (cert-manager-outage, eso-down, eso-pod-kill, openbao-sealed, openbao-tenant-policy-narrowed, envoy-gateway-outage, envoy-proxy-pod-kill, mariadb-operator-cascading-failure)
network	keystone, horizon, glance, placement, barbican, ovn, neutron, cinder	the NetworkChaos suites and every suite that needs the neutron, cinder or ovn operators: the four Neutron suites (neutron-mariadb-outage, neutron-broker-outage, neutron-keystone-outage, neutron-operator-pod-kill), ovn-operator-pod-kill, the seven Cinder suites (cinder-operator-pod-kill, cinder-broker-outage, cinder-nfs-outage, cinder-keystone-outage, cinder-mariadb-outage, cinder-volume-pod-kill, cinder-broker-pod-kill), the database outages of Glance, Placement and Barbican, and memcached-network-partition
ovn	ovn	ovn-southbound-outage
nova	keystone, horizon, glance, barbican, placement, ovn, neutron, nova	nova-broker-outage, nova-mariadb-outage, nova-placement-outage, nova-keystone-outage, nova-operator-pod-kill, neutron-metadata-agent-pod-kill
controlplane	keystone, horizon, glance, barbican, ovn, K-ORC, c5c3	the five suites that bring up a ControlPlane of their own: korc-provisioning-outage, korc-network-partition, controlplane-keystone-degraded, c5c3-operator-pod-kill, rabbitmq-operator-cascading-failure

All five legs run on the self-hosted runners. The split keeps the legs independently gated and lets them run in parallel. Each matrix entry lists its per-suite test directories explicitly.

A Resolve OVN version step reads the pin from images/ovn/source.json through hack/ci-resolve-ovn-version.sh and writes OVN_VERSION into $GITHUB_ENV, so the tag itself never appears in the workflow. The network, ovn and nova legs load ovn-operator:dev and ovn:$OVN_VERSION into kind, the network leg adds neutron-operator:dev and neutron:2026.1, the pod and network legs load placement-operator:dev and placement:2026.1 for their Placement suites, and the ovn leg loads none of the keystone stack. The keystone stack carries tempest:2026.1, which the workload pod of tests/e2e-chaos/lib/workload.sh runs on and whose openstack client the nova leg's catalog, seed and verify Jobs run. The nova leg names six images of its own: placement-operator:dev, placement:2026.1, neutron-operator:dev, neutron:2026.1, nova-operator:dev and nova:2026.1. WITH_MESSAGING is set on the network and nova legs, the first for the vhosts five Cinder suites take, the second because every Nova process dials the bus. Setup E2E infrastructure passes WITH_OVN_KERNEL_MODULES: ${{ (matrix.suite == 'ovn' || matrix.suite == 'nova') && 'true' || '' }}, which is what makes hack/deploy-infra.sh load openvswitch and geneve for the chassis DaemonSets of ovn-southbound-outage and neutron-metadata-agent-pod-kill. The diagnostics dump follows the leg through OPERATOR: ${{ matrix.suite == 'ovn' && 'ovn' || 'keystone' }}, because the ovn leg has no keystone-system namespace to dump.

The controlplane entry sets WITH_CONTROLPLANE: ${{ matrix.suite == 'controlplane' && 'true' || '' }} and CONTROLPLANE_OPERATORS: ${{ matrix.suite == 'controlplane' && 'external' || '' }} on Setup E2E infrastructure, the environment of e2e-controlplane: the infrastructure overlay comes without the standalone MariaDB, Memcached and shim ExternalSecrets, and the Flux ControlPlane stack is suspended. It loads c5c3-operator:dev beside the keystone stack (Load controlplane images into kind). Deploy K-ORC (controlplane leg) applies the Placement CRD the c5c3-operator's cache needs and runs hack/ci-deploy-korc.sh with GITHUB_TOKEN, and Deploy c5c3 operator installs the c5c3-operator into c5c3-system at the chart default of two replicas. Each suite of the entry brings up a Keystone-only ControlPlane in a namespace of its own through tests/e2e-chaos/lib/controlplane.sh. Dump diagnostic info (controlplane leg) adds the c5c3-operator sections and 500 lines of the K-ORC log. The != 'pod' OVN steps and the != 'ovn' keystone stack run on it as well, which costs about two minutes.

The wall is timeout-minutes: ${{ matrix.suite == 'nova' && 150 || matrix.suite == 'network' && 135 || matrix.suite == 'controlplane' && 100 || matrix.suite == 'ovn' && 45 || 120 }}, 550 minutes in all, which the author of #1390 accepted on 2026-10-16. Run 38903206716 took 62 minutes on pod, 125 on network, 11 on ovn and 130 on nova. pod gains the eight platform suites of #1390 at about 5 minutes each. The controlplane entry needs about 15 minutes for its bring-up and 8 to 12 for each of its five suites, from the 7 min 45 s the keystone-service suite took in run 38788542824 plus the fault and a teardown of up to 12 minutes. The nova leg loads eighteen images and runs eight operator deploys before its first suite, and its six suites each stand up a Keystone, an OVNCentral, a Neutron, a Placement, a Glance and a five-workload Nova, one after the other because tests/e2e-chaos/chainsaw-config.yaml sets parallel: 1. A wall that arrives mid-suite kills chainsaw outright: no catch block runs, no step cleanup runs, no JUnit report is written, and a non-blocking leg reports as a cancelled job that names no suite. The per-suite times of the new suites are estimates; confirm each duration against the first green run with them.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
3	hack/ci-resolve-ovn-version.sh	Writes the images/ovn/source.json pin to $GITHUB_ENV as OVN_VERSION
4	load-e2e-images composite action	Pulls the run-scoped GHCR tags this leg needs and re-tags to canonical local refs
5	kind load docker-image	Loads the keystone stack (all but ovn), placement (pod and network), the OVN images (all but pod), the neutron and cinder images (network), the seven nova-leg images (nova), and c5c3-operator:dev (controlplane)
6	setup-e2e-infra composite action	Installs Flux CLI, test deps, and deploys infra stack with WITH_CHAOS_MESH=true, plus WITH_OVN_KERNEL_MODULES=true on the ovn and nova legs and WITH_CONTROLPLANE=true with CONTROLPLANE_OPERATORS=external on controlplane
7	hack/ci-deploy-operator.sh	Installs CRDs and deploys this leg's operators via Helm, the placement-operator on the pod and network legs; the nova leg adds three steps of its own, Deploy placement operator (nova leg), Deploy neutron operator (nova leg) and Deploy nova operator, the first two suffixed so tests/lib/ci_yaml.sh still resolves the pinned steps above them; controlplane adds Deploy K-ORC (controlplane leg) (the Placement CRD and hack/ci-deploy-korc.sh) and Deploy c5c3 operator
8	chainsaw test	Runs chaos E2E tests from tests/e2e-chaos/ with tests/e2e-chaos/chainsaw-config.yaml
9	hack/ci-dump-diagnostics.sh (always)	Dumps operator pods, all pods, events, operator logs with OPERATOR=keystone, or OPERATOR=ovn on the ovn leg; a second step on the nova leg repeats the call for nova, placement, glance, ovn and neutron, and one on the network leg for neutron, ovn, cinder, placement, glance, barbican and horizon, each under OPERATOR_ONLY=1 so only the three sections that differ per operator are emitted again; one on controlplane dumps the c5c3-operator under OPERATOR_ONLY=1 and 500 lines of the K-ORC log
10	Upload JUnit report	Uploads _output/reports/ as e2e-chaos-junit-report-<suite> artifact (14-day retention)
11	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Key differences from e2e-operator:

Aspect	e2e-operator	e2e-chaos
Matrix	Dynamic per-operator	Five suites (pod / network / ovn / nova / controlplane) on different runners
Test config	tests/e2e/chainsaw-config.yaml	tests/e2e-chaos/chainsaw-config.yaml
Test directory	tests/e2e/<operator>/	per-suite test_dirs under tests/e2e-chaos/
Timeout	68 minutes, 150 for the nova leg	120 minutes for pod, 135 for network, 45 for ovn, 150 for nova, 100 for controlplane
Blocking	Yes	pod leg blocking; network, ovn, nova and controlplane legs non-blocking (continue-on-error: ${{ matrix.suite == 'network' || matrix.suite == 'ovn' || matrix.suite == 'nova' || matrix.suite == 'controlplane' }})
Dependencies	Gate jobs	Gate jobs + e2e-operator
Service images	2026.1 + 2026.1-upgraded + 2026.2	2026.1 only, plus the pinned OVN daemon image

The chaos test Chainsaw config uses parallel: 1 (serial execution) because chaos tests mutate shared infrastructure pod availability. The assert timeout is 300s (vs 120s for happy-path tests) to allow multiple reconciliation cycles and pod restart time during fault recovery.

Path filter: the tests_chaos class is tests/e2e-chaos/** alone. hack/ci-resolve-changes.sh switches the job on from that class, from the ci:chaos label or from its run-chaos alias, and a tag push or ci:full forces it on with every other flag. Nothing else reaches it: a Go code change is exercised by that operator's own e2e leg, and deploy/** is out of the class because every OpenBao or Flux bump would otherwise pay for the chaos legs.

Kernel-module-dependent suites

A suite whose workload needs host kernel modules enters CI non-blocking: it runs on the self-hosted runners with continue-on-error: true. A separate, later change flips it to blocking, once its pass history on main justifies the flip. The network leg of e2e-chaos is the precedent, and it is still on the non-blocking side of that path.

Blocking baseline. The suite that earns the flip first is the single-node one on hack/kind-config.yaml: the DaemonSet renders, its pods start, and the chassis registers itself in the OVN southbound database. Neither assertion needs a second schedulable node.

Multi-node legs. A suite that does need one uses hack/kind-config-multinode.yaml (one control-plane node plus two workers) in a job of its own, never on a runner that also holds a hack/kind-config.yaml cluster. Both configs bind the same host ports, so the second cluster fails to create. The e2e-multicluster job sidesteps the same collision by creating its management cluster with no kind config.

The path the file takes into the job is the config: input of the create-kind-cluster step, not the KIND_CONFIG variable: setup-e2e-infra runs with SKIP_KIND_CREATE: "true", so the cluster already exists by the time hack/deploy-infra.sh sees the variable, and it warns that the value is being ignored rather than recreating anything.

The modules come from WITH_OVN_KERNEL_MODULES=true in the setup-e2e-infra step's env, which threads through to hack/deploy-infra.sh and runs modprobe for openvswitch and geneve before the cluster is created. The load is best-effort, through the helper that also loads the chaos suite's ip_set and sch_netem modules. On a host without root or passwordless sudo it logs a warning and continues, and the suite then fails on its own assertions.

Where each OVN suite sits (author's decision, 2026-08-27). The single-node chassis baseline (tests/e2e/ovn/chassis-single-node) and the Neutron metadata-agent suite (tests/e2e/neutron/metadata-agent) run blocking, inside the e2e-operator legs for ovn and neutron. Both assert what one node can answer. The nova leg loads the modules too: its compute-node-pool suite runs a single-node chassis for the gate its NovaCompute pod waits on. The e2e-ovn-overlay job takes the non-blocking entry described above, and so do the ovn leg of e2e-chaos and, for neutron-metadata-agent-pod-kill, its nova leg.

e2e-ovn-overlay

Runs the tests/e2e-ovn-overlay/geneve-datapath/ Chainsaw suite on a two-worker kind cluster: an OVNCentral, an OVNChassis selecting both workers, one logical switch port per worker, and a ping between two network namespaces that can only reach each other across the Geneve tunnel. The suite asserts numberReady: 2 on the chassis, two status.nodes entries carrying two different system-ids, a Port_Binding for lsp-a bound to a chassis, and two Geneve Encap rows on two addresses. A single node answers none of those questions, so the suite lives outside tests/e2e/ and gets a job of its own.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-ovn-overlay == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

The job runs on self-hosted with continue-on-error: true under the kernel-module rule above, and creates its cluster from config: hack/kind-config-multinode.yaml. setup-e2e-infra receives WITH_OVN_KERNEL_MODULES: "true", hack/ci-deploy-operator.sh installs the ovn-operator into ovn-system, and the suite runs through make e2e-ovn-overlay, the same target a developer calls locally. The OVN daemon image is loaded at the tag hack/ci-resolve-ovn-version.sh reads from images/ovn/source.json, so the workflow spells no version of its own. Diagnostics run with OPERATOR: ovn, and _output/reports/ is uploaded as the e2e-ovn-overlay-junit-report artifact (14-day retention). cleanup-e2e-tags lists the job in its needs, so the run-scoped image tags survive until it finishes.

Path filter: tests/e2e-ovn-overlay/** (the tests_ovn_overlay class), operators/ovn/** (ovn), and images/ovn/** and images/backup-shifter/** (image_ovn). A shared Go change does not schedule the job, and ci:full forces it on.

e2e-nova-libvirt

Runs the tests/e2e-nova-libvirt/server-boot/ Chainsaw suite, which boots a server through Nova's libvirt driver on the one node of a hack/kind-config.yaml cluster. The suite brings up the 2026.1 stack of tests/e2e/nova/compute-node-pool/ from that suite's own fixtures, starts libvirtd in the DaemonSet libvirt-e2e and applies a NovaCompute pool with virtType: qemu. The server boots from the stack's 1 MiB zero image under QEMU's TCG emulation, which needs no /dev/kvm on the runner. The suite asserts that the pool is Ready with its node Active and the compute service up, that the rendered compute-pool.conf names libvirt.LibvirtDriver and virt_type = qemu, and that the server goes ACTIVE with exactly one running domain of type qemu. The domain's disk is a qcow2 overlay on a base file under /var/lib/nova/instances/_base/. Deleting the server removes the domain and the instance directory. The guest has no operating system and no NIC: the guest boot, KVM, networking and live migration are proven by the lab run of Quick Start (metal-stack).

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-nova-libvirt == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

The job runs on self-hosted with continue-on-error: true under the kernel-module rule above: the pool's chassis gate needs an OVNChassis on the node, and that needs openvswitch on the runner host. It runs alone on its runner, since hack/kind-config.yaml binds host ports 443 and 8428, and its wall is 90 minutes. The cluster comes from config: hack/kind-config.yaml, and setup-e2e-infra receives WITH_OVN_KERNEL_MODULES: "true" and WITH_MESSAGING: "true". hack/ci-deploy-operator.sh then installs the keystone, placement, glance, ovn, neutron and nova operators in that order, each into <op>-system, and the suite runs through make e2e-nova-libvirt, the target a developer calls locally.

The job loads 15 images: the six operators at :dev, keystone, placement, glance, neutron, nova and nova-compute at 2026.1, the OVN daemon image at the tag hack/ci-resolve-ovn-version.sh reads from images/ovn/source.json, tempest:2026.1, whose openstack client the suite's Jobs run, and libvirt:latest, which the suite's DaemonSet runs with IfNotPresent. The image map resolves libvirt:latest to this run's build when the pull request changed an input of the libvirt image, and otherwise to the digest latest names on GHCR, so a pull request that changes the image runs the suite against its own libvirt. Diagnostics run with OPERATOR: nova and then once per sibling with OPERATOR_ONLY=1, and _output/reports/ is uploaded as the e2e-nova-libvirt-junit-report artifact (14-day retention). cleanup-e2e-tags lists the job in its needs.

Path filter: tests/e2e-nova-libvirt/**, tests/e2e/nova/compute-node-pool/** and tests/e2e/cinder/broker-vhost.sh (the tests_nova_libvirt class), operators/nova/** (nova), images/nova/**, images/nova-compute/** and patches/nova/** (image_nova), and the libvirt image's inputs (image_libvirt). A shared Go change does not schedule the job, and ci:full forces it on.

e2e-prometheus

End-to-end kube-prometheus-stack tests using kind cluster, Flux-managed kube-prometheus-stack HelmRelease, and Chainsaw. Pulls the keystone operator image, deploys it alongside the monitoring stack, and runs the prometheus suite under tests/e2e/keystone/prometheus-stack/ to verify HelmRelease readiness, ServiceMonitor presence, and live Prometheus scraping of the operator metrics endpoint.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-prometheus == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

The setup-e2e-infra composite action is invoked with WITH_PROMETHEUS: "true" in its step env, which threads through to hack/deploy-infra.sh and gates the kube-prometheus-stack overlay (deploy/kind/prometheus/) plus the post-deploy enable_operator_servicemonitor patch (applied to the keystone-operator, horizon-operator, glance-operator, placement-operator, barbican-operator, ovn-operator, neutron-operator, cinder-operator and nova-operator HelmReleases). The Deploy operator step runs hack/ci-deploy-operator.sh with WITH_PROMETHEUS: "true" in its step env, which adds --set monitoring.serviceMonitor.enabled=true to the Helm install command — without this flag the chart's gated ServiceMonitor template renders nothing and the chainsaw step servicemonitor-exists (and the dependent prometheus-target-up) cannot pass. The kind base kustomization keeps the keystone-operator HelmRelease suspended, so the runtime kubectl patch cannot reactively enable the ServiceMonitor — the install-time flag is the single source of truth.

Unlike e2e-chaos, e2e-prometheus runs with continue-on-error: false: the kube-prometheus stack is deterministic on kind, so any failure is a genuine regression of the kind-only Quick Start observability story.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
3	load-e2e-images composite	Pulls keystone-operator:dev and keystone:2026.1 from GHCR and re-tags them to their canonical local references
4	kind load docker-image	Loads operator and service images into kind
5	setup-e2e-infra composite action	Installs Flux CLI, test deps, and deploys infra stack with WITH_PROMETHEUS: "true"
6	hack/ci-deploy-operator.sh	Installs CRDs and deploys keystone operator via Helm with WITH_PROMETHEUS: "true" (gates --set monitoring.serviceMonitor.enabled=true)
7	chainsaw test	Runs the prometheus E2E suite from tests/e2e/keystone/prometheus-stack/
8	hack/ci-dump-diagnostics.sh (always)	Dumps operator pods, all pods, events, operator logs with OPERATOR=keystone
9	Upload JUnit report	Uploads _output/reports/ as e2e-prometheus-junit-report artifact (14-day retention)
10	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Path filter: tests/e2e/keystone/prometheus-stack/**, deploy/kind/prometheus/**, operators/keystone/dashboards/** and operators/keystone/helm/keystone-operator/templates/servicemonitor.yaml (the tests_prometheus class). No Go code change and no other E2E test change schedules the job, and ci:full forces it on.

e2e-controlplane

Runs the full c5c3 ControlPlane → Keystone chain on kind. It deploys the keystone, horizon, glance, placement, barbican, ovn, neutron, cinder, and nova operators plus K-ORC and c5c3-operator as local dev images (rather than the GHCR-published Flux chart) and runs the tests/e2e/c5c3/full-controlplane-keystone/ Chainsaw suite, which asserts the whole orchestration link by link: managed MariaDB/Memcached provisioning, the projected Keystone CR, the minted restricted K-ORC application credential, the OpenBao → ESO credential round-trip, the identity catalog, and finally a live openstack token issue / catalog list against the Keystone /v3 endpoint. The suite applies a standalone OVNCentral of its own beside the ControlPlane, since the plane only references a central and never projects one, and asserts the network service on top: OVNReady mirroring that central and NeutronReady over the projected Neutron child. The block-storage service closes the chain: cinder-operator:dev and cinder:2026.1 join the images loaded into kind, and the Deploy cinder-operator step runs between Deploy neutron-operator and Deploy c5c3-operator so the three Cinder CRDs exist before c5c3-operator projects the child and its satellites, and something drives that child to Ready once it lands. A second hack/ci-dump-diagnostics.sh invocation runs with OPERATOR: cinder, which points the operator half of the dump at cinder-system while the workload half stays on openstack.

The compute service closes it in turn: nova-operator:dev and nova:2026.1 join both image lists, and the Deploy nova-operator step runs between Deploy cinder-operator and Deploy c5c3-operator, so the Nova CRD exists before c5c3-operator projects the child and something drives that child to Ready once it lands. setup-e2e-infra gains WITH_OVN_KERNEL_MODULES: "true": the suite applies an OVNChassis with a real ovn-controller, which binds the booted server's port and opens a Geneve tunnel from the kind node, and that tunnel needs the openvswitch and geneve modules on the runner host, which deploy-infra.sh modprobes under that switch. A third hack/ci-dump-diagnostics.sh invocation runs with OPERATOR: nova, pointing the operator half at nova-system while the workload half stays on openstack, where the projected Nova child and the suite's fake compute run.

A second chainsaw step on the same cluster runs tests/e2e/c5c3/keystone-service-foreign-namespace/, the cross-namespace KeystoneService suite: it brings up a Keystone-only ControlPlane of its own, seeds that plane's per-CR OpenBao paths itself, and proves an allowlisted namespace can register and authenticate while an unlisted one is refused. The two suites are separate steps because the shared chainsaw config sets failFast, so one invocation over both directories would let either abort the other; the second step writes its JUnit XML under a distinct report name so each survives in the uploaded artifact.

A third chainsaw step runs tests/e2e/c5c3/keystone-user/, the Keystone order suite, in ten steps: against a Keystone-only ControlPlane of its own that assigns one namespace, it proves the Secret delivered beside the user order, its repair and a rotation through spec.passwordGeneration and on the schedule; a project, a role assignment and a catalog entry ordered beside the user, refused until the entry admits the role and the catalog, then scoping a token and listing the entry; an application credential created as the user, its scheduled rotation on a three-minute interval, the superseded credential failing after its grace period, and a rotation by hand; the refusal of an unassigned namespace, the freeze when the assignment is withdrawn, and the teardown, which holds the assignment on the credential order and the user on its assignment, down to the OpenBao paths. It writes its JUnit XML as chainsaw-report-keystone-user.

A fourth chainsaw step runs tests/e2e/c5c3/keystone-service/, the own-namespace KeystoneService suite: it brings up a Keystone-only ControlPlane of its own and seeds that plane's OpenBao paths itself, then proves the round-trip through the materialised clouds.yaml, the rotation driven by a CredentialRotation, a collision held at ServiceCollision / ServiceAccountCollision until catalog.adopt takes the row over, and residue-free deletion. Both are separate steps for the same failFast reason, and this one writes its JUnit XML as chainsaw-report-keystone-service-own-namespace, so the uploaded artifact carries four report files.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-controlplane == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

setup-e2e-infra is invoked with WITH_CONTROLPLANE: "true", CONTROLPLANE_OPERATORS: external, and CONTROLPLANE_NAME: controlplane-keystone. Under CONTROLPLANE_OPERATORS=externalhack/deploy-infra.sh prepares only the shared prerequisites (TLS issuers, OpenBao with per-CR admin-password seeding, the ESO ClusterSecretStore) and suspends the Flux ControlPlane stack, so the dev-image operators deployed by the subsequent steps own the reconcile. K-ORC is applied by hack/ci-deploy-korc.sh at the tag pinned in deploy/flux-system/sources/k-orc.yaml.

setup-e2e-infra also carries WITH_NFS: "true" and WITH_MESSAGING: "true", neither of which deploy-infra.sh installs by default. The seventh service, Cinder, mounts both of its shares from the kind NFS export and takes a vhost of its own on the shared-rabbitmq broker, so the suite gates on both stacks being present the way the cinder e2e-operator leg does.

The suite runs with E2E_REQUIRE_CONTROLPLANE_STACK: "true", which flips its presence guard from a silent SKIP to a hard failure — so a broken operator/CRD deployment fails the build instead of going green. The same step sets E2E_NODE_BUDGET: "true", which turns the suite's Link 6z from a SKIP line into the node budget gate: once the plane is Ready, hack/ci-check-node-budget.sh sums the effective requests of the pods on the kind node, without the compute-node data plane, and fails the step above 4000m CPU or 16Gi memory. The job-level env sets OPERATOR_REPLICAS: "1", which the ten Deploy <op>-operator steps pass to hack/ci-deploy-operator.sh, so every operator runs one replica, as in the kind devstack, and the gate counts that footprint. The other e2e jobs keep the chart default of two. Like e2e-prometheus, the job runs with continue-on-error: false, and it uses a 320-minute timeout on the larger runner because a real MariaDB + Memcached + Keystone + ten operators + OpenBao + ESO + K-ORC on one node is resource-heavy, and its four chainsaw suites run in sequence on that one node, so their budgets add up rather than overlap. A suite's ceiling is not its exec budget alone: chainsaw applies that budget to every script operation, so try, catch and finally each get one, and the cleanup budget runs after all three. Each of the four suites therefore pins its catch and finally timeouts explicitly, which puts the ceilings at 85, 80, 110 and 90 minutes. The job wall has to outlast the bring-up plus the suites that pass plus the full ceiling of the one that stalls, or a stalled suite is killed before its own timeout fires and reports as a cancelled job with no JUnit XML. The 320 minutes cover roughly 40 minutes of bring-up, the 85-minute ceiling of the full-chain suite, roughly 25 minutes for the foreign-namespace pass and 55 for the keystone-user pass, and the 110-minute ceiling of the keystone-user suite, the largest of the four, 315 minutes in all; the bring-up and the two pass terms are estimates, so confirm them against the first green run of the leg and re-derive the wall if one overruns.

Under ci:measure-sizing the job runs the sizing measurement: the watch starts after Setup E2E infrastructure, and the collect and upload steps follow Run own-namespace registration E2E test and publish sizing-e2e-controlplane.

e2e-controlplane-sso

Runs the tests/e2e-controlplane-sso/ Chainsaw suite: the end-user SSO experience — the Horizon websso projection, the login page's SSO choice and domain dropdown, the websso round trip through the gateway, and LDAP-domain login. The suite lives outside tests/e2e/ so the per-CR e2e-operator matrix leg, which runs tests/e2e/<operator>/ wholesale, does not sweep it up.

A sibling job rather than a second suite directory on e2e-controlplane. The ControlPlane webhook permits one ControlPlane per namespace, and openstack-gw sets allowedRoutes.namespaces.from: Same (the operators deliberately do not manage ReferenceGrant), so the two suites can share neither the openstack namespace nor the Gateway. Each therefore needs its own kind cluster.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-controlplane-sso == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

It mirrors e2e-controlplane's setup with CONTROLPLANE_NAME: controlplane-sso (so the OpenBao bootstrap seeds the per-CR admin-password and Horizon SECRET_KEY paths the chain reads) and additionally loads keystone-federation-proxy:dev into kind, because the suite's ControlPlane CR pins services.keystone.federationProxyImage.tag: dev. Without that override the suite would validate the sidecar already published on main rather than the one under review. The job takes the proxy from the image map: built in this run by the Build Nix images step of build-e2e-images and tagged :dev when the image_proxy filter matched (images/keystone-federation-proxy/**, the flake inputs, nix/lib/** or the build script), and otherwise the digest behind ghcr.io/c5c3/keystone-federation-proxy:latest.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
3	load-e2e-images composite	Restores keystone-operator:dev, c5c3-operator:dev, horizon-operator:dev, keystone-federation-proxy:dev, keystone:2026.1, horizon:2026.1 from GHCR
4	kind load docker-image	Loads the six images into kind
5	setup-e2e-infra composite action	Deploys infra with WITH_CONTROLPLANE=true CONTROLPLANE_OPERATORS=external CONTROLPLANE_NAME=controlplane-sso WITH_CONTROLPLANE_CR=false
5a	Start the sizing measurement (ci:measure-sizing only)	Starts the background watch of Sizing measurement; step 5 then also carries WITH_VPA
6	hack/ci-deploy-korc.sh	Applies K-ORC CRDs + controller at the pinned commit; runs with GITHUB_TOKEN so the clone from github.com is authenticated, since GitHub can answer an anonymous clone from the runners' git 2.43 with a 401 challenge (the header comment of hack/ci-deploy-korc.sh has the details)
7	hack/ci-deploy-operator.sh (keystone)	Deploys the keystone-operator dev image into keystone-system
8	hack/ci-deploy-operator.sh (horizon)	Deploys the horizon-operator dev image into horizon-system
9	kubectl apply -f operators/glance/helm/glance-operator/crds/	Installs the Glance CRDs the c5c3-operator watches; the glance-operator itself stays undeployed
10	kubectl apply -f operators/placement/helm/placement-operator/crds/	Installs the Placement CRDs the c5c3-operator watches; the placement-operator itself stays undeployed
11	hack/ci-deploy-operator.sh (c5c3)	Deploys the c5c3-operator dev image into c5c3-system
12	chainsaw test	Runs tests/e2e-controlplane-sso/ with E2E_REQUIRE_CONTROLPLANE_STACK=true
12a	Collect the sizing measurement (always) (ci:measure-sizing only)	Stops the watch and writes the report
12b	Upload the sizing measurement (always) (ci:measure-sizing only)	Uploads _output/sizing/ as sizing-e2e-controlplane-sso (14-day retention)
13	hack/ci-dump-diagnostics.sh (always)	Dumps diagnostics with OPERATOR=c5c3
14	Upload JUnit report	Uploads _output/reports/ as e2e-controlplane-sso-junit-report (14-day retention)
15	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Path filter: operators/c5c3/** (c5c3) and tests/e2e-controlplane-sso/** (the tests_controlplane_sso class). The ci:controlplane and ci:measure-sizing labels also schedule the job, and ci:full forces it on. No other Go code change or E2E test change schedules it. The job pulls c5c3-operator:dev from the image map: built in this run when operators/c5c3/** changed, and otherwise the digest behind ghcr.io/c5c3/c5c3-operator:latest, so both dev images exist even for a full-chain-test-only change.

e2e-autoscaling

Runs the tests/e2e-autoscaling/ Chainsaw suite on a full ControlPlane with metrics-server and the VerticalPodAutoscaler: token load scales Keystone from one pod to its HPA maximum and back, the PodDisruptionBudget admits an eviction at the minimum, every database-backed API fits its SQL connection cap at its HPA maximum, and the operators create the VerticalPodAutoscalers the ControlPlane opts into. See e2e-autoscaling for what each step asserts. The suite lives outside tests/e2e/, so neither the e2e-operator legs nor make e2e sweep it up, and the job brings up a kind cluster of its own: the ControlPlane webhook permits one ControlPlane per namespace, and no other ControlPlane job deploys metrics-server and the VPA unconditionally.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-autoscaling == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled. Forked pull requests skip it, as they do every job that pulls the E2E images.

The setup-e2e-infra composite action threads WITH_METRICS_SERVER from the step env to hack/deploy-infra.sh, which deploys the kind metrics-server overlay (deploy/kind/metrics-server/) the HPAs read their CPU utilization from. It threads WITH_VPA: "true" the same way, which deploys the kind VPA overlay (deploy/kind/vpa/): the operators need its VerticalPodAutoscaler CRD to create the opt-in VPAs, and its recommender answers the Northbound database's VPA, which step 5 of the suite waits for. The job passes CONTROLPLANE_NAME: cp-autoscaling, so the OpenBao bootstrap seeds the admin-password path of the suite's ControlPlane, and opts into NFS and the shared broker as e2e-controlplane does.

timeout-minutes: 205 covers about 40 minutes of bring-up, as e2e-controlplane estimates it, plus the suite's 160-minute ceiling (the sum of its step, catch and cleanup timeouts) and a few minutes of margin. Both terms are estimates to confirm against the first green run.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
3	hack/ci-resolve-ovn-version.sh	Exports OVN_VERSION from the pin in images/ovn/source.json
4	load-e2e-images composite	Restores the ten operator :dev images, the keystone, glance, placement, barbican, neutron, cinder and nova images at 2026.1, ovn:${OVN_VERSION} and tempest:2026.1
5	kind load docker-image	Loads the nineteen images into kind
6	setup-e2e-infra composite action	Deploys infra with WITH_CONTROLPLANE=true CONTROLPLANE_OPERATORS=external CONTROLPLANE_NAME=cp-autoscaling WITH_CONTROLPLANE_CR=false WITH_NFS=true WITH_MESSAGING=true WITH_METRICS_SERVER=true WITH_VPA=true
7	hack/ci-deploy-korc.sh	Applies K-ORC CRDs + controller at the pinned commit, with GITHUB_TOKEN
8	hack/ci-deploy-operator.sh ×10	Deploys the keystone, horizon, glance, placement, barbican, ovn, neutron, cinder, nova and c5c3 operators, in e2e-controlplane's order
9	chainsaw test	Runs tests/e2e-autoscaling/
10	hack/ci-dump-diagnostics.sh (always)	Dumps diagnostics with OPERATOR=c5c3, then with OPERATOR_ONLY=1 for keystone, glance, placement, barbican, neutron, cinder and nova
11	Upload JUnit report	Uploads _output/reports/ as e2e-autoscaling-junit-report (14-day retention)
12	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

Path filter: the shared ControlPlane triggers (FILTER_c5c3 or the ci:controlplane label) plus tests_autoscaling: tests/e2e-autoscaling/**, deploy/kind/metrics-server/**, deploy/kind/vpa/**, internal/common/deployment/builders.go, internal/common/deployment/hpa_flow.go and internal/common/deployment/vpa_flow.go. ci:full and a v* tag force it on, and a labeled no-op resolves it to false.

e2e-external-keystone

Runs the tests/e2e/c5c3/external-keystone/ Chainsaw suite: an External-mode ControlPlane driven against a plain Keystone the operator does not own. The suite brings up its own operator-free, SQLite-backed Keystone fixture in a separate namespace, then drives four External ControlPlanes against it and asserts the whole adoption contract — convergence with zero MariaDB/Memcached/Keystone children, admin and catalog imports, the application credential minted against the external API and round-tripped through OpenBao, no catalog pollution (compared against a pre-recorded inventory), service-account usability and rotation, out-of-band password rotation and drift detection, the endpoint_type failure detection, the wrong-password and ambiguous-catalog negative paths, and zero-blast-radius deletion.

A sibling job rather than a second suite directory on e2e-controlplane. The ControlPlane webhook permits one ControlPlane per namespace, and this suite stands up a wholly different infrastructure shape (a plain, operator-free Keystone fixture plus four External ControlPlanes, none provisioning MariaDB/Memcached), so it needs its own kind cluster.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs only when e2e-external-keystone == 'true', the upstream build-e2e-images job succeeded, and no dependency failed or was cancelled.

It mirrors e2e-controlplane's setup with WITH_CONTROLPLANE: "true", CONTROLPLANE_OPERATORS: external, and WITH_CONTROLPLANE_CR: "false", but leaves CONTROLPLANE_NAME at its default: the OpenBao bootstrap only seeds the managed-mode admin-password path, which the External ControlPlanes — in their own namespaces, authenticating from user-supplied Secrets — never read, and the suite asserts their own per-CR OpenBao paths are never-seeded. It loads the keystone:2026.1 and tempest:2026.1 service images (but not horizon:2026.1 — External mode never runs a Horizon workload; the horizon-operator is deployed only for its CRD) and runs with E2E_REQUIRE_CONTROLPLANE_STACK: "true" so a broken deployment fails the build instead of the suite skipping.

Path filter: operators/c5c3/** (c5c3) and tests/e2e/c5c3/external-keystone/** (the tests_external_keystone class). The ci:controlplane label also schedules the job, and ci:full forces it on.

e2e-multicluster

Runs the tests/e2e-multicluster/placed-services/ Chainsaw suite across two kind clusters. The target cluster cobaltcore-target runs the infrastructure and no operator. The management cluster cobaltcore-mgmt runs the keystone, barbican, ovn and neutron operators and no infrastructure. The suite creates a Keystone, a Barbican with its BarbicanSecretStore, an OVNCentral and a Neutron on the management cluster and asserts that their children exist on the target. It also runs an OVNChassis and a NeutronMetadataAgent on the management cluster against the OVNCentral placed on the target. The operators reach the target only through a kubeconfig that carries the ServiceAccount token of the target-cluster-access chart, so a verb the chart does not grant shows up as a CR that never reaches Ready. The suite lives outside tests/e2e/: every suite there runs against one cluster, and make e2e sweeps that tree.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs on pull_request events when the e2e-multicluster output of changes is true, the pull request does not come from a fork (github.event.pull_request.head.repo.fork != true), build-e2e-images ended with success, and no job in needs ended with failure or cancelled. always() lets the job start when the Go jobs in needs were skipped. Permissions: contents: read, packages: read

The job runs on self-hosted with timeout-minutes: 90 and continue-on-error: false. Its env sets TARGET_CLUSTER: cobaltcore-target, the registration name the suite's fixtures reference, and MGMT_CLUSTER: cobaltcore-mgmt. The job leaves BARBICAN_SECRET_STORE_GRANTS unset, as e2e-controlplane does: the variable renders the operator's local TokenRequest Role, and a placed store gets its provisioner token on the target cluster, through the chart's grant there. cleanup-e2e-tags lists the job in its needs.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	Create target kind cluster	create-kind-cluster composite action with version: ${{ env.KIND_VERSION }}, config: hack/kind-config.yaml and cluster-name: ${{ env.TARGET_CLUSTER }}. Only the target takes the kind config with its host ports
3	Resolve OVN version	Writes OVN_VERSION to $GITHUB_ENV from hack/ci-resolve-ovn-version.sh, which reads the pin in images/ovn/source.json
4	Load E2E images	load-e2e-images composite action with the image-map output of build-e2e-images. Pulls eight images: keystone-operator, barbican-operator, ovn-operator and neutron-operator at :dev, keystone, barbican and neutron at 2026.1, and ovn:${{ env.OVN_VERSION }}
5	Load service images into the target cluster	kind load docker-image for keystone:2026.1, barbican:2026.1, neutron:2026.1 and the OVN image into cobaltcore-target. The OVN image is a service image here: the databases and northd of the placed OVNCentral run on the target
6	Setup target-cluster infrastructure	setup-e2e-infra composite action with INFRA_ONLY: "true", CLUSTER_NAME: ${{ env.TARGET_CLUSTER }} and WITH_OVN_KERNEL_MODULES: "true". With INFRA_ONLY=true, hack/deploy-infra.sh suspends every CobaltCore operator HelmRelease and scales its Deployment to zero. The chassis runs on the management cluster, but the openvswitch and geneve modules load into the host kernel that both kind clusters share
7	Create the management cluster	hack/deploy-mgmt-cluster.sh with CLUSTER_NAME: ${{ env.MGMT_CLUSTER }}. The script creates a kind cluster without a config file, so the two clusters never contend for a host port. It installs flux-operator and cert-manager, applies the Flux releases whose CRDs the operators' watches need, and installs the rabbitmq-cluster-operator. It installs no CobaltCore operator and leaves the kubectl context on the new cluster
8	Load operator and datapath images into the management cluster	kind load docker-image for the four operator images at :dev, the OVN image and neutron:2026.1 into cobaltcore-mgmt. The DaemonSets of the suite's OVNChassis and NeutronMetadataAgent run on this cluster
9	Deploy keystone-operator	hack/ci-deploy-operator.sh with OPERATOR: keystone, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/keystone-operator and NAMESPACE: keystone-system
10	Deploy barbican-operator	The same script with OPERATOR: barbican, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/barbican-operator and NAMESPACE: barbican-system
11	Deploy ovn-operator	The same script with OPERATOR: ovn, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/ovn-operator and NAMESPACE: ovn-system. No other operator in the job writes a StatefulSet or a PersistentVolumeClaim to the target, so this one exercises those two grants of the chart
12	Deploy neutron-operator	The same script with OPERATOR: neutron, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/neutron-operator and NAMESPACE: neutron-system. It follows the ovn-operator, which installs the OVNCentral CRD
13	Install the target-cluster-access chart on the target	helm install of deploy/target-cluster/target-cluster-access as the release target-cluster-access in the namespace c5c3-access, with --kube-context "kind-$TARGET_CLUSTER", --create-namespace, --set 'namespaces={openstack}' and --set createNamespaces=false. The openstack namespace already exists from step 6
14	Register the target cluster	Reads the token from the Secret target-cluster-access-token in c5c3-access, with up to 30 tries 2 seconds apart, and fails with an ::error:: when it stays empty. Builds a kubeconfig from that token, the Secret's ca.crt and the server URL of kind get kubeconfig --internal. On the management cluster it creates the namespace c5c3-clusters and the Secret cobaltcore-target with the keys kubeconfig and namespaces=openstack, and labels the Secret sigs.k8s.io/multicluster-runtime-kubeconfig=true
15	Write the chainsaw kubeconfig for the target cluster	Writes the admin kubeconfig of the target to _output/cobaltcore-target.kubeconfig. Chainsaw asserts on the target with it. The operators read the registration Secret of step 14
16	Run the two-cluster E2E suite	Creates _output/reports, switches the kubectl context to kind-$MGMT_CLUSTER and runs make e2e-multicluster. The target checks that a cluster answers, that the Secret cobaltcore-target exists in c5c3-clusters and that the chainsaw kubeconfig exists. Then it runs chainsaw test --config tests/e2e-multicluster/chainsaw-config.yaml tests/e2e-multicluster/
17	Dump diagnostic info (management cluster)	always(): switches to kind-$MGMT_CLUSTER and runs hack/ci-dump-diagnostics.sh with OPERATOR: keystone
18	Dump diagnostic info (target cluster)	always(): switches to kind-$TARGET_CLUSTER and runs the script without OPERATOR. The target runs the infrastructure and nothing else, so the script's infrastructure section is the whole dump
19	Upload JUnit report	always(): actions/upload-artifact@v7 uploads _output/reports/ as e2e-multicluster-junit-report (retention-days: 14)
20	Delete kind cluster	always(): runs hack/ci-delete-kind-cluster.sh, which covers both clusters without their names. It never fails the job

tests/e2e-multicluster/chainsaw-config.yaml sets parallel: 1 and an assert timeout of 300s, and writes the JUnit report to _output/reports.

Path filter: tests/e2e-multicluster/** and deploy/target-cluster/** (the tests_multicluster class), or the ci:multicluster label. hack/ci-resolve-changes.sh reads no operator class and no image class for this flag, so a change to one of the four operators alone does not schedule the job. ci:full forces the flag on, and a labeled no-op resolves it to false.

e2e-orders-multicluster

Runs the tests/e2e-orders-multicluster/orders/ Chainsaw suite across two kind clusters. The target cluster cobaltcore-target is bare: it carries the target-cluster-access chart and the two tenant namespaces orders-tenant and orders-outsider, and runs no infrastructure. The management cluster carries the infrastructure, K-ORC, the keystone-operator and the c5c3-operator. The suite applies the seven order kinds on the target, and Jobs there use every Secret the c5c3-operator delivers beside the orders, with the addresses the suite's plane publishes on NodePorts of the management node. The registration Secret lists the tenant namespaces under its assignedNamespaces key, so the keystone-operator engages the same target with a cache on openstack alone (see Registering a target cluster), and a step after the suite fails the job on a forbidden LIST of the keystone-operator in a tenant namespace. The suite's plane publishes Keystone over plain http and carries no gateway. The job deploys two operators where e2e-controlplane deploys ten, which is why the suite runs in a job of its own.

Dependencies: needs: [changes, lint, shellcheck, test, test-integration, verify-codegen, chainsaw-lint, build-e2e-images]Condition: Runs on pull_request events when the e2e-orders-multicluster output of changes is true, the pull request does not come from a fork (github.event.pull_request.head.repo.fork != true), build-e2e-images ended with success, and no job in needs ended with failure or cancelled. always() lets the job start when the Go jobs in needs were skipped. Permissions: contents: read, packages: read

The job runs on self-hosted with timeout-minutes: 110 and continue-on-error: false. The wall is about 25 minutes of bring-up plus the suite's ceiling of 75 minutes (exec 45m, catch 5m, finally 15m, cleanup 10m), rounded up, and is to be confirmed against the first green run. Its env sets OPERATOR_REPLICAS: "1", as e2e-controlplane does, and TARGET_CLUSTER: cobaltcore-target, the registration name the suite's plane references. cleanup-e2e-tags lists the job in its needs.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	Create target kind cluster	create-kind-cluster composite action with version: ${{ env.KIND_VERSION }} and cluster-name: ${{ env.TARGET_CLUSTER }} and no config, so the bare target takes no host port
3	Create kind cluster	The same action with config: hack/kind-config.yaml and cluster-name: ${{ env.KIND_CLUSTER }}. Created second, so the kubectl context ends on the management cluster
4	Load E2E images	load-e2e-images composite action with the image-map output of build-e2e-images. Pulls four images: keystone-operator and c5c3-operator at :dev, keystone and tempest at 2026.1
5	Load images into kind	kind load docker-image for keystone-operator:dev, c5c3-operator:dev and keystone:2026.1 into the management cluster, and for keystone:2026.1 and tempest:2026.1 into cobaltcore-target, where the suite's verify Jobs run
6	Setup E2E infrastructure	setup-e2e-infra composite action with WITH_CONTROLPLANE: "true", CONTROLPLANE_OPERATORS: external, CONTROLPLANE_NAME: controlplane-keystone and WITH_CONTROLPLANE_CR: "false". The suite's plane brings its own bus, and the RabbitMQ operators are part of the base stack, so the job sets no WITH_MESSAGING
7	Install CRDs watched by the c5c3-operator	kubectl apply -f of the keystone, horizon, glance and placement chart CRDs, and hack/ci-deploy-korc.sh with GITHUB_TOKEN, as the c5c3 entry of e2e-operator does
8	Deploy keystone-operator	hack/ci-deploy-operator.sh with OPERATOR: keystone, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/keystone-operator and NAMESPACE: keystone-system
9	Deploy c5c3-operator	The same script with OPERATOR: c5c3, IMAGE_REPO: ${{ env.IMAGE_PREFIX }}/c5c3-operator and NAMESPACE: c5c3-system
10	Create the assigned namespaces on the target	kubectl --context "kind-$TARGET_CLUSTER" create namespace for orders-tenant and orders-outsider. The chart creates no assigned namespace
11	Install the target-cluster-access chart on the target	helm install of deploy/target-cluster/target-cluster-access as the release target-cluster-access in the namespace c5c3-access, with --kube-context "kind-$TARGET_CLUSTER", --create-namespace, --set 'namespaces={openstack}' and --set 'assignedNamespaces={orders-tenant,orders-outsider}'. The default createNamespaces=true creates openstack, which the bare target lacks
12	Register the target cluster	The registration of e2e-multicluster: the token and ca.crt of the Secret target-cluster-access-token, the server URL of kind get kubeconfig --internal, and on the management cluster the namespace c5c3-clusters and the Secret cobaltcore-target with the keys kubeconfig, namespaces=openstack and assignedNamespaces=orders-tenant,orders-outsider, labelled sigs.k8s.io/multicluster-runtime-kubeconfig=true
13	Wait for both operators to engage the target	Reads the logs of the keystone-operator in keystone-system and of the c5c3-operator in c5c3-system every 5 seconds, for up to 180 seconds, until each holds a cluster engaged line for cobaltcore-target, and fails with an ::error:: otherwise. Engagement waits for no informer the controllers start on the target, so the step proves the engagement alone; step 16 checks the keystone-operator's scope
14	Write the chainsaw kubeconfig for the target cluster	Writes the admin kubeconfig of the target to _output/cobaltcore-target.kubeconfig. Chainsaw reaches the target with it. The operators read the registration Secret of step 12
15	Run the orders two-cluster E2E suite	Creates _output/reports, switches the kubectl context to kind-${{ env.KIND_CLUSTER }} and runs make e2e-orders-multicluster with E2E_REQUIRE_CONTROLPLANE_STACK: "true". The target checks that a cluster answers, that the ControlPlane stack is installed, that the Secret cobaltcore-target exists in c5c3-clusters and that the chainsaw kubeconfig exists. Then it runs chainsaw test --config tests/e2e-orders-multicluster/chainsaw-config.yaml tests/e2e-orders-multicluster/
16	Check the keystone-operator never lists an assigned namespace	Reads the keystone-operator's logs in keystone-system after the suite, when its informers on the target have long run their first LIST, and fails with an ::error:: on a forbidden line that names orders-tenant or orders-outsider. A service operator whose cache spanned an assigned namespace would LIST there, where the chart grants it nothing
17	Dump diagnostic info (management cluster)	always(): switches to kind-${{ env.KIND_CLUSTER }} and runs hack/ci-dump-diagnostics.sh with OPERATOR: c5c3
18	Dump diagnostic info (target cluster)	always(): switches to kind-$TARGET_CLUSTER and runs the script without OPERATOR
19	Upload JUnit report	always(): actions/upload-artifact@v7 uploads _output/reports/ as e2e-orders-multicluster-junit-report (retention-days: 14)
20	Delete kind cluster	always(): runs hack/ci-delete-kind-cluster.sh, which covers both clusters. It never fails the job

tests/e2e-orders-multicluster/chainsaw-config.yaml sets parallel: 1 and an assert timeout of 300s, and writes the JUnit report chainsaw-orders-multicluster-report to _output/reports.

Path filter: tests/e2e-orders-multicluster/** and deploy/target-cluster/** (the tests_orders_multicluster class), operators/c5c3/** (the c5c3 class, which schedules the ControlPlane jobs too), or the ci:controlplane or ci:multicluster label. A change to the keystone-operator or a shared Go change alone does not schedule the job. ci:full forces the flag on, and a labeled no-op resolves it to false.

tempest

Tempest API integration tests. Deploys services into a kind cluster and runs the OpenStack Tempest test suite against them. Uses a release matrix to validate each OpenStack release independently, with per-release Tempest configuration, Keystone CRs, and K8s service names. Pulls pre-built images from GHCR (run-scoped tag) via the load-e2e-images composite action.

The figure shows the test bed of one job: Tempest runs in a container on the runner host and reaches the Services through port-forwards, and the seed Jobs run inside the cluster. Data Flow (CI End-to-End) lists the commands of a job in order.

Dependencies: needs: [changes, build-e2e-images, e2e-infra, e2e-operator, e2e-chaos, e2e-prometheus]Condition: Runs only when tempest == 'true', build-e2e-images succeeded, and no other E2E job failed or was cancelled; tempest is the last E2E job in the chain. Without a label it runs for a change to its own sources, and the matrix is narrowed to the services whose configuration changed. Permissions: contents: read, packages: read (required for GHCR pull).

Matrix strategy: the workflow holds no list of legs. hack/ci-generate-tempest-matrix.sh runs in the changes job and crosses ALL_TEMPEST_SERVICES=(keystone glance barbican neutron cinder nova) with every releases/<version>/ directory, writing the result to the tempest-releases output the job consumes as matrix: ${{ fromJson(needs.changes.outputs.tempest-releases) }}. Two releases and six services make twelve legs. A service without a tests/tempest/<service>-<slug> configuration directory fails the generator. TEMPEST_SERVICES, set from the change resolver, narrows the emitted entries to the services a pull request touches; the directory check still covers all six.

Every entry carries service, release, config-dir, cr-name and service-k8s-name, and service-k8s-name equals cr-name (the Keystone identity CR the job waits on and port-forwards). Non-keystone entries add <service>-cr-name for the service CR of the leg. The neutron entries also carry ovn-cr-name and tempest-concurrency: "2". The cinder and nova entries name the compute stack their legs bring up. Both carry ovn-cr-name, neutron-cr-name, placement-cr-name, glance-cr-name and tempest-concurrency: "2"; the cinder entries add nova-cr-name, which on a nova entry is the leg's own service CR. Steps read them as matrix.release, matrix.config-dir, matrix.cr-name and so on. For the names each leg ends up with, see Tempest Test Infrastructure.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	actions/setup-go@v7	Sets up Go with go-version-file: go.work
3	create-kind-cluster composite action	Clears any cluster a cancelled job left on the runner, then creates the kind cluster (cobaltcore) at KIND_VERSION
4	Resolve OVN version	hack/ci-resolve-ovn-version.sh writes OVN_VERSION to $GITHUB_ENV; images/ovn/source.json holds the pin
5	load-e2e-images composite action	Pulls run-scoped GHCR tags and re-tags to canonical local refs; the neutron leg also pulls neutron-operator:dev, neutron:<release>, ovn-operator:dev and ovn:<OVN_VERSION>; the nova and cinder legs pull the four compute-stack operator images plus neutron:<release>, placement:<release>, nova:<release> and ovn:<OVN_VERSION>
6	kind load docker-image	Loads keystone operator and service images into kind
7	kind load docker-image (neutron leg only)	Loads the two operator images, the neutron and OVN service images, and the tempest image the catalog Job runs in-cluster
8	kind load docker-image (nova and cinder legs)	Loads the eight compute-stack refs both legs share in one call, so the layers the service images and the operator images each share are saved and imported once; each leg's glance pair and tempest image come from a step above, the cinder leg's from its own and the nova leg's from the glance one
9	setup-e2e-infra composite action	Installs Flux CLI, test deps, and deploys infra stack. WITH_MESSAGING is set on the cinder and nova legs, WITH_NFS on the cinder leg and WITH_OVN_KERNEL_MODULES on the nova leg
9a	Start the sizing measurement (ci:measure-sizing only)	Starts the background watch of Sizing measurement; step 9 then also carries WITH_VPA
10	hack/ci-deploy-operator.sh	Installs CRDs and deploys operator via Helm
11	hack/ci-deploy-operator.sh ×2 (neutron, nova and cinder legs)	Deploys the ovn-operator into ovn-system and the neutron-operator into neutron-system; a Neutron never reaches Ready without a live OVNCentral, and all three legs run one
12	hack/ci-deploy-operator.sh ×2 (nova and cinder legs)	Deploys the placement- and nova-operator, each into its own <op>-system Namespace; the glance-, ovn- and neutron-operator both legs also need come from the steps above
13	Deploy Keystone CR	Applies matrix.config-dir/00-keystone-cr.yaml and waits for matrix.cr-name Ready
14	Bootstrap network catalog (neutron leg only)	Applies matrix.config-dir/01-catalog-setup-job.yaml and waits 300 s for the neutron-tempest-catalog-setup Job to complete
15	Deploy OVNCentral (neutron leg only)	Applies 02-messaging-secret.yaml and 03-ovncentral-cr.yaml, waits 300 s for ovncentral/ovn-neutron-tempest-<slug> Ready
16	Deploy Neutron CR (neutron leg only)	Applies 04-neutron-cr.yaml, waits 600 s for matrix.neutron-cr-name Ready
17	Bootstrap block-storage catalog (cinder leg only)	Applies matrix.config-dir/01-catalog-setup-job.yaml and waits 300 s for the cinder-tempest-catalog-setup Job, which registers the block-storage, image, compute, placement and network services with their endpoints
18	Bootstrap compute catalog (nova leg only)	Applies matrix.config-dir/01-catalog-setup-job.yaml and waits 300 s for the nova-tempest-catalog-setup Job, which registers the compute, placement, image and network services with their endpoints
19	Apply the independent compute-stack CRs (nova and cinder legs)	Applies *-messaging-secret.yaml, *-ovncentral-cr.yaml and *-placement-cr.yaml, and waits on nothing. Neither CR has an upstream in this set, so both reconcile through the waits below instead of adding their own after them; the steps that wait on them re-apply the same manifests, which is a no-op. The Neutron is deliberately left out: its controller error-requeues until the OVNCentral publishes its database addresses
20	Deploy Glance CR (cinder leg only)	Applies 02-glance-cr.yaml and 03-glancebackend-cr.yaml, waits 300 s for matrix.glance-cr-name Ready; the volume tests create volumes from an image and upload volumes back to one
21	Deploy Cinder CR (cinder leg only)	Applies 04-cinderbackend-cr.yaml, 05-cinderbackupbackend-cr.yaml and 06-cinder-cr.yaml, waits 600 s for matrix.cinder-cr-name Ready. 600 s, because the Cinder brings up four Deployments and a db-sync Job, and the backends have to be attached before the volume and backup services are projected
22	Seed the image the volume tests boot from (cinder leg only)	Applies 07-image-seed-job.yaml and waits 300 s for the cinder-tempest-image-seed Job; tempest.conf pins [compute] image_ref to the UUID it creates
23	Deploy OVNCentral for the compute stack (nova and cinder legs)	Applies *-messaging-secret.yaml and *-ovncentral-cr.yaml, waits 300 s for ovncentral/<matrix.ovn-cr-name> Ready
24	Deploy Neutron CR for the compute stack (nova and cinder legs)	Applies *-neutron-cr.yaml, waits 600 s for matrix.neutron-cr-name Ready
25	Deploy Placement CR (nova and cinder legs)	Applies *-placement-cr.yaml, waits 300 s for matrix.placement-cr-name Ready; the Nova claims its inventory there
26	Deploy Glance CR for the nova leg (nova leg only)	Applies 06-glance-cr.yaml and 07-glancebackend-cr.yaml, waits 300 s for matrix.glance-cr-name Ready
27	Seed the images the compute tests boot from (nova leg only)	Applies 08-image-seed-job.yaml and waits 300 s for the nova-tempest-image-seed Job; tempest.conf pins [compute] image_ref and image_ref_alt to the two UUIDs it creates
28	Deploy Nova CR (nova and cinder legs)	Applies *-metadata-secret.yaml and *-nova-cr.yaml, waits 900 s for matrix.nova-cr-name Ready. 900 s, because the eight MariaDB CRs and the six nova-manage runs of the db-sync put Ready about 8 minutes after the apply
29	Deploy the fake compute (nova and cinder legs)	Applies *-fake-compute.yaml, waits 300 s for the rollout and runs tests/e2e/nova/discover-hosts.sh for the leg's host: fake-1 on the cinder legs, the kind node's name on the nova legs
30	Deploy the OVN chassis (nova leg only)	Labels the node openstack.c5c3.io/chassis=true, applies 12-ovnchassis-cr.yaml, waits 300 s for ovnchassis/ovn-<matrix.nova-cr-name>-chassis Ready; a port binds only to a host with a live chassis
31	Deploy the metadata agent (nova leg only)	Applies 13-neutronmetadataagent-cr.yaml, waits 300 s for neutronmetadataagent/neutron-<matrix.nova-cr-name>-agent Ready; it answers the 169.254.169.254 requests the metadata_service tests read back
32	Seed the flavors (nova and cinder legs)	Applies *-flavor-seed-job.yaml and waits 300 s for the <matrix.service>-tempest-flavor-seed Job; tempest.conf pins [compute] flavor_ref and flavor_ref_alt to the two ids it creates
33	hack/ci-run-tempest.sh	Runs Tempest API tests with CONFIG_DIR=matrix.config-dir, SERVICE_K8S_NAME=matrix.service-k8s-name, and on the neutron, cinder and nova legs NEUTRON_K8S_NAME=matrix.neutron-cr-name (empty elsewhere, which disables the 9696 port-forward). The cinder leg adds CINDER_K8S_NAME=matrix.cinder-cr-name, GLANCE_K8S_NAME=matrix.glance-cr-name and TEMPEST_CONCURRENCY=matrix.tempest-concurrency; the script's optional-target row Cinder:CINDER_K8S_NAME:8776:/healthcheck turns the filled name into an 8776 port-forward polled on /healthcheck. Both compute-stack legs add NOVA_K8S_NAME=matrix.nova-cr-name, whose row Nova:NOVA_K8S_NAME:8774:/ forwards 8774 and polls /, since Nova serves no /healthcheck, and PLACEMENT_K8S_NAME=matrix.placement-cr-name, whose row forwards 8778 and polls / for the same reason — both legs register a placement endpoint in their catalog and declare the service available, so the name has to resolve inside the container; the nova leg also sets NOVA_CONSOLE_K8S_NAME=<matrix.nova-cr-name>-novncproxy, whose row forwards 6080 and polls /vnc_lite.html for test_novnc_bad_token
34	Upload Tempest results	Uploads _output/tempest/ as tempest-<service>-<release>-results artifact (14-day retention)
34a	Collect the sizing measurement (always) (ci:measure-sizing only)	Stops the watch and writes the report, with MEASURE_LEG=<service>-<release>
34b	Upload the sizing measurement (always) (ci:measure-sizing only)	Uploads _output/sizing/ as sizing-tempest-<service>-<release> (14-day retention)
35	hack/ci-dump-diagnostics.sh (always)	Dumps diagnostic info with OPERATOR=keystone
36	hack/ci-dump-diagnostics.sh ×5 or ×6 (always) (nova and cinder legs)	One OPERATOR_ONLY=1 pass per extra operator: nova, placement, ovn, neutron, glance, and cinder on the cinder legs
37	hack/ci-delete-kind-cluster.sh (always)	Deletes the kind cluster; a cluster that survives is a warning, never a job failure

The compute-stack steps name their fixtures by suffix glob, because the numeric prefixes differ between tests/tempest/nova-<slug>/ and tests/tempest/cinder-<slug>/.

Timeout: 68 minutes, 150 for the nova legs and 120 for the cinder legs.

cleanup-e2e-tags

GH-310. Prunes the run-scoped GHCR tags pushed by build-e2e-images (e2e-${run_id}-*) so they don't accumulate on the package page. Runs as a matrix over the E2E target packages after every consumer that might still pull the images has finished. The always() && needs.build-e2e-images.result == 'success' condition means the cleanup runs on success, failure, cancelled, or skipped consumer outcomes — but only when build-e2e-images actually pushed something. A job that uses the load-e2e-images action and is missing from needs races the deletion, and on a run that skips every listed consumer its pull fails with manifest unknown. tests/unit/ci/cleanup_e2e_tags_needs_test.sh fails for every such job.

The package list is the cleanup-e2e-packages output of the changes job, derived by hack/ci-generate-cleanup-matrix.sh from images/ and operators/. It was a hardcoded list until keystone-federation-proxy was left out of it and accumulated 352 stale tags. A package whose image this run reused rather than built carries no run-scoped tag, so the narrowed plan finds no candidate there and deletes nothing. A package GHCR does not know yet, the case for a new images/<name>/ directory until its first publish from main, is reported as a warning and the leg exits 0. A full sweep still fails on an unknown package.

Deletion runs through hack/ghcr-prune-stale-versions.py in --only-tag-pattern mode, scoped to ^e2e-${run_id}-. That mode only considers versions whose tags all match the pattern, so untagged versions are out of scope by construction — the job runs in parallel with build-and-push, which uploads per-platform manifests untagged via push-by-digest and needs those digests intact for merge-operator-images (GH-312).

Dependencies: needs: [changes, build-e2e-images, e2e-operator, e2e-operator-upgrade, e2e-chaos, e2e-prometheus, e2e-controlplane, e2e-controlplane-sso, e2e-autoscaling, e2e-external-keystone, e2e-multicluster, e2e-ovn-overlay, e2e-nova-libvirt, tempest]Permissions: contents: read, packages: write

The job is continue-on-error: pruning is housekeeping, and a package whose only tagged version is the run-scoped one cannot be pruned at all (GHCR refuses to delete a package's last version). The nightly cleanup-e2e-stale-tags job in cleanup-images.yaml is the safety net for anything a cancelled run leaks.

Timeout: 15 minutes.

build-and-push

Builds operator container images per platform on native runners and pushes each single-arch image by digest. Runs only on push events (main branch or v* tags) — skipped on pull requests. The multi-arch manifest list and final tags are assembled by the subsequent merge-operator-images job.

Publish-only-on-merge: the merged commit's PR was already green, so the E2E suite is not re-run on push. The job depends only on changes (for the operator matrix) and builds the image from the flake, independent of build-e2e-images.

Dependencies: needs: [changes]Condition: if: github.event_name == 'push' && needs.changes.outputs.has-e2e-operators == 'true'Permissions: contents: read, packages: write

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	platform-pair composite action	Converts linux/amd64 → linux-amd64 for artifact names
3	registry-login composite action	Authenticates to GHCR (github.actor / GITHUB_TOKEN); skopeo in step 5 reads the credentials docker login wrote
4	setup-nix composite action	Installs Nix and reads the cobaltcore Cachix cache, pushing to it with CACHIX_AUTH_TOKEN
5	build-nix-image composite action	Builds the flake attribute <operator>-operator-image, scans its store paths with vulnix (report artifact vulnix-operator-<operator>-<platform-pair>), pushes the stream by digest with skopeo as an OCI manifest (manifest-format: oci), so the index of merge-operator-images can carry an annotation, and exports the digest as the artifact digests-operator-<operator>-<platform-pair>, retention: 1 day

Matrix strategy:

strategy:
  fail-fast: false
  matrix:
    operator: ${{ fromJson(needs.changes.outputs.e2e-operators).operator }}
    platform:
      - name: linux/amd64
        runner: ubuntu-latest
      - name: linux/arm64
        runner: ubuntu-24.04-arm

The image is nix/lib/operator-image.nix for the operator (Operator images). The Cachix cache of step 4 takes the place of a layer cache: a build whose inputs the cache holds compiles nothing and spends its time in the vulnix scan. Timeout: 30 minutes.

merge-operator-images

Downloads per-platform digests from build-and-push, assembles the multi-arch manifest list, and pushes it with the final tags.

Dependencies: needs: [changes, build-and-push]Condition: if: github.event_name == 'push' && needs.build-and-push.result == 'success'Permissions: contents: read, packages: write

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	docker/setup-buildx-action@v4 + registry-login composite action	Authenticates to GHCR
3	docker/metadata-action@v6	Generates final image tags and, with DOCKER_METADATA_ANNOTATIONS_LEVELS: index, the index annotations (created, revision, source, url, version, and title, description, licenses and vendor as the image labels set them)
4	Download digests	actions/download-artifact@v8
5	Create and push manifest list	Shell

Matrix strategy: Same operator dimension as build-and-push (via fromJson(needs.changes.outputs.e2e-operators)).

Image tagging strategy:

Trigger	Tags Applied
Push to main	sha-<full-sha>, latest
Push v* tag (from main)	sha-<full-sha>, latest, <version> (e.g. 0.1.0, v prefix stripped)
Push v* tag (from non-main)	sha-<full-sha>, <version> (no latest — restricted to default branch)

Images are published at ghcr.io/c5c3/<operator>-operator:<tag>.

helm-push

Packages and pushes operator Helm charts to the GHCR OCI registry. Runs only on push events — skipped on pull requests. Like build-and-push, this is publish-only-on-merge: the chart is packaged and pushed for every changed operator on push without re-running the E2E suite.

Dependencies: needs: [changes]Condition: if: github.event_name == 'push' && needs.changes.outputs.has-e2e-operators == 'true'Permissions: contents: read, packages: write

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	azure/setup-helm@v5	Installs Helm CLI
3	Helm registry login	Authenticates to GHCR via helm registry login
4	Package and push	Packages chart and pushes to oci://ghcr.io/c5c3/charts/

Chart version derivation:

Trigger	Version
Push to main	Default version from Chart.yaml
Push v* tag	SemVer derived from tag (v prefix stripped, e.g. v0.1.0 → 0.1.0)

Matrix strategy: Same operator dimension as build-and-push (via fromJson(needs.changes.outputs.e2e-operators)), so every changed operator's chart is packaged and pushed.

The make helm-package target packages operators/<operator>/helm/<operator>-operator/. When CHART_VERSION is set (for tag pushes), it overrides the version in Chart.yaml.

helm-push-target-cluster

Packages the target-cluster-access chart and pushes it to the GHCR OCI registry. Runs only on push events and is skipped on pull requests. Like helm-push, this is publish-only-on-merge. It is a job of its own because helm-push is a per-operator matrix: a step inside that matrix would package and push this single chart once per matrix entry, and the entries would race on the same registry tag.

Dependencies: needs: [changes]Condition: Runs on push events when the target-cluster-chart output of changes is true. Permissions: contents: read, packages: write

hack/ci-resolve-changes.sh sets target-cluster-chart when the target_cluster_chart class (deploy/target-cluster/**) changed, and forces it on for a v* tag push.

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	azure/setup-helm@v5	Installs Helm CLI
3	Login to GHCR	Authenticates to ghcr.io via helm registry login with github.actor and secrets.GITHUB_TOKEN
4	Package and push Helm chart	Removes stale target-cluster-access-*.tgz files, runs make helm-package-target-cluster and pushes the tarball to oci://ghcr.io/c5c3/charts

Chart version derivation:

Trigger	Version
Push to main	Default version from Chart.yaml
Push v* tag	The tag name without its v prefix, passed as CHART_VERSION="${GITHUB_REF_NAME#v}"

The make helm-package-target-cluster target packages deploy/target-cluster/target-cluster-access/. When CHART_VERSION is set, the target passes it to helm package as --version, which overrides the version in Chart.yaml. The target runs no helm dependency build, because the chart has no subchart.

No other job lists helm-push-target-cluster in needs. github-release waits for helm-push only, and its files: "*-operator-*.tgz" pattern does not match this chart's tarball.

Runs on ubuntu-latest. Timeout: 15 minutes.

github-release

Creates a GitHub Release with auto-generated release notes on v* tag pushes.

Dependencies: needs: [changes, merge-operator-images, helm-push]Condition: if: startsWith(github.ref, 'refs/tags/v') && needs.merge-operator-images.result == 'success' && needs.helm-push.result == 'success'Permissions: contents: write

Step	Action	Details
1	actions/checkout@v7	Checks out the repository (SHA-pinned)
2	azure/setup-helm@v5	Installs Helm CLI for chart packaging
3	Package Helm charts	Packages operator Helm charts with release version
4	softprops/action-gh-release@v3	Creates release with generate_release_notes: true and attaches chart tarballs

This job runs only after both merge-operator-images and helm-push complete successfully, ensuring the final multi-arch manifest list and charts are published before the release is created. Helm chart tarballs are attached as release assets for direct download. Timeout: 8 minutes.

Reusable CI Scripts

Repeated inline shell logic from E2E jobs is extracted into standalone scripts under hack/. Each script uses set -euo pipefail, includes an SPDX Apache-2.0 header, and passes shellcheck. All scripts are designed to work both in CI and locally against any kubeconfig.

hack/ci-dump-diagnostics.sh

Dumps diagnostic information after E2E failures. Shared across every e2e-* job and the tempest job.

Environment Variable	Required	Default	Description
OPERATOR	No	(empty)	When set, emits operator-specific diagnostics (pod logs, CR status, job logs)
OPERATOR_ONLY	No	(empty)	When set, emits only the three sections that differ per operator (operator pods, operator logs, CR status). For callers that loop over several operators in one job, where the infrastructure block and the NAMESPACE Job and pod logs are the same dump every pass
NAMESPACE	No	openstack	Kubernetes namespace for operator-specific queries

Infrastructure diagnostics (always emitted): HelmReleases, pods, DaemonSets, the K-ORC controller log (deploy/orc-controller-manager in orc-system, a SKIP: line with kubectl's reason on a cluster without it), events (last 50), and Flux logs across all namespaces.

Operator diagnostics (when OPERATOR is set): Operator pods and logs, job descriptions and logs in the target namespace, all pod logs (current and previous) in the namespace, operator CR status conditions, and ConfigMaps.

Usage:

hack/ci-dump-diagnostics.sh                    # infra-only diagnostics
OPERATOR=keystone hack/ci-dump-diagnostics.sh   # + operator-specific diagnostics
OPERATOR=nova OPERATOR_ONLY=1 hack/ci-dump-diagnostics.sh  # the operator sections alone

hack/ci-deploy-operator.sh

Deploys an operator into a kind cluster by installing CRDs, waiting for establishment, and deploying the operator via Helm with the specified container image.

Environment Variable	Required	Default	Description
OPERATOR	Yes	-	Operator name (e.g. keystone)
IMAGE_REPO	Yes	-	Full image repository (e.g. ghcr.io/c5c3/keystone-operator)
IMAGE_TAG	No	dev	Image tag
DEFAULT_IMAGE_PULL_POLICY	No	IfNotPresent	The operator's default image pull policy (Always, IfNotPresent or Never), passed as --set controller.defaultImagePullPolicy=<value>. Any other value exits 1 with an ::error:: line before anything is applied.

The script runs kubectl apply --server-side --force-conflicts -f <chart>/crds/, waits for CRD establishment, then runs helm install with image.pullPolicy=Never (suitable for kind-loaded images) and controller.defaultImagePullPolicy=IfNotPresent, so the workloads run the service images CI loaded into kind under the published tag. The script passes controller.defaultImagePullPolicy only when the chart's values.schema.json names it: a released chart installed through CHART_DIR that predates the value rejects the unknown key, and its operator sets no pull policy anyway. The banner line Image pull policy shows the value, or <chart has no such value>. Server-side apply keeps the CRD out of the 262,144-byte last-applied-configuration annotation that client-side apply writes, which the Nova and Cinder CRDs exceed.

Usage:

OPERATOR=keystone IMAGE_REPO=ghcr.io/c5c3/keystone-operator hack/ci-deploy-operator.sh

hack/ci-nix-build-e2e-images.sh

Builds the Nix images of an e2e run inside the nixos/nix container step Build Nix images of build-e2e-images, with the workspace as its working directory. It reads the image set hack/ci-resolve-e2e-images.sh wrote.

Environment Variable	Required	Default	Description
BUILD_SERVICE_IMAGES	No	(empty)	One <service> <release> [<image>] line per image; a line without a third field builds the service's own image, a line with one (nova-compute) the attribute of that derived image
BUILD_TEMPEST_RELEASES	No	(empty)	The releases whose Tempest image is built
BUILD_OVN	No	(empty)	true builds the ovn image, the flake attribute ovn-image
OVN_VERSION	With BUILD_OVN	(empty)	The pinned OVN version, the tag docker load prints for the ovn image; hack/ci-resolve-e2e-images.sh writes it
BUILD_LIBVIRT	No	(empty)	true builds the libvirt image, the flake attribute libvirt-image
BUILD_OPERATORS	No	(empty)	One operator name per line; each builds the flake attribute <op>-operator-image
GITHUB_WORKSPACE	Yes	-	The checkout the flake is evaluated in; set by the runner

The script writes the group e2ebld (gid 40000) and one build user per CPU (uids from 40001) to the container's account files (NIX_E2E_ETC, /etc by default), then builds the flake attributes <name>-<rel_>-image of every image (<name> the third field of a line that has one), ovn-image with BUILD_OVN, libvirt-image with BUILD_LIBVIRT and <op>-operator-image for each name of BUILD_OPERATORS, in one nix build with --option sandbox false, --option build-users-group e2ebld and one job per CPU. The builders, which run the setup code of every sdist the locks build, run as those users: the runner mounts its Docker socket and the workspace into the container, and a build user can neither open the socket nor write the workspace. Their uids are not those of a host Nix daemon's build users. It then writes each image stream to .nix-e2e/<name>-<release>.tar and appends <name> <release> to .nix-e2e/images.txt; the ovn image is listed as ovn <OVN_VERSION>, the libvirt image as libvirt latest and an operator image as <op>-operator latest, the tag docker load prints for it. A line with four fields exits 1 with ci-nix-build-e2e-images.sh: malformed line '<line>', a BUILD_OVN without OVN_VERSION with ci-nix-build-e2e-images.sh: BUILD_OVN is true but OVN_VERSION is empty, both before nix runs, and a missing attribute or a failed build with nix's exit status; an empty image set prints ci-nix-build-e2e-images.sh: nothing to build and exits 0. Nix substitutes from cache.nixos.org and, once the script's CACHIX_PUBLIC_KEY constant names the key of the cobaltcore cache, from that cache. Before anything else it adds the workspace to git's safe.directory, because the container's root does not own the checkout, and its EXIT trap hands .nix-e2e/ back to the owner of the workspace. tests/unit/ci/nix_e2e_build_test.sh drives it with stub nix, git and nproc binaries and scratch account files.

hack/ci-resolve-ovn-version.sh

Prints the OVN version pinned in images/ovn/source.json. It reads the "version" key with jq and writes the tag without its leading v to stdout, so v26.03.2 becomes 26.03.2. The Nix build and hack/nix-lock.sh read the file themselves; everywhere else in CI the script parses that key: the merge-ovn-image job in build-images.yaml, the e2e jobs of this workflow, hack/ci-resolve-e2e-images.sh and tests/container-images/verify_ovn.sh call it.

Environment Variable	Required	Default	Description
OVN_SOURCE	No	images/ovn/source.json	Pin file whose "version" key is read

Five cases make the script print an ::error:: annotation on stderr and exit 1: jq is not on PATH, the pin file does not exist, it is not valid JSON, it has no "version", or the value is not a vX.Y.Z tag. Annotations go to stderr, so stdout carries the version and nothing else.

Usage:

hack/ci-resolve-ovn-version.sh
OVN_SOURCE=path/to/source.json hack/ci-resolve-ovn-version.sh

hack/ci-run-tempest.sh

CI-specific Tempest execution wrapper that handles port-forwarding, config generation, and Docker-based test execution. This is the CI counterpart to hack/run-tempest.sh (which handles local execution including image building).

Environment Variable	Required	Default	Description
SERVICE	No	keystone	Service under test
CONFIG_DIR	No	tests/tempest/<SERVICE>-2026-1	Directory containing tempest.conf template and include/exclude lists
NAMESPACE	No	openstack	Kubernetes namespace
ADMIN_SECRET	No	keystone-admin	Secret name holding admin password
OUTPUT_DIR	No	_output/tempest	Test output directory
TEMPEST_IMAGE	No	c5c3/tempest:local	Tempest container image
SERVICE_K8S_NAME	No	<SERVICE>-tempest-2026-1	K8s Service name for port-forwarding; the CI matrix passes its service-k8s-name entry for release-specific CR names (e.g. keystone-tempest-2026-2)

The script:

1. Extracts the admin password from the Kubernetes secret
2. Sets up kubectl port-forward to the service and waits for readiness
3. Generates tempest.conf from the template, substituting endpoint and credentials
4. Runs Tempest in a Docker container with --network host and host-alias DNS entries
5. Converts subunit output to JUnit XML and checks for failures

Usage:

hack/ci-run-tempest.sh
SERVICE=keystone OUTPUT_DIR=_output/tempest hack/ci-run-tempest.sh

Composite Action: create-kind-cluster

.github/actions/create-kind-cluster/action.yaml

A composite GitHub Action that creates the kind cluster for every cluster-bound job. It replaces the bare helm/kind-action step those jobs used to carry. That step creates a cluster and does nothing else, which is a problem on the self-hosted runners, because they are reused between jobs.

The failure it removes. Nothing in the runner's teardown deletes a kind cluster, and a job GitHub cancels mid-flight never reaches the post step that would (every push to a pull request cancels the run in progress). Its node containers outlive the job, and the next e2e job scheduled onto that host fails within its first minute:

ERROR: failed to create cluster: node(s) already exist for a cluster with the name "cobaltcore"

A leftover under another name fails the job differently. hack/kind-config.yaml and hack/kind-config-multinode.yaml bind the same two host ports, so one stale control-plane node blocks every cluster on that runner.

Step	Description
1	helm/kind-action with install_only: true and ignore_failed_clean: true: installs the pinned kind and kubectl binaries and registers the post-job kind delete cluster as a best-effort net, without creating anything
2	hack/ci-create-kind-cluster.sh: resets the runner through hack/ci-reset-kind-cluster.sh, creates the cluster, and retries a failed creation

Input	Default	Description
version	(required)	kind version to install; every caller passes ${{ env.KIND_VERSION }}
cluster-name	(required)	The cluster to create, and the one the post-job net aims at when a cancelled job never reaches its own teardown step
config	''	kind config file. Empty creates a cluster with no config, the way hack/deploy-mgmt-cluster.sh creates the management cluster

What the reset removes. hack/ci-reset-kind-cluster.sh deletes every kind cluster with node containers on the host, found through the io.x-k8s.kind.cluster label kind stamps on them. kind delete cluster runs first, because it also drops the cluster's kubeconfig entry. docker rm -f then takes whatever survives, a node container kind itself no longer recognises included. A leftover that survives both fails the step instead of letting the creation hit the same error again. The registry pull-through caches of hack/deploy-infra.sh carry a label of their own and stay.

That force-remove is retried, because the docker daemon rejects a removal whose container it has not yet seen exit:

Error response from daemon: cannot remove container "cobaltcore-control-plane": could not kill container: tried to kill container, but did not receive an exit event

The exit event arrives seconds later, so three attempts five seconds apart (KIND_RM_ATTEMPTS, KIND_RM_RETRY_DELAY) separate that race from a container that is genuinely stuck. The daemon's own message is printed either way.

Sweeping every cluster is safe because an e2e job owns its runner while it runs: two clusters on one host contend for those host ports, so a cluster already there when the job starts cannot belong to a job running alongside it. The sweep also clears a stale cobaltcore-mgmt that hack/deploy-mgmt-cluster.sh would otherwise adopt as an existing cluster. A job that creates a second cluster of its own keeps it: the first reset in a job leaves a marker in RUNNER_TEMP, and later resets remove only the cluster they are about to create.

The retry. A failed creation leaves its partial cluster on the host, so the reset in front of the next attempt is what makes the retry a clean one. Two attempts is the default. A job that wants more sets KIND_CREATE_ATTEMPTS in its own env, and KIND_WAIT overrides the 60s control-plane wait the same way. Before each retry, and before the step fails for good, the script prints the node containers on the host and the tail of their logs. The cluster never came up, so hack/ci-dump-diagnostics.sh has no API server to read instead.

The teardown. Every cluster-bound job ends with a hack/ci-delete-kind-cluster.sh step, guarded by if: always(). It runs the same sweep at KIND_RESET_SCOPE=all, which needs no cluster name and therefore covers the jobs that bring up two (e2e-multicluster). It is the last step of the job, so hack/ci-dump-diagnostics.sh above it still reads a live cluster.

It cannot fail a job. A cluster that survives the teardown is a ::warning::, never a failure. Before the step existed, the deletion was left to the post step of helm/kind-action — one kind delete cluster, no retry, and its exit code is the job's. Run 34714006750 is what that cost: every suite of e2e-infra passed, the post step then lost the race above, and a job with nothing wrong in it was reported red. Nothing depends on the deletion succeeding right then. The reset in front of the next job on that runner clears whatever is left, and the runner's own job-completed hook usually gets there first.

Usage in a workflow job:

- name: Create kind cluster
  uses: ./.github/actions/create-kind-cluster
  with:
    version: ${{ env.KIND_VERSION }}
    config: hack/kind-config.yaml
    cluster-name: ${{ env.KIND_CLUSTER }}

# ... every other step of the job ...

- name: Delete kind cluster
  if: always()
  run: hack/ci-delete-kind-cluster.sh

Pinned by four shell suites: tests/unit/ci/create_kind_cluster_wiring_test.sh asserts that no job creates a cluster on its own and that every job that creates one ends with the teardown, tests/unit/hack/ci_reset_kind_cluster_test.sh covers the sweep, tests/unit/hack/ci_create_kind_cluster_test.sh covers the creation and its retry, and tests/unit/hack/ci_delete_kind_cluster_test.sh covers the teardown.

Composite Action: setup-test-deps

.github/actions/setup-test-deps/action.yaml

A composite GitHub Action that encapsulates the shared cache + make install-test-deps step used by every job that needs the pinned chainsaw/flux/kind/kubectl binaries. Extracted so the cache key, restore-keys:, and PATH wiring live in one place: setup-e2e-infra (cluster-bound jobs) and the lightweight chainsaw-lint job both consume this and inherit any future tweaks (key bump, additional pinned tool) for free.

Step	Description
1	Restores $HOME/.local/bin from cache, keyed on the hash of hack/install-test-deps.sh (auto-invalidates when any pinned tool version changes)
2	Runs make install-test-deps (no-op on cache hit thanks to the script's skip-if-correct-version logic) and appends ~/.local/bin to GITHUB_PATH

The action takes no inputs.

Composite Action: setup-e2e-infra

.github/actions/setup-e2e-infra/action.yaml

A composite GitHub Action that encapsulates the shared Flux CLI + test dependencies + infrastructure deployment sequence used by every e2e-* job and the tempest job. This replaces a step sequence duplicated in each of those jobs with a single uses: reference.

Prerequisite: A kind cluster must already exist (the action sets SKIP_KIND_CREATE=true internally).

Step	Description
1	Installs Flux CLI via fluxcd/flux2/action@v2 (SHA-pinned)
2	Delegates to the setup-test-deps composite action (cache restore + make install-test-deps + PATH wiring)
3	Runs make deploy-infra with SKIP_KIND_CREATE=true

Usage in a workflow job:

- name: Setup E2E infrastructure
  uses: ./.github/actions/setup-e2e-infra

The action takes no inputs. All configuration is handled by existing Makefile targets and environment variables.

The Deploy infrastructure stack step threads the opt-in flags WITH_CHAOS_MESH, WITH_PROMETHEUS, WITH_OVN_KERNEL_MODULES, WITH_NFS, WITH_METRICS_SERVER, WITH_VPA and WITH_MESSAGING from the calling step's env; an unset flag reaches hack/deploy-infra.sh empty and takes its false default. WITH_VPA=true implies WITH_METRICS_SERVER=true inside the script, whatever the calling step sets.

Composite Action: load-e2e-images

.github/actions/load-e2e-images/action.yaml

A composite GitHub Action that pulls pre-built E2E images from GHCR and re-tags them to their canonical local references so downstream kind load docker-image calls work unchanged. Shared between the tempest job and every e2e-* job except e2e-infra.

Step	Description
1	The registry-login composite action authenticates to GHCR using the workflow's GITHUB_TOKEN
2	For each input ref, docker pull the reference image-map gives for it, then docker tag to the canonical local ref

The map holds one of two forms per ref: <repo>:e2e-<run-id>-<tag> for an image this run built, or <repo>@sha256:... for one it reused from main. A ref the map does not carry falls back to the run-scoped tag, and so does every ref when image-map is left empty. A malformed map fails the step before the first pull.

Input	Default	Description
run-id	${{ github.run_id }}	Run ID used as the tag prefix (e2e-<run-id>-)
images	(required)	Multiline list of canonical local refs (e.g. ghcr.io/c5c3/keystone:2026.1); blank/comment lines are ignored
image-map	''	The image-map output of build-e2e-images; empty means pull every ref by run-scoped tag
registry	ghcr.io	Registry to authenticate against
username	${{ github.actor }}	Login user
password	${{ github.token }}	Login token

Usage in a workflow job:

- name: Load E2E images
  uses: ./.github/actions/load-e2e-images
  with:
    image-map: ${{ needs.build-e2e-images.outputs.image-map }}
    images: |
      ${{ env.IMAGE_PREFIX }}/keystone-operator:dev
      ${{ env.IMAGE_PREFIX }}/keystone:2026.1

GH-310 replaced the previous actions/download-artifact + zstd | docker load sequence: the 355 MB single-blob artifact intermittently timed out at the five-minute window (actions/download-artifact has no built-in retry on a stalled download). Layer-level pull retries plus the GHCR CDN dramatically reduce the failure rate.

How the Pieces Fit Together

The E2E jobs follow a common pattern with shared components:

1. Checkout + Go setup + kind cluster creation     (workflow steps)
2. Pull pre-built images from GHCR                  (load-e2e-images composite action)
3. Load images into kind                            (workflow steps)
4. Deploy infrastructure                            (setup-e2e-infra composite action)
5. Deploy operator                                  (hack/ci-deploy-operator.sh)
6. Run tests                                        (chainsaw / hack/ci-run-tempest.sh)
7. Dump diagnostics                                 (hack/ci-dump-diagnostics.sh)
8. Upload artifacts                                 (workflow steps)

Image building is centralised in build-e2e-images, which runs once before the E2E jobs and pushes the images it built to GHCR under a run-scoped tag. An image it reused from main is pulled by digest and carries no run-scoped tag. The e2e-infra job uses steps 1, 4, 6-8 (no operator or service images needed). The tempest job and every e2e-* job except e2e-infra use all steps, pulling their required images from GHCR via load-e2e-images. The e2e-chaos job uses a chaos-specific Chainsaw config (tests/e2e-chaos/chainsaw-config.yaml) and test directory (tests/e2e-chaos/). The tempest job additionally deploys a Keystone CR before running hack/ci-run-tempest.sh instead of Chainsaw. The cleanup-e2e-tags job prunes the run-scoped tags after every consumer finishes.

Go Setup Convention

All Go-based jobs use actions/setup-go@v7 with:

go-version-file: go.work

This reads the Go version from go.work (currently Go 1.27.2) rather than hardcoding a go-version value. The repository root contains go.work (not go.mod) because the project uses a Go Workspace with multiple modules (internal/common, operators/keystone, operators/c5c3). Module dependency caching is enabled by default in actions/setup-go@v7.

Concurrency

The workflow uses a concurrency group scoped per-branch per-workflow:

concurrency:
  group: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !startsWith(github.event.label.name, 'ci:') && github.event.label.name != 'run-chaos' && format('noop-{0}', github.run_id) || format('{0}-{1}', github.ref, github.workflow) }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

For pull requests, pushing new commits cancels any in-progress CI run for that same PR branch, preventing wasted CI resources on outdated code. For pushes to main, in-progress runs are not cancelled, ensuring every merge commit is fully validated. Different branches do not cancel each other's runs.

Applying a label that does not steer CI used to cancel the run in flight and start it again from scratch, because every labeled event landed in the branch's group. Such an event now gets a group of its own, keyed on the run id, and the changes job resolves it to nothing; the run costs one ubuntu-latest runner for about twenty seconds. A ci:* label (or run-chaos) stays in the branch's group and is meant to supersede the run in flight, since it asks for jobs that run did not schedule.

Action Pinning

All GitHub Actions except actions/cache, which is referenced by its v6 tag, are referenced by full SHA hash with a trailing version comment:

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

This prevents supply chain attacks via mutable tag retargeting and provides audit traceability. The version comment preserves human readability.

SPDX Header

The file starts with the standard SPDX license header:

# SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company
#
# SPDX-License-Identifier: Apache-2.0
---

Codecov Configuration

.codecov.yml defines coverage status checks and component-level thresholds.

Ignored Paths

The top-level ignore block drops generated code from every coverage denominator (project, patch, and component targets):

Pattern	Reason
**/zz_generated*.go	controller-gen DeepCopy plumbing — mechanically generated, no hand-written logic to test. Counting it understates real coverage, notably for the webhooks component (target 90%), whose api/ paths include the generated deepcopy file.

This mirrors the lint exclusion of the same files (see the note above the verify-codegen job).

Status Checks

Check	Target	Description
Project	auto (threshold: 1%)	Overall coverage must not decrease by more than 1%
Patch	90%	New/changed lines in a PR must meet 90% coverage

fail_ci_if_error: false is set on each codecov/codecov-action step in the workflow (not in .codecov.yml, where it is not a valid key) because fork PRs do not have access to CODECOV_TOKEN. This prevents CI from failing due to upload issues on forks.

Flag Management

The flag_management section in .codecov.yml links CI-uploaded flags to coverage tracking rules. Flags follow the [unit|integration]-<target> naming convention, matching the CI matrix targets (common, keystone, c5c3, horizon, glance, placement, barbican, ovn, neutron, cinder, nova). Each flag has carryforward: true, which ensures that when only a subset of flags is uploaded (e.g., only one operator changed), the missing flags carry forward their last-known coverage instead of reducing the total.

Defined flags:

Flag	Paths	Source
unit-common	internal/common/	test job, common matrix leg
unit-keystone	operators/keystone/	test job, keystone matrix leg
unit-c5c3	operators/c5c3/	test job, c5c3 matrix leg
unit-horizon	operators/horizon/	test job, horizon matrix leg
unit-glance	operators/glance/	test job, glance matrix leg
unit-placement	operators/placement/	test job, placement matrix leg
unit-barbican	operators/barbican/	test job, barbican matrix leg
unit-ovn	operators/ovn/	test job, ovn matrix leg
unit-neutron	operators/neutron/	test job, neutron matrix leg
unit-cinder	operators/cinder/	test job, cinder matrix leg
unit-nova	operators/nova/	test job, nova matrix leg
integration-common	internal/common/	test-integration job, common matrix leg
integration-keystone	operators/keystone/	test-integration job, keystone matrix leg
integration-c5c3	operators/c5c3/	test-integration job, c5c3 matrix leg
integration-horizon	operators/horizon/	test-integration job, horizon matrix leg
integration-glance	operators/glance/	test-integration job, glance matrix leg
integration-placement	operators/placement/	test-integration job, placement matrix leg
integration-barbican	operators/barbican/	test-integration job, barbican matrix leg
integration-ovn	operators/ovn/	test-integration job, ovn matrix leg
integration-neutron	operators/neutron/	test-integration job, neutron matrix leg
integration-cinder	operators/cinder/	test-integration job, cinder matrix leg
integration-nova	operators/nova/	test-integration job, nova matrix leg

Component Thresholds

Each component is tracked independently on the Codecov dashboard:

Component	Paths	Target	Rationale
common	internal/common/**	80%	Shared library code underpinning all operators
controllers	operators/*/internal/controller/**	70%	Controller reconciliation logic (envtest-dependent paths harder to cover)
webhooks	operators/*/api/**	90%	Webhook validation/defaulting (incorrect admission logic causes silent data corruption)

Makefile Targets

The CI workflow depends on several Makefile targets:

image

Builds the operator image, the flake attribute <operator>-operator-image, with Nix and flakes (Nix Image Build), loads it into Docker as <operator>-operator:latest and tags it IMG.

make image OPERATOR=keystone [IMG=custom:tag]

The IMG variable controls the image tag, defaulting to ghcr.io/c5c3/<operator>-operator:latest. The OPERATOR variable is required. The CI workflow does not call the target: build-e2e-images and build-and-push build the flake attribute directly.

helm-package

Packages the operator Helm chart from operators/<operator>/helm/<operator>-operator/.

make helm-package OPERATOR=keystone [CHART_VERSION=1.2.3]

When CHART_VERSION is set, it overrides the version in the chart's Chart.yaml. The packaged .tgz is output to the current directory. The OPERATOR variable is required.

test-common

Runs unit tests for internal/common only, producing a single coverage profile.

make test-common

Produces cover-unit-common.out. Used by the common matrix leg in the test CI job to deduplicate common coverage into a single upload.

test-operator

Runs unit tests for a single operator without internal/common.

make test-operator OPERATOR=keystone

Produces cover-unit-<operator>.out. Used by operator matrix legs in the test CI job. The OPERATOR variable is required.

test-integration

Runs envtest-based integration tests (tagged with //go:build integration) for operators. Requires setup-envtest to be installed.

make test-integration [OPERATOR=keystone]

Sets KUBEBUILDER_ASSETS via setup-envtest use <pinned-k8s-version> -p path, then runs go test -tags=integration for each operator module. Produces cover-integration-<operator>.out files. Without OPERATOR, runs for all operators in the OPERATORS list.

test-integration-common

Runs envtest-based integration tests for internal/common only.

make test-integration-common

Sets KUBEBUILDER_ASSETS via setup-envtest use <pinned-k8s-version> -p path, then runs go test -tags=integration ./internal/common/.... Produces cover-integration-common.out. Used by the common matrix leg in CI to meet the 80% codecov target for internal/common/.

Dependencies on Prior Features

The CI workflow depends on the following artifacts:

Artifact	Used by	Purpose
Makefile (lint target)	lint job	Iterates over OPERATORS variable to run golangci-lint per module
Makefile (test-common target)	test job (common leg)	Runs unit tests for internal/common with coverage profile
Makefile (test-operator target)	test job (operator legs)	Runs unit tests for a single operator with coverage profile
Makefile (test-integration target)	test-integration job (operator legs)	Runs envtest integration tests per operator with coverage profiles
Makefile (test-integration-common target)	test-integration job (common leg)	Runs envtest integration tests for internal/common with coverage profile
Makefile (image target)	none, the local build only	Builds and loads an operator image; build-e2e-images builds the flake attribute through hack/ci-nix-build-e2e-images.sh
Makefile (nix-lock-check-operators target)	verify-codegen job	Checks the operator vendor hashes against the Go module files
Makefile (helm-package target)	helm-push job	Packages operator Helm charts
.golangci.yml	lint job	Provides linter configuration (enabled linters, exclusion rules, timeout)
go.work	All Go-based jobs	Provides the Go version for actions/setup-go@v7
hack/*.sh	shellcheck job	Shell scripts validated by shellcheck
.codecov.yml	Codecov integration	Component-level coverage thresholds
hack/ci-dump-diagnostics.sh	every e2e-* job and the tempest job	Shared diagnostic dump
hack/ci-nix-build-e2e-images.sh	build-e2e-images job, in the nixos/nix container	Builds the Nix-built operator, service, nova-compute, Tempest, ovn, libvirt and federation proxy images and writes their streams to .nix-e2e/
flake.nix, flake.lock, nix/locks/	build-e2e-images job	The flake the e2e images are built from
hack/ci-deploy-korc.sh	e2e-operator (c5c3 leg), e2e-controlplane, e2e-controlplane-sso, e2e-autoscaling, e2e-external-keystone jobs	Applies K-ORC from an authenticated clone at the pinned commit
hack/ci-deploy-operator.sh	the tempest job and every e2e-* job except e2e-infra	Deploys operator via Helm; e2e-controlplane sets OPERATOR_REPLICAS=1
hack/ci-check-node-budget.sh	e2e-controlplane job, through the full-chain suite's Link 6z	Fails when the pods on the kind node request more than 4000m CPU or 16Gi memory
hack/ci-vpa-recommendations.sh	e2e-controlplane, e2e-controlplane-sso, tempest jobs under ci:measure-sizing, and by hand in the lab measurement	Records the VPA recommendations of the openstack workloads for the sizing measurement; its prepare subcommand removes the MariaDB scale subresource on the lab, which refuses WITH_VPA=true
hack/ci-run-tempest.sh	tempest job	Runs Tempest API tests
.github/actions/setup-test-deps/	chainsaw-lint job, setup-e2e-infra composite action	Composite action for testdeps cache + make install-test-deps
.github/actions/setup-e2e-infra/	every e2e-* job and the tempest job	Composite action for infra setup
.github/actions/load-e2e-images/	the tempest job and every e2e-* job except e2e-infra	Composite action that pulls run-scoped GHCR tags and re-tags them to canonical local refs (GH-310)
hack/ghcr-prune-stale-versions.py	cleanup-e2e-tags job, cleanup-images.yaml	Deletes GHCR package versions that carry no keeper tag; resolves multi-arch children and cosign referrers first
hack/ci-generate-cleanup-matrix.sh	changes job, cleanup-images.yaml	Derives the GHCR package lists from images/ and operators/
tests/e2e-chaos/chainsaw-config.yaml	e2e-chaos job	Chaos-specific Chainsaw configuration