Infrastructure Manifests

Reference documentation for the FluxCD infrastructure manifests. These manifests define HelmRepository sources, HelmRelease operators, and infrastructure custom resources that provision the shared platform services required by OpenStack operators. Deployment is split into two phases: base resources (namespaces, sources, releases) and CRD-dependent infrastructure resources (applied after operators install their CRDs).

Directory Layout

deploy/
└── flux-system/
    ├── kustomization.yaml                Base kustomize overlay (namespaces, FluxInstance, sources, releases)
    ├── namespaces.yaml                   Namespace resources for all components
    ├── fluxinstance.yaml                 FluxInstance CR driving the flux-operator
    ├── sources/                          Flux sources (HelmRepository, OCIRepository, GitRepository)
    │   ├── cert-manager.yaml             Jetstack Helm chart registry
    │   ├── mariadb-operator.yaml         MariaDB Operator Helm chart registry
    │   ├── external-secrets.yaml         External Secrets Operator Helm chart registry
    │   ├── openbao.yaml                  OpenBao Helm chart registry
    │   ├── openbao-operator.yaml         OpenBao Operator OCI chart artifact (digest-pinned OCIRepository)
    │   ├── c5c3-charts.yaml              C5C3 shared OCI chart registry
    │   ├── k-orc.yaml                    K-ORC GitRepository (commit pinned)
    │   ├── rabbitmq-cluster-operator.yaml RabbitMQ Cluster Operator GitRepository (tag + commit pinned)
    │   ├── messaging-topology-operator.yaml RabbitMQ Messaging Topology Operator GitRepository (tag + commit pinned)
    │   ├── prometheus-community.yaml     Prometheus Community OCI chart registry
    │   └── garage-operator.yaml          Garage Operator OCI chart registry
    ├── releases/                         HelmRelease CRs and three Flux Kustomizations
    │   ├── cert-manager.yaml             cert-manager
    │   ├── prometheus-operator-crds.yaml Prometheus Operator CRDs
    │   ├── mariadb-operator-crds.yaml    MariaDB Operator CRDs
    │   ├── mariadb-operator.yaml         MariaDB Operator
    │   ├── external-secrets.yaml         External Secrets Operator
    │   ├── memcached-operator.yaml       Memcached Operator (from c5c3-charts)
    │   ├── openbao.yaml                  OpenBao HA Raft cluster
    │   ├── openbao-operator.yaml         OpenBao Operator (per-service OpenBao instances)
    │   ├── garage-operator.yaml          Garage Operator
    │   ├── keystone-operator.yaml        Keystone Operator (from c5c3-charts)
    │   ├── horizon-operator.yaml         Horizon Operator (from c5c3-charts)
    │   ├── glance-operator.yaml          Glance Operator (from c5c3-charts)
    │   ├── placement-operator.yaml       Placement Operator (from c5c3-charts)
    │   ├── barbican-operator.yaml        Barbican Operator (from c5c3-charts)
    │   ├── ovn-operator.yaml             OVN Operator (from c5c3-charts)
    │   ├── neutron-operator.yaml         Neutron Operator (from c5c3-charts)
    │   ├── cinder-operator.yaml          Cinder Operator (from c5c3-charts)
    │   ├── nova-operator.yaml            Nova Operator (from c5c3-charts)
    │   ├── k-orc.yaml                    K-ORC (Flux Kustomization over config/default)
    │   ├── rabbitmq-cluster-operator.yaml RabbitMQ Cluster Operator (Flux Kustomization over config/installation)
    │   ├── messaging-topology-operator.yaml RabbitMQ Messaging Topology Operator (Flux Kustomization over config/installation)
    │   └── c5c3-operator.yaml            c5c3-operator ControlPlane orchestrator (from c5c3-charts)
    └── infrastructure/                   CRD-dependent infrastructure resources
        ├── kustomization.yaml            Infrastructure kustomization, also lists ../../eso
        ├── cluster-issuer.yaml           Self-signed ClusterIssuer (requires cert-manager CRDs)
        ├── db-ca-issuer.yaml             OpenStack DB CA Certificate + ClusterIssuer
        ├── ovn-ca-issuer.yaml            OVN CA Certificate + ClusterIssuer
        ├── mariadb.yaml                  MariaDB Galera cluster for OpenStack (with TLS)
        ├── memcached.yaml                Memcached cluster for OpenStack
        ├── garage.yaml                   Garage object store (GarageCluster, GarageBuckets, GarageKey)
        ├── openbao-ca-issuer.yaml        OpenBao CA Certificate + ClusterIssuer
        ├── openbao-tls-cert.yaml         OpenBao server Certificate
        └── openbao-client-tls-cert.yaml  OpenBao client Certificates

The proving OpenBaoCluster instance is not part of this tree. It is CI/dev-only and lives at deploy/kind/infrastructure/openbao-instance.yaml — see OpenBao Proving Instance.

All YAML files carry the SPDX Apache-2.0 license header (3 lines: copyright, blank comment, license identifier).

Namespaces

Twenty-one Namespace resources are defined in namespaces.yaml and included as the first entry in the base kustomization. Kustomize applies Namespace resources before other resource kinds, ensuring target namespaces exist before any namespaced resources are created.

Namespace	Purpose
cert-manager	cert-manager operator and its resources
mariadb-system	MariaDB Operator
external-secrets	External Secrets Operator
monitoring	Prometheus Operator CRDs (consumed by the optional kube-prometheus-stack of the kind overlay and of the Lab Prometheus stack, which declares no Namespace of its own)
memcached-system	Memcached Operator
garage-system	Garage Operator (S3 object store for the CI/e2e stack)
keystone-system	Keystone Operator controller (workload CRs continue to live in openstack)
horizon-system	Horizon Operator controller (Horizon CRs and the operator-managed dashboard live in openstack)
glance-system	Glance Operator controller (Glance/GlanceBackend CRs and the operator-managed payload live in openstack)
placement-system	Placement Operator controller (Placement CRs and the operator-managed payload live in openstack)
barbican-system	Barbican Operator controller (Barbican/BarbicanSecretStore CRs and the operator-managed payload live in openstack)
ovn-system	OVN Operator controller (OVNCentral/OVNChassis CRs and the operator-managed payload live in the tenant namespace they are created in)
neutron-system	Neutron Operator controller (Neutron/NeutronMetadataAgent CRs and the operator-managed payload live in the tenant namespace they are created in)
cinder-system	Cinder Operator controller (Cinder/CinderBackend/CinderBackupBackend CRs and the operator-managed payload live in the tenant namespace they are created in)
nova-system	Nova Operator controller (Nova CRs and the operator-managed payload live in the tenant namespace they are created in)
openstack	Infrastructure instance CRs that exist to run the operators standalone (MariaDB cluster, Memcached cluster; on kind also the OpenBao proving instance and the shared Gateway)
shared-services	Infrastructure consumed by more than one control plane: the OpenBao HA Raft cluster and the Garage object store
openbao-operator-system	openbao-operator controller. It stays out of shared-services so the shared OpenBao cluster and the operator that manages per-service instances keep separate lifecycles
c5c3-system	c5c3-operator controller; the ControlPlane and its child CRs are created in the ControlPlane's own namespace
orc-system	K-ORC (OpenStack Resource Controller) and its installer resources
rabbitmq-system	RabbitMQ Cluster Operator and RabbitMQ Messaging Topology Operator controllers and their webhook certificates. A RabbitmqCluster a ControlPlane declares, and the Vhost, User and Permission CRs of a RabbitMQVhost order, live in that ControlPlane's own namespace, not here

shared-services is a trust zone, not just a placement bucket. It holds the credentials that unlock every other secret in the stack — openbao-init-keys (the root token and all Shamir unseal key shares, in plaintext), openbao-tls, and eso-openbao-client-tls — and it is on the openbao-cluster-store allow-list. Read access to Secrets in this namespace, whether through RBAC or through an ExternalSecret created there, is therefore equivalent to read access to the whole store. Garage is the one accepted co-tenant and holds neither grant; do not add a workload or a Secret-reading Role here without treating it as a security review.

The chaos-mesh namespace is not part of the production base. It is created inline by the kind-only opt-in overlay at deploy/kind/chaos-mesh/ when WITH_CHAOS_MESH=true make deploy-infra is used, and by the lab overlay deploy/lab/metal-stack/chaos-mesh/, which takes that overlay as its base, under EXTERNAL_CLUSTER=true. See Chaos Mesh (kind-only opt-in) and Lab Chaos Mesh below.

Note: The install.createNamespace: true setting on HelmReleases instructs FluxCD's helm-controller to create namespaces when installing charts. However, this does not help when applying HelmRelease CRs via kubectl apply -k — the target namespace must already exist for the API server to accept namespaced resources. The explicit Namespace resources solve this chicken-and-egg problem.

Migrating a cluster deployed before the relocation

The OpenBao cluster moved out of openbao-system, and Garage out of openstack, into shared-services. Both keep their state in namespace-scoped StatefulSet PVCs, so the move brings up empty volumes and leaves the originals running: kubectl apply -k does not prune, and the FluxInstance declares no spec.sync, so Flux does not either. An abandoned OpenBao keeps serving every historical secret alongside its plaintext root token, and an abandoned Garage keeps holding the objects Glance's database still points at while Glance is repointed at empty buckets.

make deploy-infra refuses to run against such a cluster and names what to delete. On a cluster you do not intend to recreate, capture the old store first — openbao-init-keys holds the root token and every Shamir unseal-key share, and it has no second copy:

(umask 077 && kubectl get secret openbao-init-keys -n openbao-system -o yaml \
  > ~/openbao-init-keys.backup.yaml)

That file carries the root token and every unseal share in the clear, so it unseals the retired instance forever and expires never. Keep it where you keep a root credential — outside the working tree, so a routine git add -A cannot stage it — and delete it once the secrets it protects have been re-applied at the source.

Then delete both explicitly. The openstack namespace also holds the MariaDB volume carrying the Keystone database, so scope the PVC listing to Garage rather than deleting from an unfiltered one:

kubectl delete namespace openbao-system
kubectl delete garagecluster garage -n openstack
kubectl get pvc -n openstack | grep garage   # delete only these

Re-bootstrapping OpenBao afterwards re-seeds bootstrap/* from scratch, so any secret that was rotated in the old instance has to be re-applied at the source. To read those secrets out of the old instance before it goes away, run ALLOW_PRE_RELOCATION=true make deploy-infra instead: the guard downgrades to a warning and both stacks run side by side for a migration window. That defers the split-brain rather than resolving it — delete the retired stack once the new one serves.

FluxInstance

File: deploy/flux-system/fluxinstance.yaml

A single FluxInstance CR drives the flux-operator, which replaces the imperative flux install / flux bootstrap path with a declarative, operator-managed Flux lifecycle. The flux-operator reconciles the Flux controller Deployments from this spec and publishes a FluxReport/flux summarizing the installation state.

Property	Value
API version	fluxcd.controlplane.io/v1
Kind	FluxInstance
Name	flux
Namespace	flux-system

Spec fields:

Field	Value	Purpose
distribution.version	"2.x"	Minor-version track pinned by the operator; picks the latest Flux 2.x release
distribution.registry	ghcr.io/fluxcd	Controller image registry
components	source-controller, kustomize-controller, helm-controller, notification-controller	Four Flux controllers installed — image-automation and image-reflector controllers are omitted (not used in this project)
cluster.type	kubernetes	Generic Kubernetes distribution (not OpenShift/EKS-specific)
cluster.size	small	Small resource profile suitable for single-node kind and low-traffic management clusters
cluster.multitenant	false	Cross-namespace references allowed — simplifies the single-tenant management cluster model
cluster.networkPolicy	false	No NetworkPolicies applied to flux-system (kind overlay assumes a permissive default; production overlays opt in)

No spec.sync block. The kind Quick Start applies Helm sources and releases directly via kubectl apply -k deploy/kind/base/, so the FluxInstance here does not carry a GitRepository sync. Production overlays that want continuous reconciliation from Git add a spec.sync block on top of this base, with the overlay technique that Sizing and placement overrides uses.

Namespace. namespaces.yaml does not declare flux-system: the install.yaml of the flux-operator creates it. hack/deploy-infra.sh applies that file (pinned FLUX_OPERATOR_VERSION, kubectl apply -f install.yaml) before this kustomization, so the namespace exists when the FluxInstance is created.

HelmRepository Sources

Seven HelmRepository CRs define the Helm chart registries that FluxCD pulls from. All use apiVersion: source.toolkit.fluxcd.io/v1, are deployed to the flux-system namespace, and poll at interval: 1h.

File	metadata.name	Registry URL	Type
sources/cert-manager.yaml	cert-manager	https://charts.jetstack.io	HTTPS
sources/mariadb-operator.yaml	mariadb-operator	https://mariadb-operator.github.io/mariadb-operator	HTTPS
sources/external-secrets.yaml	external-secrets	https://charts.external-secrets.io	HTTPS
sources/openbao.yaml	openbao	https://openbao.github.io/openbao-helm	HTTPS
sources/c5c3-charts.yaml	c5c3-charts	oci://ghcr.io/c5c3/charts	OCI
sources/prometheus-community.yaml	prometheus-community	oci://ghcr.io/prometheus-community/charts	OCI
sources/garage-operator.yaml	garage-operator	oci://ghcr.io/rajsinghtech/charts	OCI

The openbao-operator chart is sourced from an OCIRepository, not a HelmRepository.sources/openbao-operator.yaml addresses the chart artifact directly (oci://ghcr.io/dc-tec/charts/openbao-operator) and pins spec.ref.digest next to spec.ref.tag. A HelmRepository can only carry a chart version, and on a mutable OCI tag that gates version drift alone: Flux resolves the tag on every interval and tracks the resulting artifact digest, so a re-pushed tag upgrades the release with no Git change and no reviewer. See OpenBao Operator for why that matters for this particular chart.

K-ORC is sourced from Git, not Helm. K-ORC publishes no Helm chart (its github.io page serves no Helm index), so sources/k-orc.yaml is a GitRepository — still source.toolkit.fluxcd.io/v1, in flux-system, polling at interval: 1h — pinned by ref.commit to an upstream main commit. No released K-ORC ships the RoleAssignment and Region kinds the c5c3-operator owns, so the pin returns to a release tag once one does. spec.sparseCheckout limits the checkout to config, and spec.ignore scopes the artifact to the same directory. The sparse checkout is what keeps the source working while upstream moves: for a ref.commit the source-controller clones the tip of main and hard-resets to the pin, and that reset fails when the tip carries a symlink to a non-empty directory that the pin lacks. A sparse checkout starts from an empty worktree, so paths outside config are never removed. tests/unit/deploy/korc_flux_source_test.sh checks that the list covers the path the Kustomization builds. It is applied by a Flux Kustomization, not a HelmRelease; see K-ORC (OpenStack Resource Controller).

The RabbitMQ Cluster Operator is sourced from Git too. Upstream publishes no chart, so sources/rabbitmq-cluster-operator.yaml is a second GitRepository in flux-system at interval: 1h, scoped to /config via spec.ignore. It pins both halves of the ref: ref.tag: "v2.23.0" as the readable version and as Renovate's lookup handle, and ref.commit as the content pin. Flux's gogit client evaluates commit ahead of every other ref field, so a re-pushed tag cannot hand different bytes to a controller that runs with cluster-wide RBAC. This source is applied by a Flux Kustomization as well; see RabbitMQ Cluster Operator.

The chaos-mesh HelmRepository ships in the kind-only opt-in overlay at deploy/kind/chaos-mesh/source.yaml — it is intentionally absent from deploy/flux-system/{sources,kustomization.yaml}. See Chaos Mesh (kind-only opt-in).

The c5c3-charts, prometheus-community, and garage-operator repositories are OCI-type sources (spec.type: oci). c5c3-charts hosts internally-built operator charts (e.g., memcached-operator) in the GitHub Container Registry; prometheus-community hosts Prometheus community charts (e.g., prometheus-operator-crds); garage-operator is the first third-party OCI-type source and hosts the upstream garage-operator chart. For every OCI-type HelmRepository the registry URL is the chart namespace and the chart name lives in the HelmRelease's chart.spec.chart — the OCIRepository above is the exception, because it names one artifact. All other repositories use standard HTTPS Helm registries.

HelmRelease Operators

Nineteen HelmRelease CRs deploy the CRD charts, the infrastructure operators and the CobaltCore operators (K-ORC and the RabbitMQ Cluster Operator are applied separately, each via a Flux Kustomization — see K-ORC (OpenStack Resource Controller) and RabbitMQ Cluster Operator). All use apiVersion: helm.toolkit.fluxcd.io/v2 and share these common settings:

Setting	Value	Purpose
spec.interval	30m	Reconciliation interval
spec.install.crds	CreateReplace	Install CRDs if missing, replace if outdated
spec.install.createNamespace	true	Auto-create target namespace
spec.upgrade.crds	CreateReplace	Update CRDs on chart upgrade
spec.upgrade.remediation.retries	3	Retry failed upgrades up to 3 times

Dependency Order

cert-manager, prometheus-operator-crds and mariadb-operator-crds declare no dependsOn. Every other release waits for cert-manager: c5c3-operator through keystone-operator, every other one by naming it. Most of them need a certificate for a webhook server, ovn-operator needs the certificates of the OVN databases, and openbao mounts Secrets that cert-manager issues. Some releases also wait for a CRD chart or for another operator.

The figure draws the releases in four layers. A solid arrow is a dependsOn entry, and the two dotted arrows are dependencies that no manifest can declare. The table lists every entry of every release, in the order of the layers.

Release	dependsOn
cert-manager	none
mariadb-operator-crds	none
prometheus-operator-crds	none
mariadb-operator	cert-manager, mariadb-operator-crds
memcached-operator	cert-manager, prometheus-operator-crds
external-secrets	cert-manager
garage-operator	cert-manager
openbao	cert-manager
openbao-operator	cert-manager
ovn-operator	cert-manager
keystone-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets
horizon-operator	cert-manager, memcached-operator, external-secrets, keystone-operator
glance-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator
placement-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator
cinder-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator
nova-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator
barbican-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator, openbao-operator
neutron-operator	cert-manager, mariadb-operator, memcached-operator, external-secrets, keystone-operator, ovn-operator
c5c3-operator	keystone-operator, external-secrets, mariadb-operator, memcached-operator

K-ORC is not in the table and has no solid arrow: it is applied by a Flux Kustomization, not a HelmRelease, and a HelmRelease dependsOn can only reference other HelmReleases. The c5c3-operator therefore does not dependsOn K-ORC even though K-ORC is a hard dependency: SetupWithManager Owns the K-ORC kinds, so the manager only starts once those CRDs are installed (until then the pod restarts), and converges once they appear. The crash-looping pod also fails the install Helm waits on, so c5c3-operator is the one release that widens spec.install.remediation.retries to 30: the default budget of 3 can be spent before the k-orc Kustomization lands, and a spent one stalls the release until the chart, the values or the spec change. At the 5m default wait, 30 attempts are ~2.5h of installing. It stays bounded on purpose — -1 would never set Stalled on a permanently broken install, and each remediation uninstalls the release, so an unbounded budget keeps reopening a window in which the templated webhook configurations are gone while the CRDs under crds/ survive. Its spec.upgrade.remediation.retries stays at 3, where remediation rolls back to the running operator.

The rabbitmq-cluster-operator Kustomization is outside the graph for the same reason, and it needs cert-manager: the upstream base carries the two admission webhooks, a self-signed Issuer, and the Certificates backing them. It declares no dependsOn, because a Flux Kustomization can only depend on other Kustomizations and cert-manager is a HelmRelease. On a cold cluster the Issuer/Certificate apply fails while cert-manager's CRDs are still missing, the Kustomization reports NotReady, and one of the following retryInterval passes (2m) succeeds once cert-manager is up. The NotReady window is that convergence. That is also why neutron-operator, cinder-operator and nova-operator carry no edge to it: the Neutron agents, the Cinder services and every Nova process talk over the shared message bus, but a HelmRelease cannot depend on a Kustomization.

The messaging-topology-operator Kustomization is outside the table as well. It is the one Flux Kustomization of the directory that declares a dependsOn: rabbitmq-cluster-operator, whose RabbitmqCluster CRD its controller watches. Its own cert-manager objects converge over retryInterval the same way.

The c5c3-operator HelmRelease is in the last layer: it dependsOn four of the operators whose CRs it projects (keystone-operator, external-secrets, mariadb-operator, memcached-operator). It also drives K-ORC's ApplicationCredential / Service / Endpoint CRDs, but K-ORC is applied by the separate Flux Kustomization above, so it cannot be a dependsOn edge — the c5c3-operator's manager instead requires those CRDs to be present at startup.

FluxCD resolves this dependency graph and installs operators in the correct order. If cert-manager is not ready, dependent operators are held in a pending state.

The kind-only chaos-mesh HelmRelease (deploy/kind/chaos-mesh/) also declares dependsOn: cert-manager but is only installed when WITH_CHAOS_MESH=true make deploy-infra is used. Production overlays do not install it. See Chaos Mesh (kind-only opt-in).

cert-manager

File: deploy/flux-system/releases/cert-manager.yaml

Property	Value
Target namespace	cert-manager
Chart	cert-manager
Version constraint	>=1.16.0 <2.0.0
Source	cert-manager HelmRepository
Dependencies	None (base layer)

Helm values:

Key	Value	Purpose
crds.enabled	true	Install CRDs via the Helm chart
prometheus.enabled	false	Prometheus metrics disabled
startupapicheck.enabled	true	Run the startup API check Job, so the release is Ready only once the webhook admits a request

The production release sets startupapicheck.enabled, and the kind and the metal-stack lab base renders inherit it. A dependent release can create an Issuer or a Certificate as soon as cert-manager reports Ready. Without the check the release turns Ready before the webhook admits, and a dependent release can use up its install retries and stay Stalled.

The chart renders the check as Job cert-manager-startupapicheck, a post-install hook. It runs on install, and helm-controller waits for it, so dependsOn: cert-manager holds the dependent releases until the webhook answers. An upgrade of an existing installation starts no Job.

A fresh install pulls quay.io/jetstack/cert-manager-startupapicheck at the chart's app version. It is a fourth image beside the controller, cainjector and webhook, from the same registry, and a cluster that mirrors or allowlists images has to provide it.

The Job runs check api --wait=1m and restarts up to backoffLimit: 4. The release sets no spec.timeout, so helm-controller's default release timeout of 5 minutes bounds the wait. When the check does not pass in that time, the install fails. The release has no install remediation, so it stays failed and not Ready, the terminal state the install comment in deploy/flux-system/releases/external-secrets.yaml describes. Every dependent release waits at dependsOn, so the release that reports the failure is cert-manager itself. tests/unit/deploy/cert_manager_release_test.sh pins the value in the three renders.

Prometheus Operator CRDs

File: deploy/flux-system/releases/prometheus-operator-crds.yaml

Property	Value
Target namespace	monitoring
Chart	prometheus-operator-crds
Version constraint	>=17.0.0 <20.0.0
Source	prometheus-community HelmRepository
Dependencies	None

The Prometheus Operator CRDs chart installs ServiceMonitor, PodMonitor, PrometheusRule, and related monitoring.coreos.com CRDs. These are required by the memcached-operator controller, which unconditionally watches ServiceMonitor resources via Owns().

MariaDB Operator CRDs

File: deploy/flux-system/releases/mariadb-operator-crds.yaml

Property	Value
Target namespace	mariadb-system
Chart	mariadb-operator-crds
Version constraint	>=0.30.0 <1.0.0
Source	mariadb-operator HelmRepository
Dependencies	None

A separate CRD chart is required since mariadb-operator v0.35.0. Must be installed before mariadb-operator so CRDs are available for the operator and for infrastructure CRs (e.g., MariaDB Galera cluster).

MariaDB Operator

File: deploy/flux-system/releases/mariadb-operator.yaml

Property	Value
Target namespace	mariadb-system
Chart	mariadb-operator
Version constraint	>=0.30.0 <1.0.0
Source	mariadb-operator HelmRepository
Dependencies	cert-manager in cert-manager namespace, mariadb-operator-crds in mariadb-system namespace

Helm values:

Key	Value	Purpose
metrics.enabled	false	Prometheus metrics disabled
webhook.enabled	true	Enable admission webhooks for MariaDB CRDs

External Secrets Operator

File: deploy/flux-system/releases/external-secrets.yaml

Property	Value
Target namespace	external-secrets
Chart	external-secrets
Version constraint	>=0.10.0 <3.0.0
Source	external-secrets HelmRepository
Dependencies	cert-manager in cert-manager namespace

Helm values:

Key	Value	Purpose
installCRDs	true	Install CRDs via the Helm chart
webhook.port	9443	Webhook server listen port
certController.enabled	true	Manage webhook TLS certificates

The production ESO kustomization renders one store, ClusterSecretStore/openbao-cluster-store (deploy/eso/). It serves the namespaces openstack and shared-services: standalone service CRs that set no spec.secretStoreRef, and the static ExternalSecrets of the kind overlay. A ControlPlane does not use it by default. The c5c3-operator provisions a namespaced SecretStore/openbao-tenant-store in every ControlPlane namespace (reconcileESOTenantStore) and routes the ControlPlane and its children through it. spec.secretStoreRef on the ControlPlane is the opt-out for a store you manage yourself, and deploy/openbao/bootstrap/setup-eso-tenant.sh provisions the same store by hand for a standalone namespace (see the OpenBao bootstrap reference and the multi-tenant deployment guide).

The figure shows both stores and the paths that run through them.

Memcached Operator

File: deploy/flux-system/releases/memcached-operator.yaml

Property	Value
Target namespace	memcached-system
Chart	memcached-operator
Version constraint	>=0.1.0 <1.0.0
Source	c5c3-charts HelmRepository (shared OCI registry)
Dependencies	cert-manager in cert-manager namespace, prometheus-operator-crds in monitoring namespace

Source reference: The Memcached Operator chart is published to the shared c5c3-charts OCI registry (oci://ghcr.io/c5c3/charts), not a dedicated HelmRepository. The sourceRef.name is c5c3-charts, matching the OCI HelmRepository in sources/.

Helm values:

Key	Value	Purpose
metrics.enabled	true	Expose Prometheus metrics
webhook.enabled	true	Enable admission webhooks for Memcached CRDs

Garage Operator

File: deploy/flux-system/releases/garage-operator.yaml

Property	Value
Target namespace	garage-system
Chart	garage-operator
Version constraint	>=0.6.26 <1.0.0
Source	garage-operator HelmRepository (third-party OCI registry)
Dependencies	cert-manager in cert-manager namespace

The garage-operator deploys Garage, a lightweight S3-compatible object store, as the in-cluster S3 backend for the CI/e2e stack (the Glance multi-store e2e suites need an S3 endpoint before any glance-operator lands). Its instance CRs are described under Garage Object Store below.

No Helm values are overridden. The Garage image rides the operator/chart releases (no custom GarageCluster.spec.image pin), and the operator's admission/conversion webhooks are on by default — hence the dependsOn: cert-manager edge.

Accepted risk (decided 2026-07-15): garage-operator is a young, single-maintainer, pre-1.0 (v0.6.x) project. It is accepted for test infrastructure only — never a production dependency of the operators — and the consuming surface is deliberately thin (four instance CRs plus three ExternalSecrets), so a later provider swap stays local to this layer.

OpenBao Operator

File: deploy/flux-system/releases/openbao-operator.yaml

Property	Value
Target namespace	openbao-operator-system
Chart	openbao-operator
Version constraint	0.4.2, pinned by artifact digest
Source	openbao-operator OCIRepository, referenced via spec.chartRef
Dependencies	cert-manager in cert-manager namespace

The openbao-operator manages OpenBaoCluster instances: static-seal auto-unseal, cert-manager TLS in External mode, declarative self-init, raft snapshots, and upgrade strategies. It delivers the per-service OpenBao instances a Barbican secret store builds on. The in-repo instance it manages is described under OpenBao Proving Instance. The shared management OpenBao (OpenBao below) stays on the upstream Helm chart and is untouched by this operator.

The chart ships ValidatingAdmissionPolicies and requires Kubernetes 1.33 or newer. The operator needs no certificate of its own from cert-manager, but the instance TLS Certificates do, and dependsOn: cert-manager keeps this release behind the same base layer as garage-operator.

Helm values:

Key	Value	Purpose
tenancy.mode	multi	Deploy the provisioner next to the controller, so the controller reconciles in every namespace an OpenBaoTenant admits
tenancy.namespacePodSecurityLabels.mode	external	Deny the provisioner every Namespace mutation, so it stamps no Pod Security labels

The c5c3 operator projects a dedicated OpenBaoCluster into the Barbican service namespace of every ControlPlane, and those namespaces are created at runtime, so no single watched namespace covers them. Multi-tenant mode covers them: the controller reconciles in every namespace an OpenBaoTenant has onboarded.

tenancy.mode only shapes the chart's own output. The controller reads its tenancy from the WATCH_NAMESPACE environment variable, and no chart template sets it, so leaving controller.extraEnv unset is what runs it multi-tenant. A value reintroduced there pins the controller to one namespace and strands every other one.

Under the chart default (enforce) the provisioner holds update and patch on namespaces cluster-wide and stamps restricted Pod Security labels onto every namespace it onboards, openstack included, which hosts every OpenStack service workload that restricted would start rejecting. external drops those verbs from the ClusterRole and widens the chart's ValidatingAdmissionPolicy to deny the provisioner any Namespace update. Tenant RBAC onboarding is unaffected.

Accepted gap. Nothing in the repository stamps the Pod Security labels instead, so an onboarded namespace admits pods at whatever the cluster default is. For a namespace holding a dedicated OpenBao instance, the seal key and the raft volume are then in reach of any pod that namespace admits.

Every namespace hosting an OpenBaoCluster needs an OpenBaoTenant. Without one the controller waits for the openbao-operator-tenant-rolebinding RoleBinding that tenant onboarding creates and pauses every reconcile in that namespace. It logs that pause at V(1), so the visible symptom is an OpenBaoCluster that keeps an empty status and never gets a StatefulSet. The c5c3 operator creates a tenant for each service namespace it projects into, and the kind overlay carries a static one for the proving instance (see OpenBao Proving Instance).

Accepted risk (decided 2026-08-05): openbao-operator is a young, single-maintainer, pre-1.0 (v0.4.x) project, and unlike garage-operator it sits in the production data path of every future Barbican secret store. Three things bound the exposure. The consumed surface is a single OpenBaoCluster CR shape. A Barbican secret store can also attach to the shared management OpenBao, where the bootstrap scripts provision the same mount, policy, and AppRole role without this operator taking part (see the OpenBao bootstrap reference). And an in-repo StatefulSet projection derived from the openbao Helm chart stays available as a fallback. Accepted for the Barbican onboarding.

Why the chart is pinned by digest. The chart carries no cosign signature Flux can verify, so no spec.verify policy can gate it, and the operator holds cluster-wide RBAC over the namespace that stores the instance seal key. Two weaker pins were rejected. The >=x <1.0.0 range garage-operator uses would install every future 0.x release of an individual-owned GHCR namespace within the 30-minute reconcile interval. An exact chart version on a HelmRepository is not enough either: 0.4.2 is a mutable OCI tag, Flux tracks the resolved artifact digest rather than the tag string, and re-pushing that tag with different bytes would upgrade the release on its own — no Git change, no reviewer. Pinning spec.ref.digest on the OCIRepository is what makes every change to what runs here a reviewed commit.

Renovate keeps that pin from becoming a freeze. Its native Flux manager reads deploy/flux-system/, but a packageRule switches it off for sources/openbao-operator.yaml: the pin is tracked by an explicit customManagers entry in renovate.json that rewrites the tag and the digest in the same pull request. The entry never automerges: every bump of this operator is a human decision.

OpenBao

File: deploy/flux-system/releases/openbao.yaml

Property	Value
Target namespace	shared-services
Chart	openbao
Version constraint	>=0.5.0 <1.0.0
Source	openbao HelmRepository
Dependencies	cert-manager in cert-manager namespace

OpenBao is deployed as a 3-replica HA Raft cluster with mutual TLS (mTLS) enforced on the API listener. The injector is disabled. The server TLS certificate is sourced from a cert-manager-provisioned Secret (openbao-tls), and two additional cert-manager-provisioned client certificates (openbao-client-tls, eso-openbao-client-tls) are required so that the OpenBao pods themselves (Raft retry_join + in-pod bao exec wrappers) and the External Secrets Operator can complete the TLS handshake. The listener carries tls_client_ca_file = "/openbao/tls/ca.crt" and tls_require_and_verify_client_cert = true, so every connection on :8200 — whether from a Raft peer, the in-pod bootstrap script, or ClusterSecretStore/openbao-cluster-store — must present a client certificate that chains to the same self-signed CA bundle as the server cert; the Kubernetes-token auth method (auth.kubernetes) is unchanged and runs after the transport-layer admission gate.

Helm values:

Key	Value	Purpose
global.tlsDisable	false	Enable TLS globally
server.authDelegator.enabled	true	Enable ClusterRoleBinding for TokenReview API (ESO auth)
server.ha.enabled	true	Enable HA mode
server.ha.replicas	3	3-node Raft cluster
server.ha.raft.enabled	true	Use Raft storage backend
server.ha.raft.config listener tls_client_ca_file	/openbao/tls/ca.crt	CA the listener uses to verify presented client certs (same bundle as server cert)
server.ha.raft.config listener tls_require_and_verify_client_cert	true	Reject any TLS handshake without a valid client cert before app-layer auth runs
server.ha.raft.config retry_join.leader_client_cert_file × 3	/openbao/client-tls/tls.crt	Client cert each Raft peer presents on retry_join to every other peer (same value in all three stanzas)
server.ha.raft.config retry_join.leader_client_key_file × 3	/openbao/client-tls/tls.key	Matching private key for leader_client_cert_file
server.volumes / server.volumeMounts — client-tls	Secret openbao-client-tls → /openbao/client-tls (readOnly: true)	Mounts the in-pod client keypair distinct from the server cert at /openbao/tls so server and client lifecycles do not collide
server.dataStorage.size	10Gi	Persistent volume size
injector.enabled	false	Disable the Vault/Bao agent injector

Client certificates. The two client Certificate resources are declared in deploy/flux-system/infrastructure/openbao-client-tls-cert.yaml and registered in deploy/flux-system/infrastructure/kustomization.yaml immediately after openbao-tls-cert.yaml, so cert-manager reconciles them before the OpenBao StatefulSet and ClusterSecretStore consume them (first-apply ordering, see also "Apply ordering" notes below):

Certificate	Secret (namespace)	Consumer	Reference
openbao-client-tls	openbao-client-tls (shared-services)	OpenBao pods — Raft retry_join + in-pod bao exec	StatefulSet volume client-tls mounted at /openbao/client-tls; env vars VAULT_CLIENT_CERT / VAULT_CLIENT_KEY in every exec wrapper (deploy/openbao/bootstrap/common.sh, init-unseal.sh, hack/deploy-infra.sh)
eso-openbao-client-tls	eso-openbao-client-tls (shared-services)	ESO ClusterSecretStore/openbao-cluster-store	spec.provider.vault.tls.certSecretRef / keySecretRef in deploy/eso/clustersecretstore.yaml; auth.kubernetes block (mountPath kubernetes/management, role eso-management) is unchanged — mTLS is purely transport-layer

Both client certs are issued from the same openbao-ca-issuer as openbao-tls (a CA-type ClusterIssuer defined in deploy/flux-system/infrastructure/openbao-ca-issuer.yaml and itself bootstrapped by selfsigned-cluster-issuer). Sharing one CA is what makes the listener's tls_client_ca_file = /openbao/tls/ca.crt validate every presented client cert — a SelfSigned issuer would mint each Certificate as its own root and leave the chains unrelated. Both client certs carry usages: ["client auth"], with the same duration / renewBefore as openbao-tls so server and client rotation cadences stay aligned. See OpenBao Bootstrap Procedure — TLS Configuration for the full SAN/usages table, the VAULT_CLIENT_CERT / VAULT_CLIENT_KEY operator interface, and the runnable mTLS-enforcement probe.

Keystone Operator

File: deploy/flux-system/releases/keystone-operator.yaml

Property	Value
Target namespace	keystone-system (controller); operator-managed Keystone workload remains in openstack
Chart	keystone-operator
Version constraint	>=0.8.0 <1.0.0 (floor = first chart whose values schema accepts image.digest)
Source	c5c3-charts HelmRepository (shared OCI registry)
Dependencies	cert-manager, mariadb-operator, memcached-operator, external-secrets

The Keystone Operator manages OpenStack Keystone identity service instances. It depends on four upstream operators: cert-manager for TLS, mariadb-operator for database provisioning, memcached-operator for caching, and external-secrets for secret management.

Helm values:

Key	Value	Purpose
replicas	2	Run 2 controller replicas for HA
leaderElection.enabled	true	Enable leader election for HA
image.tag	latest	Use latest image until a versioned release publishes a semver tag
image.digest	(injected)	Optional immutable digest merged in via the valuesFrom ConfigMap; absent by default

The release carries an optional valuesFrom reference to a keystone-operator-image-digest ConfigMap (key values.yaml) in keystone-system. hack/refresh-operator-image-digests.sh (re)applies that ConfigMap at deploy time on the WITH_CONTROLPLANE=true flux path, pinning the mutable latest tag to the digest current at deploy so a freshly merged operator image actually rolls out. When the ConfigMap is absent — the default Quick Start and CI paths — the release renders tag-only, exactly as before. Flux merges valuesFrom first and spec.values on top per-key, so image.tag and image.digest coexist.

K-ORC (OpenStack Resource Controller)

File: deploy/flux-system/releases/k-orc.yaml

Property	Value
Kind	Kustomization (kustomize.toolkit.fluxcd.io/v1)
Target namespace	orc-system (the upstream installer self-namespaces)
Source	k-orc GitRepository (commit da276e44b40ca218ed9bb7b62114f6a461a29689 on main)
Path	./config/default
Image	quay.io/orc/openstack-resource-controller:commit-da276e4, pulled by digest
Dependencies	None

K-ORC (the OpenStack Resource Controller) installs the declarative Keystone resource CRDs — ApplicationCredential, Service, Endpoint, and related kinds — that the c5c3-operator drives to project a ControlPlane's desired state into Keystone.

K-ORC ships no Helm chart, so it is applied as a Flux Kustomization rather than a HelmRelease. Upstream flattens its installer into dist/install.yaml only at release time, and no release ships the RoleAssignment and Region kinds the c5c3-operator owns. The GitRepository source is therefore pinned to a main commit and vendors ./config, and the Kustomization builds ./config/default (prune: true, wait: true). That base references the placeholder image controller:latest, so spec.images rewrites it to the per-commit image upstream publishes. The digest is what resolves the pull, and the commit-<short sha> tag next to it is the drift anchor hack/ci-deploy-korc.sh checks against the source commit. A commit bump moves the source commit, the image tag and digest, and the Go pseudo-version in operators/c5c3/go.mod together. The base declares the orc-system Namespace and namespaces every resource into it, so no spec.targetNamespace is set. The short, stable name k-orc (not the upstream openstack-resource-controller) keeps diagnostics and cross-references terse.

K-ORC at the pinned commit evicts its cached client the first time a service client cannot be built for a missing endpoint, so a service registered after its first login is picked up on the retry (k-orc/openstack-resource-controller#967). K-ORC evicts its cached client only for a missing endpoint. After the URL of an existing endpoint changes, K-ORC keeps calling the old URL until its cached client expires, half the token lifetime after its login. Run kubectl rollout restart deployment/orc-controller-manager -n orc-system to pick up the new URL at once.

c5c3-operator chart 0.18.0 relies on the missing-endpoint eviction and no longer restarts K-ORC. Upgrade K-ORC to commit da276e44 or later before or together with the chart, by applying deploy/flux-system/ of the same checkout again (see Deployment). The c5c3-operator HelmRelease floats the chart version and the image tag, while the K-ORC pin moves only when that directory is applied. With an older K-ORC, a service registered after K-ORC's first login stays missing from its catalog for up to half a token lifetime, and the ControlPlane gets no KORCRestartSkipped event for it. Restart K-ORC by hand with the same kubectl rollout restart command.

c5c3-operator chart versions 0.15.0 to 0.17.0 wrote one pod-template annotation per ControlPlane, c5c3.io/korc-catalog-epoch-<hash>, onto Deployment orc-controller-manager to restart K-ORC. Chart 0.18.0 neither writes nor removes it, because the operator no longer holds a verb on the Deployment. A cluster upgraded from those versions keeps the key until it is removed by hand, which rolls the K-ORC pod once:

kubectl get deployment orc-controller-manager -n orc-system \
  -o jsonpath='{.spec.template.metadata.annotations}'
kubectl patch deployment orc-controller-manager -n orc-system --type=json \
  -p '[{"op":"remove","path":"/spec/template/metadata/annotations/c5c3.io~1korc-catalog-epoch-<hash>"}]'

The upstream installer has no global-cloud-config knob (the previous HelmRelease set globalCloudConfig.secretName). That is not on the credential critical path: K-ORC authenticates per resource via each CR's CloudCredentialsRef, resolved in the CR's own (control-plane) namespace, so the credential chain below materialises a co-located k-orc-clouds-yaml copy there via the per-ControlPlane ExternalSecret the c5c3-operator creates and owns (reconcileKORC). K-ORC therefore needs no global default clouds.yaml mount, so there is no longer an orc-system copy — the static manifest that previously declared it has been removed. The orc-system Namespace itself remains because the K-ORC installer's own resources land there. See Admin Credential Chain below.

Watch scope. The one K-ORC instance in orc-system watches every namespace. The upstream manager takes a repeatable --namespace flag that would restrict it to the named namespaces; neither the vendored base nor this repo sets it, and hack/ci-deploy-korc.sh fails the deploy if the manager args ever carry one. The c5c3-operator places every K-ORC child (ApplicationCredential, Service, Endpoint, Region, RoleAssignment, Project, User, and the imports) and the k-orc-clouds-yaml Secret in the ControlPlane's own namespace, openstack in the lab, and nothing in orc-system. The same instance therefore serves every ControlPlane on the cluster: each CR authenticates through the Secret its cloudCredentialsRef names in its own namespace, so two ControlPlanes in two namespaces never share credentials. The multi-controlplane suite under tests/e2e/c5c3/ is written to assert that against a live stack, but no CI leg exercises it yet: the c5c3 e2e-operator leg carries no keystone-operator, so the suite skips there, and the e2e-controlplane leg runs a fixed list of suites without it. K-ORC caches one authenticated client per distinct cloud configuration, --scope-cache-max-size entries (upstream default 10), and a ControlPlane uses about three (the admin password cloud, the admin application credential, and a short-lived mint cloud per order). Past the cap the oldest client is evicted and re-authenticated on its next use, which costs a Keystone login, not correctness; a cluster hosting several ControlPlanes can raise the cap with a spec.patches entry on the k-orc Kustomization.

RabbitMQ Cluster Operator

File: deploy/flux-system/releases/rabbitmq-cluster-operator.yaml

Property	Value
Kind	Kustomization (kustomize.toolkit.fluxcd.io/v1)
Target namespace	rabbitmq-system (the upstream base self-namespaces)
Source	rabbitmq-cluster-operator GitRepository (tag v2.23.0, commit 3ad2b0ec0b43dbeb4b57c1142fc5c813a5716033)
Path	./config/installation
Image	ghcr.io/rabbitmq/cluster-operator, newTag: "2.22.5", digest: "sha256:2727b84b835ada97247bbb65ebfa6998168b4e8ee11b0e6cece56ac2c9c4f0fb"
Dependencies	None declared; cert-manager is an implicit one

The RabbitMQ Cluster Operator serves the rabbitmq.com/v1beta1 RabbitmqCluster CRD. The c5c3-operator projects one such CR for every ControlPlane that declares managed spec.infrastructure.messaging, and this operator turns it into the message bus that control plane's services share.

The base at ./config/installation installs the rabbitmq-system Namespace, the rabbitmqclusters.rabbitmq.com CRD (served version v1beta1), the operator's RBAC, the manager Deployment (the installer requests and limits it at 200m / 500Mi; the Kustomization's spec.patches lowers the requests to 100m / 128Mi and keeps the limits, see the Messaging Topology Operator for why), the two admission webhooks it has shipped since v2.22.0, and the self-signed Issuer plus the Certificates that back them. cert-manager has been required since v2.20.

Why a Kustomization and no chart. Upstream publishes none; the install path it documents is the kustomize base carried in its own repository. The only OCI chart that ever wrapped this operator was Bitnami's oci://registry-1.docker.io/bitnamicharts/rabbitmq-cluster-operator, and it is retired: it stopped at chart 4.4.34 on 2025-08-21 (operator 2.16.1), and the docker.io/bitnami/rabbitmq-cluster-operator images it points at are gone from Docker Hub.

Accepted posture. Upstream is maintained by VMware/Broadcom under the Mozilla Public License 2.0 and publishes no cosign signature, so spec.verify would have nothing to check. Both halves are pinned by content instead: the Git commit in the source, the image digest here. The inner kustomization already rewrites the dev placeholder rabbitmqoperator/cluster-operator-dev to ghcr.io/rabbitmq/cluster-operator with the mutable tag latest, so the images override keys on that resolved name and is what keeps latest out of the cluster. newTag records the tag the digest was resolved from (crane digest ghcr.io/rabbitmq/cluster-operator:<tag>) and doubles as the offline drift anchor, the same split releases/k-orc.yaml uses.

Keying on a name is the weak point: kustomize's image transformer silently no-ops when no resource matches, so if upstream ever re-points the base at a different image (it already moved once, from the dev placeholder), the override would apply to nothing, no error would be raised, and the cluster would run the mutable latest tag for a controller holding cluster-wide RBAC over RabbitmqClusters, StatefulSets, and Secrets. Neither the Renovate tests (which guard the literal YAML pairing) nor the production-posture test (which guards byte-identity of this file) render the kustomize output, so the check has to look at what actually landed: after waiting for the Kustomization, both hack/deploy-infra.sh and hack/deploy-mgmt-cluster.sh read every container image of every Deployment in rabbitmq-system and abort the deploy unless each one carries an @sha256: digest.

Renovate. Two customManagers track the pair, one on the source (github-tags, capturing tag and commit) and one on the image (docker, capturing newTag and digest). Their two packageRules share groupName: "rabbitmq cluster-operator", so all four values move in one reviewed PR, with automerge: false and minimumReleaseAge: "3 days". Both managers anchor on the literal YAML shape (the paired lines adjacent, both values double-quoted), which tests/unit/renovate/rabbitmq_cluster_operator_source_custommanager_test.sh and tests/unit/renovate/rabbitmq_cluster_operator_image_custommanager_test.sh guard.

RabbitMQ Messaging Topology Operator

File: deploy/flux-system/releases/messaging-topology-operator.yaml

Property	Value
Kind	Kustomization (kustomize.toolkit.fluxcd.io/v1)
Target namespace	rabbitmq-system (the upstream base self-namespaces)
Source	messaging-topology-operator GitRepository (tag v1.20.3, commit 2d780409f494944b3a2786c6687cebddef5bfb33)
Path	./config/installation
Image	ghcr.io/rabbitmq/messaging-topology-operator, newTag: "1.20.3", digest: "sha256:f377d3c3e2215a8dfb8a7a5ab953ef7f180f295c401e5d2c2060f3b0748fefd6"
Dependencies	rabbitmq-cluster-operator; cert-manager is an implicit one

The RabbitMQ Messaging Topology Operator serves the rabbitmq.com/v1beta1 topology CRDs. The c5c3-operator writes three of them for every RabbitMQVhost order: a Vhost, and per credential generation a User and a Permission, all in the ControlPlane's namespace beside the RabbitmqCluster they name. This operator creates them on the broker through its management API, with the admin credential the RabbitmqCluster reports in its status.binding Secret.

The base at ./config/installation installs the rabbitmq-system Namespace, the thirteen topology CRDs, the operator's RBAC, the manager Deployment (the installer requests 300m / 128Mi and limits 500m / 512Mi; the Kustomization's spec.patches lowers the requests to 50m / 64Mi and keeps the limits), its validating and mutating admission webhooks, and the self-signed Issuer plus the two Certificates that back the webhook and the metrics endpoint. Every name carries the base's namePrefixmessaging-topology-, so it shares rabbitmq-system with the Cluster Operator without a collision.

Both RabbitMQ operators run below their installers' requests because the upstream figures pushed the full stack over the 16Gi node budget that hack/ci-check-node-budget.sh enforces on the e2e-controlplane job (16498Mi against 16384Mi with both at upstream sizing). Each patch adds one member under requests at a time, so the installer's limits stay in place.

The Kustomization dependsOn rabbitmq-cluster-operator: the controller watches the RabbitmqCluster kind, so its CRD has to be served first. cert-manager is the same implicit dependency the Cluster Operator has, for the same reason.

Accepted posture. Upstream is the Cluster Operator's (VMware/Broadcom, Mozilla Public License 2.0) and publishes no cosign signature either. Both halves are pinned by content: the Git commit in the source, the image digest here. The inner kustomization rewrites the placeholder rabbitmqoperator/messaging-topology-operator to ghcr.io/rabbitmq/messaging-topology-operator with the mutable tag latest, so the images override keys on that resolved name. newTag records the tag the digest was resolved from (crane digest ghcr.io/rabbitmq/messaging-topology-operator:<tag>). The name-keyed override has the weak point the Cluster Operator section describes, and the same check covers it: both deploy scripts read every image of every Deployment in rabbitmq-system after the two Kustomizations are Ready and abort unless each one carries an @sha256: digest.

Renovate. Two customManagers track the pair the way they track the Cluster Operator's, one on the source (github-tags) and one on the image (docker). Their two packageRules share groupName: "rabbitmq messaging-topology-operator", with automerge: false and minimumReleaseAge: "3 days", so the four values move in one reviewed PR. tests/unit/renovate/messaging_topology_operator_source_custommanager_test.sh and tests/unit/renovate/messaging_topology_operator_image_custommanager_test.sh guard the YAML shape the managers anchor on.

c5c3-operator

File: deploy/flux-system/releases/c5c3-operator.yaml

Property	Value
Target namespace	c5c3-system
Chart	c5c3-operator
Version constraint	>=0.7.0 <1.0.0 (floor = first chart whose values schema accepts image.digest)
Source	c5c3-charts HelmRepository (shared OCI registry)
Dependencies	keystone-operator, external-secrets, mariadb-operator, memcached-operator

The c5c3-operator runs the ControlPlane reconciler that orchestrates a Keystone control plane end-to-end. It depends on four of the operators whose CRs it projects — keystone-operator for the Keystone instance, external-secrets and mariadb-operator and memcached-operator for the supporting platform services. It also drives K-ORC's ApplicationCredential / Service / Endpoint CRDs to register the catalog and rotate the admin credential, but K-ORC is the separate Flux Kustomization above, not a dependsOn edge (so K-ORC is a hard dependency: the manager does not start without its CRDs, as without those of MariaDB, Memcached, cert-manager and ESO, while the kinds of the service operators and RabbitmqCluster are optional watches). The operator child CRs are created in the ControlPlane's own namespace, not a hard-coded one. For the reconciliation contract see the ControlPlane reconciler reference.

Helm values:

Key	Value	Purpose
replicas	2	Run 2 controller replicas for HA
leaderElection.enabled	true	Enable leader election for HA
image.tag	latest	Use latest image until a versioned release publishes a semver tag
image.digest	(injected)	Optional immutable digest merged in via the valuesFrom ConfigMap; absent by default

The release consumes the same image-digest valuesFrom mechanism as the keystone-operator release above (a c5c3-operator-image-digest ConfigMap in c5c3-system); the horizon-operator release is wired identically.

HelmRelease–HelmRepository Cross-Reference

Each HelmRelease sourceRef.name must match a HelmRepository metadata.name in sources/. This table shows the mapping. The openbao-operator release is the one exception: it names its source through spec.chartRef instead, because that source is an OCIRepository.

HelmRelease	sourceRef.name	HelmRepository file
cert-manager	cert-manager	sources/cert-manager.yaml
prometheus-operator-crds	prometheus-community	sources/prometheus-community.yaml
mariadb-operator-crds	mariadb-operator	sources/mariadb-operator.yaml
mariadb-operator	mariadb-operator	sources/mariadb-operator.yaml
external-secrets	external-secrets	sources/external-secrets.yaml
memcached-operator	c5c3-charts	sources/c5c3-charts.yaml
openbao	openbao	sources/openbao.yaml
garage-operator	garage-operator	sources/garage-operator.yaml
openbao-operator	openbao-operator (chartRef)	sources/openbao-operator.yaml
keystone-operator	c5c3-charts	sources/c5c3-charts.yaml
horizon-operator	c5c3-charts	sources/c5c3-charts.yaml
glance-operator	c5c3-charts	sources/c5c3-charts.yaml
placement-operator	c5c3-charts	sources/c5c3-charts.yaml
barbican-operator	c5c3-charts	sources/c5c3-charts.yaml
ovn-operator	c5c3-charts	sources/c5c3-charts.yaml
neutron-operator	c5c3-charts	sources/c5c3-charts.yaml
cinder-operator	c5c3-charts	sources/c5c3-charts.yaml
nova-operator	c5c3-charts	sources/c5c3-charts.yaml
c5c3-operator	c5c3-charts	sources/c5c3-charts.yaml

k-orc and rabbitmq-cluster-operator are not in this table: each is a Flux Kustomization whose sourceRef is a GitRepository (sources/k-orc.yaml and sources/rabbitmq-cluster-operator.yaml), not a HelmRelease backed by a HelmRepository.

The kind-only chaos-mesh HelmRelease ships in the opt-in overlay at deploy/kind/chaos-mesh/release.yaml, with its own local source.yaml. It is intentionally absent from this always-on table because production overlays do not install it.

Infrastructure Custom Resources

Infrastructure CRs are instance-level resources managed by the operators installed via HelmReleases above. They are separated into their own kustomization (infrastructure/kustomization.yaml) because they depend on CRDs that are only available after the corresponding operator HelmReleases install their Helm charts.

Self-Signed ClusterIssuer

File: deploy/flux-system/infrastructure/cluster-issuer.yaml

Property	Value
API version	cert-manager.io/v1
Kind	ClusterIssuer
Name	selfsigned-cluster-issuer
Scope	Cluster-scoped (no namespace)

The self-signed ClusterIssuer signs the CA certificates of the trust domains: openstack-db-ca, openstack-ovn-ca and openbao-ca, and libvirt-migration-ca on the metal-stack lab. Each CA signs the leaves of its own domain through a CA issuer. The kind overlay also uses the self-signed issuer directly for the Gateway listener certificates. It requires cert-manager CRDs (cert-manager.io/v1) which are installed by the cert-manager HelmRelease.

The figure shows the four CAs with their leaves. The CAs are separate because each server accepts every client certificate of its own CA.

OpenStack DB CA Issuer

File: deploy/flux-system/infrastructure/db-ca-issuer.yaml

Provisions the dedicated cert-manager CA that anchors the OpenStack database trust domain. The file declares two resources:

Resource	API version	Kind	Name	Namespace
CA keypair Certificate	cert-manager.io/v1	Certificate	openstack-db-ca	cert-manager
CA ClusterIssuer	cert-manager.io/v1	ClusterIssuer	openstack-db-ca-issuer	Cluster-scoped

The selfsigned-cluster-issuer mints a self-signed CA Certificate (isCA: true, 3-year lifetime, 30-day renewBefore) into the openstack-db-ca Secret in the cert-manager namespace — cert-manager's default --cluster-resource-namespace, which is where a CA-type ClusterIssuer looks up its secretName. The openstack-db-ca-issuer ClusterIssuer then signs every leaf certificate inside the OpenStack DB trust domain:

• MariaDB Galera server TLS material (spec.tls.serverCertIssuerRef, see below).
• MaxScale listener TLS material (same issuer via inheritance / explicit serverCertIssuerRef).
• The Keystone DB-client keypair issued by the keystone-operator's reconcileDatabaseTLS sub-reconciler — the constant dbCAIssuerName hard-codes the same string ("openstack-db-ca-issuer"), so a rename here MUST be matched in the operator.

Apply ordering. This manifest is also applied out-of-band from the infrastructure kustomization by hack/deploy-infra.sh (Phase 2 of its Step 4, alongside cluster-issuer.yaml, openbao-ca-issuer.yaml and openbao-tls-cert.yaml) so that MariaDB has the issuer available the moment it tries to render its server certificate. The infrastructure kustomization still references db-ca-issuer.yaml so subsequent kubectl apply -k runs are idempotent.

For the end-to-end TLS path the issuer participates in, see the Enable Keystone Database TLS how-to.

OVN CA Issuer

File: deploy/flux-system/infrastructure/ovn-ca-issuer.yaml

Provisions the cert-manager CA that anchors the OVN database trust domain. The file declares two resources:

Resource	API version	Kind	Name	Namespace
CA keypair Certificate	cert-manager.io/v1	Certificate	openstack-ovn-ca	cert-manager
CA ClusterIssuer	cert-manager.io/v1	ClusterIssuer	openstack-ovn-ca-issuer	Cluster-scoped

The bootstrap path matches the DB CA above. selfsigned-cluster-issuer mints a self-signed CA Certificate (isCA: true, ECDSA P-256, 3-year lifetime, 30-day renewBefore) into the openstack-ovn-ca Secret in the cert-manager namespace, and openstack-ovn-ca-issuer signs the leaves from it: the Northbound and Southbound ovsdb-server certificates, plus one shared client certificate, {central}-client, which ovn-northd, the backup Job and Neutron present and of which every chassis holds a copy ({chassis}-ovn-client), and the relay's own certificate, {central}-sb-relay, when spec.relay is set.

OVN keeps a CA of its own instead of sharing openstack-db-ca-issuer. OVS/OVN authenticates a peer with a single check: the peer certificate must chain to the configured CA. Every leaf this issuer signs is therefore a valid OVSDB client, so it signs OVN database and client certificates and nothing else.

The OVNCentral CR names the ClusterIssuer in spec.tls.issuerRef. That field defaults to kind: ClusterIssuer because the chassis agents run outside the OVNCentral's namespace, which a namespaced Issuer cannot reach. An issuer that exposes no ca.crt is rejected by the ovn-operator's TLS sub-reconciler.

MariaDB Galera Cluster

File: deploy/flux-system/infrastructure/mariadb.yaml

Property	Value
API version	k8s.mariadb.com/v1alpha1
Kind	MariaDB
Name	openstack-db
Namespace	openstack
Replicas	3
Galera	Enabled (spec.galera.enabled: true)
MaxScale	Enabled, 2 replicas (spec.maxScale.enabled: true, spec.maxScale.replicas: 2)
Storage	100Gi, storage class ceph-rbd

The MariaDB CR provisions a 3-node Galera cluster with synchronous replication managed by the mariadb-operator. MaxScale is enabled with 2 replicas to provide intelligent query routing and read/write splitting across the Galera nodes.

The root password is sourced from a Kubernetes Secret (mariadb-root-password, key password). The production stack ships no ExternalSecret for it — a non-kind Flux MariaDB baseline is expected to provide the mariadb-root-password Secret itself. On kind, a kind-only overlay shim (deploy/kind/infrastructure/mariadb-root-password-externalsecret.yaml) materialises it from the OpenBao path infrastructure/mariadb so the single-node Quick Start stays self-contained.

Non-Goal — operator-owned root credential. Unlike the Keystone admin password (which the c5c3-operator now projects per ControlPlane as a dedicated ExternalSecret), the MariaDB root password is deliberately not operator-owned. Provisioning it is left to the MariaDB baseline — the kind shim above, or a production Flux baseline — keeping the operator off the database superuser credential path.

Services:

Service	Type	Purpose
Primary	ClusterIP	Read-write endpoint for application connections
Secondary	ClusterIP	Read-only endpoint for read replicas

Monitoring: Prometheus metrics are enabled (spec.metrics.enabled: true).

TLS. Galera inter-node replication, the MaxScale client listener, and every Keystone-to-database connection all sit inside the OpenStack DB trust domain rooted at the openstack-db-ca-issuer ClusterIssuer documented above. The MariaDB CR enables TLS in spec.tls and the MaxScale sub-spec inherits it:

Field	Value	Purpose
spec.tls.enabled	true	Turn on TLS for the MariaDB cluster
spec.tls.required	true	Reject any non-TLS connection at the transport layer (verified by the chainsaw plaintext-rejection probe in tests/e2e/keystone/database-tls/chainsaw-test.yaml)
spec.tls.serverCertIssuerRef	openstack-db-ca-issuer (ClusterIssuer, cert-manager.io)	Issue server certs for Galera + MaxScale from the shared DB CA
spec.tls.clientCertIssuerRef	openstack-db-ca-issuer (ClusterIssuer, cert-manager.io)	Trust client certs minted by the same DB CA (the Keystone operator issues its DB-client keypair from this issuer; see reconcile_databasetls.go)
spec.maxScale.tls.enabled	true	MaxScale terminates TLS on its client listener (proxy-side); explicit block documents intent even where the proxy would otherwise inherit spec.tls

The rendered YAML in deploy/flux-system/infrastructure/mariadb.yaml is:

spec:
  tls:
    enabled: true
    required: true
    serverCertIssuerRef:
      name: openstack-db-ca-issuer
      kind: ClusterIssuer
      group: cert-manager.io
    clientCertIssuerRef:
      name: openstack-db-ca-issuer
      kind: ClusterIssuer
      group: cert-manager.io
  maxScale:
    enabled: true
    replicas: 2
    tls:
      enabled: true

The mariadb-operator (v0.30+) auto-derives the server and client CA bundles from the referenced issuer, so explicit serverCASecretRef / clientCASecretRef entries are intentionally omitted — see the inline DECISION comment in mariadb.yaml for the trade-off against the cross-namespace *CASecretRef form. End-to-end verification that the live connection is encrypted lives in tests/e2e/keystone/database-tls/chainsaw-test.yaml (asserts SHOW STATUS LIKE 'Ssl_cipher' reports a non-empty cipher).

To turn the path on for a Keystone CR, follow the Enable Keystone Database TLS guide.

Memcached Cluster

File: deploy/flux-system/infrastructure/memcached.yaml

Property	Value
API version	memcached.c5c3.io/v1beta1
Kind	Memcached
Name	openstack-memcached
Namespace	openstack
Replicas	3
Image	memcached:1.6

The Memcached CR provisions a 3-replica Memcached cluster for OpenStack session and token caching. The memcached-operator manages pod lifecycle and provides stable DNS-based service discovery for operator consumers.

API group: The API group is memcached.c5c3.io, matching the CRD definition shipped by the memcached-operator Helm chart.

Garage Object Store

File: deploy/flux-system/infrastructure/garage.yaml

Four instance CRs (API group garage.rajsingh.info) declare the S3 object store the garage-operator manages. All live in the shared-services namespace:

Kind	API version	Name	Purpose
GarageCluster	garage.rajsingh.info/v1beta2	garage	Storage tier StatefulSet, config, and cluster layout — declarative, no manual garage layout assign/apply
GarageBucket	garage.rajsingh.info/v1beta1	glance-images	Pre-created bucket with an explicit globalAlias (deterministic S3 name)
GarageBucket	garage.rajsingh.info/v1beta1	glance-images-2	Second, independent bucket for the Glance S3 multi-store e2e suite (same explicit-globalAlias rationale)
GarageKey	garage.rajsingh.info/v1beta1	glance-s3	Imported S3 credentials with read/write on both the glance-images and glance-images-2 buckets

GarageCluster is written against the v1beta2 storage version with replication.factor: 1, a single storage tier, network.service.type: ClusterIP, and the S3 API on :3900 with SigV4 region garage (path-style addressing — virtual-host style would need wildcard DNS). No spec.image is set, so the Garage version rides the operator/chart releases. The kind overlay (deploy/kind/infrastructure/kustomization.yaml) patches the storage tier to a single node with small PVCs on the standard storage class, mirroring the MariaDB/Memcached single-node kind footprint. This is a CI/dev fixture — plain HTTP in-cluster, single tier; production-grade multi-node/zone-aware guidance is out of scope.

Credential flow — OpenBao stays the single source of truth. No key material is read back from Garage, and no Secret is copied from one namespace to another: each consuming namespace gets its own ExternalSecret on the same OpenBao path.

1. write-bootstrap-secrets.sh seeds an admin token at bootstrap/openstack/garage/admin-token and a GK-prefixed S3 access/secret pair at bootstrap/openstack/garage/s3-credentials (see the OpenBao bootstrap reference). The openstack segment names the consuming tenant, not the namespace Garage runs in.
2. Kind-only ExternalSecrets (deploy/kind/infrastructure/garage-{admin-token,s3-credentials}-externalsecret.yaml) materialize them through the shared openbao-cluster-store: garage-admin-token and one garage-s3-credentials copy into shared-services, beside the CRs that read them, and a second garage-s3-credentials copy into openstack.
3. The GarageCluster reads the admin token via spec.admin.adminTokenSecretRef; the GarageKey imports the pre-existing S3 pair via spec.importKey.secretRef (rather than the operator minting a fresh key), so the key material never diverges from OpenBao. A GlanceBackend resolves its credentialsSecretRef in the Glance service's own namespace, which is what the openstack copy serves.

OpenBao Proving Instance

File: deploy/kind/infrastructure/openbao-instance.yaml

Five resources declare the proving instance: the single-replica OpenBaoCluster that the openbao-operator manages, plus the TLS and RBAC objects it depends on. All live in the openstack namespace except the cluster-scoped ClusterRoleBinding:

Kind	API version	Name	Purpose
Certificate	cert-manager.io/v1	openbao-instance-tls-server	Server certificate for the API listener, carrying the SAN openbao-cluster-openbao-instance.local that the operator's External-mode validation requires
Certificate	cert-manager.io/v1	openbao-instance-tls-ca	Delivers the trust-domain CA into the fixed-name Secret the operator reads (data key ca.crt)
ServiceAccount	v1	openbao-instance-provisioner	Client identity the Kubernetes-auth role provisioner binds to
ClusterRoleBinding	rbac.authorization.k8s.io/v1	openbao-instance-auth-delegator	Grants system:auth-delegator to the operator-created instance ServiceAccount openbao-instance-serviceaccount
OpenBaoCluster	openbao.org/v1alpha1	openbao-instance	Profile Development, version 2.6.2, one replica, 1Gi raft storage, TLS mode External, static seal, self-init enabled, applied paused, API-server egress patched in at deploy time

Tenant. deploy/kind/infrastructure/openbao-tenant.yaml declares the OpenBaoTenant openstack that admits the openstack namespace to the multi-tenant operator. It lives in openbao-operator-system and targets openstack from there, because the chart's admission policy accepts spec.quota and spec.limitRange only on a tenant in the operator namespace. Without those overrides the provisioner writes its default LimitRange into openstack, which gives every container without a CPU limit a 500m one. The overrides are the provisioner defaults of chart 0.4.2 minus the CPU limits:

Object	Setting	Value
ResourceQuota	pods	50
ResourceQuota	requests.cpu	20
ResourceQuota	requests.memory	64Gi
ResourceQuota	limits.memory	128Gi
LimitRange (Container)	default.memory	512Mi
LimitRange (Container)	defaultRequest.cpu	100m
LimitRange (Container)	defaultRequest.memory	128Mi

The upstream default default.cpu: 500m and limits.cpu: 40 are left out: the service operators set no CPU limit, and a limits.cpu quota rejects every pod that sets none. An override replaces the whole default object, so every kept key is spelled out. hack/deploy-infra.sh waits for status.provisioned on the tenant in openbao-operator-system before it waits for the instance. A kind cluster created before the tenant moved keeps its old openstack/openstack tenant beside the new one, so recreate it.

The instance runs in every kind deploy, so the primitives a managed Barbican secret store needs are exercised with no Barbican attached: static-seal auto-unseal, cert-manager TLS in External mode, declarative self-init, and the AppRole and Kubernetes-auth methods a service and its operator log in with.

Why it is kind-only. Everything above is a proving posture, not a production one. The Development profile is what keeps the instance on a single node — one replica, so no PodDisruptionBudget and no anti-affinity — and the static seal keeps its key material in a Secret in the shared openstack namespace. Hardened, the profile that forces an external-KMS unseal and three replicas, is the production control, and this instance deliberately opts out of it to exercise the static-seal path. Shipping that from deploy/flux-system/infrastructure/ would put it, a cluster-scoped RBAC binding, and a self-initialized AppRole with no consumer into every production deployment. Only the openbao-operator itself ships in the production base. The dedicated instance a ControlPlane projects for a managed Barbican secret store carries the same proving-grade Development profile; a Hardened production-shaped instance is still future work. tests/unit/deploy/openbao_instance_overlay_test.sh asserts both directions of that split.

Two Certificates, one trust domain. The operator's External TLS contract reads two fixed-name Secrets, <cluster>-tls-server and <cluster>-tls-ca. cert-manager has no primitive that materializes a CA-only Secret in another namespace, so openbao-instance-tls-ca is a minimal Certificate that exists for one reason: every Secret issued by a CA-type ClusterIssuer carries the trust-domain CA in ca.crt. Its leaf keypair goes unused. Both Certificates are issued by openbao-ca-issuer, the same root as the management OpenBao's server and client certificates, so the server certificate chains directly to the CA the operator hands the instance. The operator mounts both Secrets and waits for them, but never rotates them; cert-manager owns the renewal.

Because openbao-ca-issuer is also what the management OpenBao trusts as tls_client_ca_file behind tls_require_and_verify_client_cert, both Certificates pin usages to digital signature, key encipherment, and server auth. Without the pin cert-manager emits no EKU at all, and a certificate without an EKU passes client-auth verification — these Secrets live in the shared workload namespace, so they would be ready-made client identities for that mTLS gate. For the same reason the server certificate carries no loopback IP SAN: such a SAN authenticates whatever listens on localhost rather than this instance. In-pod clients connect over the loopback address and verify against the operator's own SAN by passing VAULT_TLS_SERVER_NAME.

TokenReview authority. OpenBao validates a Kubernetes-auth login by issuing a TokenReview under the instance pod's own projected ServiceAccount token. The operator creates that ServiceAccount (<cluster>-serviceaccount), and neither the operator nor its chart grants it TokenReview, so openbao-instance-auth-delegator supplies the binding. Without it every login fails with 403 permission denied. The binding is overlay-owned while its subject is created and garbage-collected by the operator with the CR, so deleting the CR leaves it dangling — the operator exposes no way to point the instance at a ServiceAccount whose lifecycle the overlay controls, which is one more reason the instance stays out of the production overlay.

Unseal-key custody. The management OpenBao is the root of trust for the static seal:

1. write-bootstrap-secrets.sh seeds a generated key at bootstrap/openstack/openbao-instance/unseal-key (key key), behind the write_secret_if_missing guard.
2. The ExternalSecret deploy/kind/infrastructure/openbao-instance-unseal-key-externalsecret.yaml reads that path through the shared openbao-cluster-store and materializes the Secret openbao-instance-unseal-key in openstack. It is refreshPolicy: CreatedOnce: the Secret is materialized once and never converged again. Its target is creationPolicy: Orphan, which leaves the Secret's single controller ownerReference slot free for step 4.
3. The CR is applied with spec.paused: true. The operator blind-creates the same fixed-name Secret with a random key on its first reconcile and adopts a pre-existing one only against an ownership proof, so it must not run first.
4. hack/deploy-infra.sh syncs the ExternalSecret, patches a controller ownerReference to the CR onto the materialized Secret, un-pauses the CR, and later waits for condition Available. The operator accepts either that reference or its openbao.org/owner-uid annotation as proof, but the openbao-lock-managed-resource-mutations ValidatingAdmissionPolicy the operator chart ships reserves the annotation for the operator's own ServiceAccounts and denies every other writer, so the reference is the only proof the deploy can attach.
5. The operator adopts the Secret and mounts the key at /etc/bao/unseal, where the static seal reads it on every start. Nothing ever sends an unseal command.

The key is never rotated, and both ends enforce that rather than assume it: a static seal whose key material changes seals the instance permanently, with no path back to the stored data. write_secret_if_missing keeps the seed side from re-generating it, and refreshPolicy: CreatedOnce keeps the consuming side from re-materializing a changed seed — the case that arises when the management OpenBao loses its raft PVC and is re-seeded while the instance's own PVC survives. No PushSecret targets the path.

Accepted risk (openbao-operator upstream). The operator's adoption guard is metadata carrying the CR's UID, and that UID is readable by any principal with get openbaocluster. It is an ownership marker, not an ownership proof: anyone able to create Secrets in the namespace before the instance first initializes can plant a seal key of their choosing. Accepted because this instance is CI/dev-only; a production instance must not depend on that guard.

API-server egress. The operator wraps the instance pods in a deny-by-default NetworkPolicy and derives the API-server egress rule from the in-cluster service VIP on port 443. kindnet enforces egress against the post-DNAT destination from kind 0.32 onwards, and the packet it inspects is addressed to the API server's own endpoint on port 6443, so the VIP rule never matches. The instance then loses the API server: raft auto-join times out, self-init cannot complete, and the partial raft state wedges every later initialization attempt, recoverable only by deleting the CR together with its PVC.

spec.network.apiServerEndpointIPs closes that gap with one egress rule per address on port 6443. The manifest sets no value, because a kind node address does not survive a cluster re-creation. hack/deploy-infra.sh reads the addresses from the EndpointSlice kubernetes in default, which kube-apiserver maintains itself, and applies them in the same patch that un-pauses the CR, so the operator's first reconcile already renders the rules. Resolving no address aborts the deploy. The operator reports its own verdict on the result as condition APIServerNetworkReady, which stays Unknown with reason APIServerEndpointIPsRecommended while only the service VIP is allowed.

The operator renders the rule for apiServerEndpointIPs on port 6443 and no other. The same patch therefore also sets spec.network.egressRules: one rule per address, a /32 block (/128 for IPv6) on TCP at the port the EndpointSlice publishes. On kind that is 6443 and the rule duplicates the operator's own; behind Gardener's apiserver-proxy on the metal-stack lab it is 443, where the addresses alone allow nothing. A slice that publishes no port aborts the deploy like one without an address.

Self-init surface. The operator renders spec.selfInit.requests into OpenBao's initialize stanzas, which run once against freshly initialized storage; OpenBao revokes the root token afterwards. The list below is therefore the instance's complete permanent configuration:

Requests	Paths	Result
barbican_kv	sys/mounts/barbican	KV v2 mount barbican/
barbican_secretstore_policy	sys/policies/acl/barbican-secretstore	Policy barbican-secretstore: create/read/update/delete/list on barbican/data/*, and the same minus delete on barbican/metadata/*
approle_auth, barbican_approle_role	sys/auth/approle, auth/approle/role/barbican	AppRole role barbican with token_policies=barbican-secretstore, token_ttl=1h, token_max_ttl=4h, secret_id_ttl=720h
kubernetes_auth, kubernetes_auth_config	sys/auth/kubernetes, auth/kubernetes/config	Kubernetes auth mount, kubernetes_host=https://kubernetes.default.svc
provisioner_policy, provisioner_k8s_role	sys/policies/acl/provisioner, auth/kubernetes/role/provisioner	Policy provisioner (read the barbican role ID, create the secret ID, the same barbican/ data and metadata grants as above, read/update on sys/mounts/barbican) and the Kubernetes role bound to the openbao-instance-provisioner ServiceAccount in openstack, audience=openbao-instance

No policy grants delete on barbican/metadata/*. In KV v2 that verb permanently destroys every version of a secret and its metadata, which is the only in-store recovery mechanism tenant key material has; castellan's Vault key manager deletes through barbican/data/*, where the delete is a recoverable soft delete. secret_id_ttl is 30 days rather than a year for a related reason: an AppRole secret ID carries no use count and no CIDR bound, so its TTL is the only thing that bounds a leaked one.

The Kubernetes-auth role is audience-bound. Without audience, the role accepts any default-audience token of that ServiceAccount, including one auto-mounted into an unrelated pod; bound to openbao-instance, only a token minted deliberately for this instance logs in (kubectl create token --audience=openbao-instance). The ServiceAccount itself sets automountServiceAccountToken: false, since every consumer mints explicitly.

Self-init is one-shot. Changing the request list on a running instance changes nothing. The stanzas only run against freshly initialized storage, so a different configuration means recreating the instance: delete the CR and its PVC. Runtime work (minting AppRole secret IDs, writing secrets) goes through the provisioner role instead. Because that failure is silent — the CR reconciles, Available stays True, and the new request is simply never applied — tests/unit/deploy/openbao_instance_overlay_test.sh pins the request names, so an edit cannot land without confronting it.

Shared names with the brownfield leg. The mount, policy, and role names (barbican/, barbican-secretstore, barbican) match the ones the bootstrap scripts provision on the shared management OpenBao (enable_barbican_kv in setup-secret-engines.sh, deploy/openbao/policies/barbican-secretstore.hcl, and the barbican AppRole role in setup-auth.sh; see the OpenBao bootstrap reference). A deployment that attaches Barbican to the shared instance instead of a dedicated one therefore differs only in which instance it points at.

tests/e2e/infrastructure/openbao-instance/chainsaw-test.yaml locks the whole path: the operator Deployment and its HelmRelease, the unseal-key ExternalSecret reporting SecretSynced and the instance's ownerReference adoption of the Secret it materialized, the CR reporting Available, a Kubernetes-auth login that reads the AppRole role ID and mints a secret ID, a rejected login with a wrong secret ID, and a KV v2 round-trip on barbican/. The last step triggers a rolling restart through spec.runtime.restartAt on the CR and waits for the replacement pod to come back unsealed. It does not delete the pod directly: the operator ships a ValidatingAdmissionPolicy that denies any mutation of the resources it manages, so a restart has to go through the parent CR. The suite issues no unseal command anywhere, which is what makes that step a proof of the static seal.

Admin Credential Chain

The c5c3-operator mints a single restricted admin Application Credential per cluster and mirrors it to OpenBao, from where the External Secrets Operator materialises it as the clouds.yaml Secret that K-ORC authenticates with. The chain materialises the Kubernetes Secret k-orc-clouds-yaml via a single ExternalSecret, created per ControlPlane by the operator.

The figure numbers the steps of the chain. K-ORC admin credential chain lists them.

Namespace	Source	Purpose
openstack (control-plane)	operator-created per-CR (reconcileKORC → ensureKORCCloudsYAMLExternalSecret)	C1 co-location — the c5c3-operator creates the K-ORC ApplicationCredential/Service/Endpoint CRs in the control-plane namespace, and K-ORC resolves each CR's CloudCredentialsRef Secret in that same namespace, so the admin clouds.yaml must live here for K-ORC to authenticate. This is the copy the AdminCredentialReady gate waits on.

Control-plane copy (operator-created per ControlPlane) — the control-plane-namespace k-orc-clouds-yaml ExternalSecret is not a static manifest: the c5c3-operator creates and owns one per ControlPlane (reconcileKORC → ensureKORCCloudsYAMLExternalSecret), owner-ref'd to the CR for GC and created in the ControlPlane's child namespace. It is named after spec.korc.adminCredential.cloudCredentialsRef.secretName (default k-orc-clouds-yaml) and reads the per-CR OpenBao key openstack/keystone/{namespace}/{name}/admin/app-credential (property clouds.yaml, store-relative to the KV-v2 mount) through the ControlPlane's store, by default the namespaced SecretStore openbao-tenant-store, with creationPolicy: Owner and refreshInterval: 1h. Because both the ExternalSecret name and the OpenBao key are derived per-CR, an arbitrarily named ControlPlane resolves to the correct key with no manifest edit — the operator now resolves what was previously deferred for this ExternalSecret.

Optional cacert entry (External keystone mode) — when the ControlPlane sets spec.services.keystone.external.caBundleSecretRef, the ExternalSecret carries a second data entry that reads the cacert property back from the same per-CR OpenBao key. K-ORC reads that inline PEM key natively from the credentials Secret, so the materialised k-orc-clouds-yaml ends up carrying the private-CA trust anchor next to clouds.yaml. No extra push plumbing is needed: the PushSecret mirrors the source Secret whole (it declares no match.secretKey), so the cacert key the operator projects into the app-credential Secret already reaches OpenBao alongside clouds.yaml. Clearing the ref drops the read-back entry and removes the cacert key from the source Secret on the next reconcile. The operator then forces a re-push (c5c3.io/push-cacert-hash), so the Secret in OpenBao no longer carries the key. The PushSecret's deletionPolicy is Delete: the OpenBao key leaves with the ControlPlane.

No orc-system copy — the static deploy/eso/externalsecrets/k-orc-clouds-yaml.yaml manifest that previously declared K-ORC's global default clouds.yaml mount has been removed. K-ORC authenticates per resource via each CR's CloudCredentialsRef, resolved in the control-plane namespace, so no cluster-global default mount is needed. The orc-system Namespace itself is retained (co-declared in deploy/flux-system/namespaces.yaml) because the K-ORC installer's own resources land there — it no longer hosts a clouds.yaml copy.

On a fresh cluster the bootstrap clouds.yaml at the per-CR OpenBao key is seeded by the operator (reconcileKORC → seedBootstrapCloudsYAML, write-if-empty): it writes a password-based bootstrap clouds.yaml into the admin Application Credential Secret, and the operator's PushSecret mirrors it to OpenBao so the ExternalSecrets can materialise before any credential is minted. Once the c5c3-operator mints the admin Application Credential the same PushSecret overwrites the key with the App-Cred-based clouds.yaml.

OpenBao policy — deploy/openbao/policies/eso-tenant.hcl

This per-tenant policy grants the write path for each ControlPlane's admin credential PushSecret. Because the admin credential path is keyed per ControlPlane (openstack/keystone/{namespace}/{name}/admin/app-credential), the grant templates the namespace to the caller's OWN service_account_namespace ({ns} below) and matches the ControlPlane name with a single + segment. The policy is bound to the eso-tenant auth role and reached through the namespaced openbao-tenant-store SecretStore, so a tenant's PushSecret can only write its own namespace's leaves:

Path	Capabilities	Purpose
kv-v2/data/openstack/keystone/{ns}/+/admin/app-credential	create, update, read, delete	Write (and soft-delete on teardown) each ControlPlane's admin Application Credential clouds.yaml data leaf ({ns} is templated to the caller's namespace; the + segment is the ControlPlane name)
kv-v2/metadata/openstack/keystone/{ns}/+/admin/app-credential	create, update, read	Allow ESO's Vault provider to write custom_metadata on the KV-v2 PushSecret (a data-only grant 403s on the metadata PUT and the PushSecret never reaches Ready)
kv-v2/data/openstack/keystone/{ns}/+/service-accounts/+	create, update, read, delete	Write (and soft-delete on teardown) each declared service account's password clouds.yaml data leaf; the + segments are the ControlPlane's name and the account name (each a DNS-1123 label, so a single + leaf suffices)
kv-v2/metadata/openstack/keystone/{ns}/+/service-accounts/+	create, update, read	Allow ESO's Vault provider to write custom_metadata on the service-account KV-v2 PushSecret

{ns} is substituted by OpenBao ACL identity templating with the caller's own service-account namespace, and each + matches exactly one path segment, so the grants terminate at the literal /admin/app-credential and /service-accounts/+ leaves and admit no deeper, sibling, or cross-tenant paths. Read coverage needs no separate grant: the same eso-tenant policy carries read on the caller's own kv-v2/data/openstack/keystone/{ns}/* subtree, which covers the read-back leg of every per-CR admin/app-credential and service-accounts PushSecret. These grants stay scoped to the per-tenant admin-credential and service-account leaves, adding no blast radius beyond them. For the mTLS transport gate and the SecretStore auth path these manifests ride on, see OpenBao Bootstrap Procedure.

Kustomization

Deployment is split into two kustomize overlays to separate base resources from CRD-dependent infrastructure resources:

Base Kustomization

File: deploy/flux-system/kustomization.yaml

The base kustomization uses apiVersion: kustomize.config.k8s.io/v1beta1 and includes namespaces, the FluxInstance CR, HelmRepository sources, and HelmRelease operators. These resources depend on no CRD that an operator of this stack installs. The Flux kinds among them need the flux-operator, which registers the FluxInstance kind and, once it has reconciled the instance, the Flux toolkit kinds.

Resource count: 35 files producing 55 Kubernetes resources.

Category	Count	Resources
Namespace	21	cert-manager, mariadb-system, external-secrets, monitoring, memcached-system, openstack, shared-services, keystone-system, horizon-system, glance-system, placement-system, barbican-system, ovn-system, neutron-system, cinder-system, nova-system, c5c3-system, garage-system, orc-system, openbao-operator-system, rabbitmq-system
FluxInstance	1	flux (drives the flux-operator)
HelmRepository	7	cert-manager, mariadb-operator, external-secrets, openbao, c5c3-charts, prometheus-community, garage-operator
OCIRepository	1	openbao-operator
GitRepository	3	k-orc, rabbitmq-cluster-operator, messaging-topology-operator
HelmRelease	19	cert-manager, mariadb-operator-crds, prometheus-operator-crds, mariadb-operator, memcached-operator, external-secrets, garage-operator, openbao, openbao-operator, ovn-operator, keystone-operator, horizon-operator, glance-operator, placement-operator, cinder-operator, nova-operator, barbican-operator, neutron-operator, c5c3-operator
Kustomization	3	k-orc, rabbitmq-cluster-operator, messaging-topology-operator
Total	55

The chaos-mesh HelmRepository, HelmRelease, and Namespace ship in the kind-only opt-in overlay at deploy/kind/chaos-mesh/ and are not counted here.

Infrastructure Kustomization

File: deploy/flux-system/infrastructure/kustomization.yaml

The infrastructure kustomization includes CRD-dependent resources that require their operator CRDs to be installed first. This kustomization must be applied after the base kustomization and after operators have finished installing their CRDs.

Resource count: 9 manifests and the ../../eso kustomization, producing 17 Kubernetes resources (the db-ca-issuer.yaml, ovn-ca-issuer.yaml, and openbao-ca-issuer.yaml manifests declare two resources each: a CA Certificate and the CA-type ClusterIssuer that signs from it; garage.yaml declares four; openbao-client-tls-cert.yaml declares two).

Category	Count	Resources
ClusterIssuer	4	selfsigned-cluster-issuer, openstack-db-ca-issuer, openstack-ovn-ca-issuer, openbao-ca-issuer (all require cert-manager CRDs)
Certificate (CA keypairs)	3	openstack-db-ca, openstack-ovn-ca, openbao-ca — all three CA keypair Secrets in the cert-manager namespace, signed by selfsigned-cluster-issuer
Certificate (OpenBao TLS)	3	openbao-tls, openbao-client-tls, eso-openbao-client-tls in shared-services
MariaDB	1	openstack-db (requires mariadb-operator CRDs; TLS enabled per MariaDB Galera Cluster)
Memcached	1	openstack-memcached (requires memcached-operator CRDs)
GarageCluster / GarageBucket / GarageKey	4	garage, glance-images, glance-images-2, glance-s3 (require garage-operator CRDs; see Garage Object Store)
ClusterSecretStore	1	openbao-cluster-store, from deploy/eso/
Total	17

The proving instance's own five resources (two Certificates, a ServiceAccount, a ClusterRoleBinding, and the OpenBaoCluster) are not counted here: they ship in the kind overlay — see OpenBao Proving Instance.

Deployment

The two steps below apply the two kustomizations by hand. make deploy-infra applies the same two phases, on the kind or the lab overlay, as its Step 3 and Step 5, and runs the waits and the OpenBao bootstrap around them. The figure shows that run. Deployment Sequence describes each numbered step, and Steps and phases maps the step and phase numbers of the two pages.

Step 1: Apply base resources

kubectl apply -k deploy/flux-system/

On a cluster where FluxInstance/flux is not Ready yet, the sources and releases fail with no matches for kind "HelmRepository": the flux-operator registers the Flux toolkit CRDs only when it reconciles the instance. Apply namespaces.yaml and fluxinstance.yaml first and wait for the instance, as Step 2 of hack/deploy-infra.sh does.

This applies 55 resources: 21 namespaces, 1 FluxInstance, 11 sources (7 HelmRepository, 1 OCIRepository, 3 GitRepository), 19 HelmReleases and 3 Flux Kustomizations (K-ORC, the RabbitMQ Cluster Operator and the RabbitMQ Messaging Topology Operator). FluxCD resolves the dependency graph between HelmReleases and installs operators in the correct order. Wait for all operators to finish installing before proceeding to step 2.

Step 2: Apply infrastructure resources

kubectl apply -k deploy/flux-system/infrastructure/

This applies the CRD-dependent resources: the selfsigned-cluster-issuer ClusterIssuer, the openstack-db-ca-issuer ClusterIssuer plus its backing CA Certificate, the openstack-ovn-ca-issuer ClusterIssuer plus its backing CA Certificate, the openbao-ca-issuer ClusterIssuer plus its backing CA Certificate, the MariaDB Galera cluster, the Memcached cluster, and the Garage object store (GarageCluster / GarageBucket / GarageKey). These resources require CRDs that are installed by the operator HelmReleases in step 1. If CRDs are not yet available, the apply will fail — wait for the operators to finish installing and retry.

hack/deploy-infra.sh ordering. The end-to-end deploy script applies the four TLS-prerequisite manifests (cluster-issuer.yaml, openbao-ca-issuer.yaml, openbao-tls-cert.yaml, db-ca-issuer.yaml) directly in Phase 2 of its Step 4, before the main infrastructure kustomization, so that MariaDB has openstack-db-ca-issuer available the moment it tries to render its server certificate. The kustomization apply that follows is idempotent — the same manifests are listed in infrastructure/kustomization.yaml so a manual kubectl apply -k path also works.

Expected transient failure: The MariaDB cluster references a rootPasswordSecretKeyRef Secret (mariadb-root-password). On kind, the overlay shim materialises it from OpenBao via the External Secrets Operator; in production a Flux MariaDB baseline must provide it. Until that Secret exists, the mariadb-operator will enter a failed reconciliation loop with Secret "mariadb-root-password" not found errors. This is expected and resolves automatically once the Secret is provisioned.

Validate manifests locally

kustomize build deploy/flux-system/
kustomize build deploy/flux-system/infrastructure/

These commands render the manifest output without applying it. Use them to verify YAML syntax and resource inclusion before deployment.

Prerequisites

• A Kubernetes cluster with the flux-operator installed. The FluxInstance of the base installs the Flux controllers, among them the kustomize-controller that the two Flux Kustomizations need
• kubectl configured with cluster access
• For local validation only: kustomize CLI

Sizing and placement overrides

The manifests under deploy/flux-system/ ship the components' own sizing and no node placement. A deployer changes both with a kustomize overlay per apply phase and leaves the shipped files untouched. deploy/examples/sizing-overlay/ is an example of such an overlay. No script applies it, and tests/unit/deploy/sizing_overlay_example_test.sh renders it and pins every field it patches. The kind devstack's deploy/kind/base/ is an overlay built the same way.

Recipe

1. Copy deploy/examples/sizing-overlay/, including its infrastructure/ subdirectory. Keep, change or drop each patch, and add one for every other workload you size, using the field the table below names.
2. Point the base entry in each phase's resources list at its base; the priorityclass.yaml entry stays as it is. Inside a checkout of this repository a relative path works. Elsewhere, use a remote base pinned to a commit: https://github.com/C5C3/cobaltcore//deploy/flux-system?ref=<commit> for the base phase and https://github.com/C5C3/cobaltcore//deploy/flux-system/infrastructure?ref=<commit> for the infrastructure phase. When you move <commit> later, compare each copied spec.chart.spec.version with the base's at the new commit. The patch replaces the whole range, so an upper bound the base has moved past holds the release on an older chart, and Flux downgrades a release that already runs a newer one.
3. Render both phases with kustomize build <overlay>/ and kustomize build <overlay>/infrastructure/ and review the output.
4. Label the nodes the platform workloads move to before you apply either phase: kubectl label node <node> node.c5c3.io/role=platform. A pod whose nodeSelector matches no node stays Pending. To keep other workloads off those nodes, also taint them with kubectl taint node <node> node.c5c3.io/role=platform:NoSchedule; the example's tolerations admit its own pods. Check that kubectl get nodes -l node.c5c3.io/role=platform lists the nodes you expect.
5. Run kubectl apply -k <overlay>/ in place of Step 1, then kubectl apply -k <overlay>/infrastructure/ in place of Step 2. The base phase applies the cobaltcore-platform PriorityClass that the patches of both phases reference. The infrastructure phase does not include it, so kubectl delete -k <overlay>/infrastructure/ leaves the class in place for the base phase's pods. On a running deployment the MariaDB patch changes the Galera pod template, and mariadb-operator replaces the pods one at a time, replicas first and the primary last (its default ReplicasFirstPrimaryLast update strategy). A replaced pod that cannot be scheduled stops the update with the cluster one member short, so watch kubectl get mariadb openstack-db -n openstack until it reports Ready again before you change anything else.

The example uses the node label node.c5c3.io/role: platform, a matching NoSchedule toleration, and the PriorityClass cobaltcore-platform at 1000000. Its resource figures are illustrative; size them against the usage you observe.

Example files

The base phase, deploy/examples/sizing-overlay/kustomization.yaml:

# SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company
#
# SPDX-License-Identifier: Apache-2.0

# Example sizing and placement overlay for the base FluxCD manifests.
#
# This directory is an example to copy. No script applies it;
# tests/unit/deploy/sizing_overlay_example_test.sh renders it and pins every
# patched field. A deployer copies the directory, points the resources entry
# below at a checkout or a remote base, and runs kubectl apply -k on the copy
# in place of deploy/flux-system/. infrastructure/ does the same for the
# second apply phase. The recipe is in
# docs/reference/infrastructure/infrastructure-manifests.md, section "Sizing
# and placement overrides".
#
# Each patch shows one shape the stack uses: a CobaltCore operator HelmRelease,
# a third-party HelmRelease, the two Git-sourced Flux Kustomizations and the
# FluxInstance. The figures are illustrative; size them against the usage you
# observe. The placement values are:
#   node label     node.c5c3.io/role: platform
#   toleration     node.c5c3.io/role=platform:NoSchedule
#   PriorityClass  cobaltcore-platform (priorityclass.yaml, which only this
#                  phase applies; infrastructure/ references it)
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../flux-system
  - priorityclass.yaml

patches:
  # A CobaltCore operator chart: a merge patch on spec.chart.spec.version and
  # spec.values. nodeSelector, tolerations and priorityClassName need
  # keystone-operator chart 0.11.0 or later (operator-library 0.9.0); an older
  # chart rejects them through its values schema, so the patch raises the
  # release's version floor with them. The patch replaces the base's whole
  # range, so it repeats the base's upper bound; re-check it whenever the
  # base moves. A patch copied to one of the other nine operator releases
  # raises that release's floor to its chart version of the same change:
  # barbican, glance, neutron, ovn and placement 0.3.0, c5c3 0.14.0, horizon
  # 0.4.0, cinder and nova 0.2.0.
  - target:
      group: helm.toolkit.fluxcd.io
      version: v2
      kind: HelmRelease
      name: keystone-operator
      namespace: keystone-system
    patch: |
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: keystone-operator
        namespace: keystone-system
      spec:
        chart:
          spec:
            version: ">=0.11.0 <1.0.0"
        values:
          # One replica fits a single-node cluster; keep two for HA.
          replicas: 1
          resources:
            requests:
              cpu: 20m
              memory: 96Mi
            limits:
              cpu: 500m
              memory: 192Mi
          nodeSelector:
            node.c5c3.io/role: platform
          tolerations:
            - key: node.c5c3.io/role
              operator: Equal
              value: platform
              effect: NoSchedule
          priorityClassName: cobaltcore-platform

  # A third-party chart: a merge patch on spec.values with the chart's own
  # keys, checked against cert-manager chart v1.21.2. The top-level keys size
  # and place the controller; webhook.* and cainjector.* take the same keys
  # for the other two Deployments, and startupapicheck.* for the Job the
  # release runs once at install. global.priorityClassName applies to all
  # four pods.
  - target:
      group: helm.toolkit.fluxcd.io
      version: v2
      kind: HelmRelease
      name: cert-manager
      namespace: cert-manager
    patch: |
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: cert-manager
        namespace: cert-manager
      spec:
        values:
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
          nodeSelector:
            node.c5c3.io/role: platform
          tolerations:
            - key: node.c5c3.io/role
              operator: Equal
              value: platform
              effect: NoSchedule
          global:
            priorityClassName: cobaltcore-platform

  # A Git-sourced Flux Kustomization over an upstream installer: spec.patches
  # holds a JSON6902 patch that the kustomize-controller applies to the
  # installer it renders. The K-ORC installer (config/default) and the
  # RabbitMQ cluster-operator installer (config/installation) each render
  # exactly one Deployment with one container, so target kind: Deployment
  # selects the manager whatever name prefix the upstream base applies, and
  # containers/0 is the manager container. op: add replaces a member that
  # already exists, such as the upstream resources.
  - target:
      group: kustomize.toolkit.fluxcd.io
      version: v1
      kind: Kustomization
      name: k-orc
      namespace: flux-system
    patch: |
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: k-orc
        namespace: flux-system
      spec:
        patches:
          - target:
              kind: Deployment
            patch: |
              - op: add
                path: /spec/template/spec/nodeSelector
                value:
                  node.c5c3.io/role: platform
              - op: add
                path: /spec/template/spec/tolerations
                value:
                  - key: node.c5c3.io/role
                    operator: Equal
                    value: platform
                    effect: NoSchedule
              - op: add
                path: /spec/template/spec/priorityClassName
                value: cobaltcore-platform
              - op: add
                path: /spec/template/spec/containers/0/resources
                value:
                  requests:
                    cpu: 10m
                    memory: 64Mi
                  limits:
                    memory: 128Mi

  # The RabbitMQ cluster operator: the placement plus the two request
  # operations the base release
  # (deploy/flux-system/releases/rabbitmq-cluster-operator.yaml) already
  # carries. spec.patches is a list, and the merge patch replaces the base's
  # list whole (merge rule 1 in
  # docs/reference/infrastructure/infrastructure-manifests.md), so an entry
  # that named the placement alone would drop the base's 100m / 128Mi
  # requests and put the installer's 200m / 500Mi back. The limits stay the
  # installer's: an add on containers/0/resources would replace that block
  # whole, an add on one member of requests sets that member alone.
  - target:
      group: kustomize.toolkit.fluxcd.io
      version: v1
      kind: Kustomization
      name: rabbitmq-cluster-operator
      namespace: flux-system
    patch: |
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: rabbitmq-cluster-operator
        namespace: flux-system
      spec:
        patches:
          - target:
              kind: Deployment
            patch: |
              - op: add
                path: /spec/template/spec/nodeSelector
                value:
                  node.c5c3.io/role: platform
              - op: add
                path: /spec/template/spec/tolerations
                value:
                  - key: node.c5c3.io/role
                    operator: Equal
                    value: platform
                    effect: NoSchedule
              - op: add
                path: /spec/template/spec/priorityClassName
                value: cobaltcore-platform
              - op: add
                path: /spec/template/spec/containers/0/resources/requests/cpu
                value: 100m
              - op: add
                path: /spec/template/spec/containers/0/resources/requests/memory
                value: 128Mi

  # The FluxInstance: spec.kustomize.patches (flux-operator v0.60.0, the
  # version hack/deploy-infra.sh installs) patches the Flux controllers the
  # flux-operator renders. An entry names one controller; source-controller,
  # kustomize-controller and notification-controller take the same entry with
  # their own name. The Flux manifests already run the source, kustomize and
  # helm controllers with priorityClassName system-cluster-critical, which
  # ranks above cobaltcore-platform, so this patch leaves the priority class
  # alone. The notification-controller has none and takes the class as a
  # fourth operation.
  - target:
      group: fluxcd.controlplane.io
      version: v1
      kind: FluxInstance
      name: flux
      namespace: flux-system
    patch: |
      apiVersion: fluxcd.controlplane.io/v1
      kind: FluxInstance
      metadata:
        name: flux
        namespace: flux-system
      spec:
        kustomize:
          patches:
            - target:
                kind: Deployment
                name: helm-controller
              patch: |
                - op: add
                  path: /spec/template/spec/nodeSelector
                  value:
                    node.c5c3.io/role: platform
                - op: add
                  path: /spec/template/spec/tolerations
                  value:
                    - key: node.c5c3.io/role
                      operator: Equal
                      value: platform
                      effect: NoSchedule
                - op: add
                  path: /spec/template/spec/containers/0/resources
                  value:
                    requests:
                      cpu: 50m
                      memory: 128Mi
                    limits:
                      memory: 512Mi

The PriorityClass, deploy/examples/sizing-overlay/priorityclass.yaml, which the base phase includes:

# SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company
#
# SPDX-License-Identifier: Apache-2.0

# Example PriorityClass for the CobaltCore operators and platform workloads.
#
# This directory is an example to copy. No script applies it;
# tests/unit/deploy/sizing_overlay_example_test.sh renders it. The value ranks
# the pods above tenant workloads (priority 0 without a class) for preemption
# and node-pressure eviction, and stays far below the system-* classes
# (2000000000 and up) that cluster-critical pods such as the Flux controllers
# use.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: cobaltcore-platform
value: 1000000
globalDefault: false
description: "CobaltCore operators and platform workloads; ranked above tenant workloads for preemption and node-pressure eviction"

The infrastructure phase, deploy/examples/sizing-overlay/infrastructure/kustomization.yaml:

# SPDX-FileCopyrightText: Copyright 2026 SAP SE or an SAP affiliate company
#
# SPDX-License-Identifier: Apache-2.0

# Example sizing and placement overlay for the infrastructure CRs, the second
# apply phase.
#
# This directory is an example to copy. No script applies it;
# tests/unit/deploy/sizing_overlay_example_test.sh renders it and pins every
# patched field. A deployer runs kubectl apply -k on the copy in place of
# deploy/flux-system/infrastructure/, after the base phase (../). The MariaDB
# patch references the cobaltcore-platform PriorityClass that the base phase
# applies; this phase takes it from there, as it takes its namespace, CRDs
# and operators. It does not include the class: kubectl delete -k on either
# phase deletes an object both apply, and without the class the API server
# rejects every new pod that names it, the base phase's operators included.
# The figures are illustrative; size them against the usage you observe.
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../flux-system/infrastructure

patches:
  # MariaDB: spec.resources, spec.nodeSelector, spec.tolerations and
  # spec.priorityClassName are the ContainerTemplate and PodTemplate fields of
  # mariadb-operator v0.38.1, which the operator copies onto the MariaDB pods.
  - target:
      group: k8s.mariadb.com
      version: v1alpha1
      kind: MariaDB
      name: openstack-db
      namespace: openstack
    patch: |
      apiVersion: k8s.mariadb.com/v1alpha1
      kind: MariaDB
      metadata:
        name: openstack-db
        namespace: openstack
      spec:
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            memory: 2Gi
        nodeSelector:
          node.c5c3.io/role: platform
        tolerations:
          - key: node.c5c3.io/role
            operator: Equal
            value: platform
            effect: NoSchedule
        priorityClassName: cobaltcore-platform

  # Memcached: spec.resources only; the Memcached CRD has no placement
  # fields. The memory limit stays at or above the 96Mi the Memcached
  # operator's webhook requires for its default maxMemoryMB of 64 (64Mi plus
  # 32Mi overhead).
  - target:
      group: memcached.c5c3.io
      version: v1beta1
      kind: Memcached
      name: openstack-memcached
      namespace: openstack
    patch: |
      apiVersion: memcached.c5c3.io/v1beta1
      kind: Memcached
      metadata:
        name: openstack-memcached
        namespace: openstack
      spec:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            memory: 128Mi

Where each workload is sized

One row per entry of deploy/flux-system/kustomization.yaml and deploy/flux-system/infrastructure/kustomization.yaml that runs pods:

Entry	Kind	Field to patch	Sizing keys
fluxinstance.yaml	FluxInstance	spec.kustomize.patches, one entry per controller Deployment (source-controller, kustomize-controller, helm-controller, notification-controller)	JSON6902 recipe
releases/cert-manager.yaml	HelmRelease	spec.values	Checked against chart v1.21.2: resources, nodeSelector and tolerations for the controller, the same keys under webhook and cainjector for the other two Deployments and under startupapicheck for the Job the release runs once at install, and global.priorityClassName for all four pods
releases/mariadb-operator.yaml	HelmRelease	spec.values	The chart's values reference at https://mariadb-operator.github.io/mariadb-operator (sources/mariadb-operator.yaml)
releases/external-secrets.yaml	HelmRelease	spec.values	The chart's values reference at https://charts.external-secrets.io (sources/external-secrets.yaml)
releases/memcached-operator.yaml	HelmRelease	spec.values	The chart's values reference at oci://ghcr.io/c5c3/charts (sources/c5c3-charts.yaml)
releases/openbao.yaml	HelmRelease	spec.values; the shipped release already sets server.resources	The chart's values reference at https://openbao.github.io/openbao-helm (sources/openbao.yaml)
releases/garage-operator.yaml	HelmRelease	spec.values	The chart's values reference at oci://ghcr.io/rajsinghtech/charts (sources/garage-operator.yaml)
releases/openbao-operator.yaml	HelmRelease	spec.values	The chart's values reference at oci://ghcr.io/dc-tec/charts/openbao-operator (sources/openbao-operator.yaml)
releases/keystone-operator.yaml, horizon-operator.yaml, glance-operator.yaml, placement-operator.yaml, barbican-operator.yaml, ovn-operator.yaml, neutron-operator.yaml, cinder-operator.yaml, nova-operator.yaml, c5c3-operator.yaml	HelmRelease	spec.values, and spec.chart.spec.version with the placement keys (merge rule 3)	replicas, resources, nodeSelector, tolerations, priorityClassName
releases/k-orc.yaml	Flux Kustomization	spec.patches	JSON6902 recipe
releases/rabbitmq-cluster-operator.yaml	Flux Kustomization	spec.patches	JSON6902 recipe
infrastructure/mariadb.yaml	CR (MariaDB)	spec.resources, spec.nodeSelector, spec.tolerations, spec.priorityClassName	The ContainerTemplate and PodTemplate fields of mariadb-operator v0.38.1. The embedded spec.maxScale carries none of them
infrastructure/memcached.yaml	CR (Memcached)	spec.resources	The CRD has no placement fields. The operator's webhook requires a memory limit of at least maxMemoryMB plus 32Mi, 96Mi at the default of 64
infrastructure/garage.yaml	CR (GarageCluster)	spec.storage.resources, spec.storage.nodeSelector, spec.storage.tolerations, spec.storage.priorityClassName	The pod template GarageCluster v1beta2 inlines into spec.storage; confirm against the installed CRD with kubectl explain garagecluster.spec.storage

The two CRD-only releases (prometheus-operator-crds, mariadb-operator-crds), the sources, namespaces, issuers and certificates, and the External Secrets ClusterSecretStore run no pods. The flux-operator itself is installed out of band by hack/deploy-infra.sh and is outside the overlay's reach.

Patching the upstream installers and the Flux controllers

K-ORC and the RabbitMQ cluster operator are Flux Kustomizations over an upstream installer, and the FluxInstance renders the Flux controllers. None of them has a values file, so the overlay adds a JSON6902 patch that the kustomize-controller or the flux-operator applies to the rendered Deployment:

• The K-ORC installer (config/default) and the RabbitMQ cluster-operator installer (config/installation) each render one Deployment with one container at the pinned commits; a pin move has to re-check that. target: {kind: Deployment} selects the Deployment whatever name prefix the upstream base applies (K-ORC's namePrefix: orc- turns controller-manager into orc-controller-manager), and containers/0 is the manager container.
• A FluxInstance entry names one controller Deployment, for example target: {kind: Deployment, name: helm-controller}.
• op: add sets a member whether or not it exists, and replaces the whole value when it does. A patch on resources therefore replaces the upstream block instead of merging into it. The example sizes the K-ORC manager this way. For the RabbitMQ cluster operator it adds one member at a time under requests, repeating the 100m / 128Mi the base release already sets, and leaves the installer's 200m / 500Mi limits in place: spec.patches is a list, so the merge patch replaces the base's list whole (merge rule 1 below), and an entry that named the placement alone would put the installer's requests back.
• The Flux manifests run the source, kustomize and helm controllers with priorityClassName: system-cluster-critical, which ranks above cobaltcore-platform, so the example's FluxInstance patch leaves the helm controller's class alone. The notification-controller has no class and can take cobaltcore-platform as a fourth operation.

Merge rules

1. kustomize applies a merge patch to a HelmRelease's spec.values: a patched list, such as tolerations, replaces the base list, and a patched map merges key by key. Helm then merges the result with the chart defaults the same way, so cert-manager's default kubernetes.io/os: linux node selector stays beside a label the patch adds.
2. A chart value the shipped release sets can be removed only by setting it to null in the patch; omitting the key keeps the shipped value.
3. The nodeSelector, tolerations and priorityClassName keys of the CobaltCore operator charts need operator-library 0.9.0: keystone-operator 0.11.0, c5c3-operator 0.14.0, horizon-operator 0.4.0, cinder-operator and nova-operator 0.2.0, the other five charts 0.3.0, or later. An older chart rejects the keys through its values schema, and the HelmRelease reports the failure in its Ready condition. The shipped releases' spec.chart.spec.version floors still admit such a chart, so a patch that sets the keys raises the floor to the version above as well, as the example does for keystone-operator with >=0.11.0 <1.0.0.

Extensibility

The manifest structure is designed for straightforward extension. Adding a new operator (e.g., OpenBao) requires four steps:

1. Add a source file in sources/ (e.g., sources/openbao.yaml) — or reuse an existing HelmRepository if the chart is in a shared registry
2. Add a release file in releases/ (e.g., releases/openbao.yaml) with the HelmRelease CR, dependsOn for cert-manager, and the standard install/upgrade settings
3. Add both paths to the resources list in kustomization.yaml
4. Add the release's target namespace to namespaces.yaml (e.g., shared-services, where releases/openbao.yaml puts the OpenBao HelmRelease) so the namespace exists before kubectl apply -k creates the namespaced HelmRelease CR

The garage-operator is the worked example of an OCI-type third-party operator following this recipe: step 1 adds an OCI HelmRepository (sources/garage-operator.yaml, spec.type: oci, the registry namespace as url); the release (releases/garage-operator.yaml) references it by sourceRef.name and carries the chart name in chart.spec.chart. The OCI variant changes nothing else in the recipe. Renovate's native Flux manager reads the range of an OCI chart through the docker datasource, and a generic packageRule sets versioning: helm for every such range that starts with >=, so a new OCI chart needs no rule of its own as long as its range is spelled >=X <Y (see Flux HelmRelease chart versions).

An operator with no chart takes the Git-sourced variant of the recipe, which K-ORC and the RabbitMQ Cluster Operator both follow. Step 1 adds a GitRepository scoped to the installer path with spec.ignore, and step 2 adds a Flux Kustomization over that path instead of a HelmRelease, with the image pinned through spec.images (newTag plus digest) because a kustomize base carries no values file. Renovate's native Flux manager is switched off for both files, since a customManager moves a tag together with its commit or digest in one match. Each half needs a customManager anchored on the literal YAML shape, a packageRule sharing one groupName so the pair moves in one PR, and one test per manager under tests/unit/renovate/, and both files join the matchFileNames of the flux rule with enabled: false in renovate.json. Two more places have to learn the name: the FLUX_KUSTOMIZATIONS array in hack/deploy-mgmt-cluster.sh, which applies and waits for such a Kustomization after the HelmReleases, and the Phase 3b wait in hack/deploy-infra.sh, since a Kustomization is invisible to wait_for_helmreleases.

Infrastructure instance CRs (e.g., a new database, cache, or object-store cluster) follow the same pattern: add a file in infrastructure/ and list it in infrastructure/kustomization.yaml (see Garage Object Store and its single-node kind patch for a multi-CR example).

Attaching a Barbican secret store. Two targets are already provisioned. The shared management OpenBao carries the brownfield outputs from the bootstrap scripts: the KV v2 mount barbican/, the policy barbican-secretstore, and the AppRole role barbican (see the OpenBao bootstrap reference). For a dedicated instance, OpenBao Proving Instance is the template: the same three names, self-initialized by the openbao-operator, plus the Kubernetes-auth role a service operator authenticates with to mint the AppRole secret ID.

Design Decisions

Two-phase kustomization

Resources are split into a base kustomization (namespaces, sources, releases) and an infrastructure kustomization (CRD-dependent resources). This separation ensures that kubectl apply -k does not attempt to create CRD-dependent resources before the corresponding CRDs exist. The base kustomization can be applied independently, and the infrastructure kustomization is applied after operators have installed their CRDs.

In FluxCD-managed clusters, this pattern maps to two FluxCD Kustomization CRs where the infrastructure Kustomization depends on the base Kustomization (using spec.dependsOn), eliminating noisy first-apply failures.

Explicit namespace resources

All target namespaces are defined as explicit Namespace resources in namespaces.yaml. While HelmReleases set install.createNamespace: true for FluxCD's helm-controller, the explicit namespace resources ensure namespaces exist before kubectl apply -k attempts to create namespaced resources (HelmRelease CRs specify a target namespace in their metadata).

Namespace auto-creation

All HelmReleases set install.createNamespace: true as a safety net for FluxCD deployments. This is complementary to the explicit Namespace resources — the explicit resources handle the kubectl apply -k path, while createNamespace handles edge cases in FluxCD reconciliation.

No secret configuration

The manifests intentionally contain no password, credential, or secret configuration. Secret management is handled by the External Secrets Operator integration, which provisions secrets from an external vault into the cluster.

Memcached Operator source

The Memcached Operator chart is sourced from the shared c5c3-charts OCI registry rather than a dedicated HelmRepository. This follows the project convention of publishing internally-built charts to oci://ghcr.io/c5c3/charts.

Kind Overlay Demo Addons

The kind overlay (deploy/kind/base/kustomization.yaml) layers a small set of kind-only demo manifests on top of the production base. These files live under deploy/kind/base/ and are not referenced from deploy/flux-system/kustomization.yaml, so they never reach production clusters. The section below catalogues these addons; earlier kind-only manifests (Headlamp, OpenBao UI patch) are documented in the Quick Start. Chaos Mesh ships as a separate opt-in kind overlay at deploy/kind/chaos-mesh/ — applied only when WITH_CHAOS_MESH=true is set on make deploy-infra; see Chaos Mesh (kind-only opt-in) below.

Flux Web UI ResourceSet

File: deploy/kind/base/flux-web.yaml

A single ResourceSet CR drives the flux-operator's bundled Flux Web UI as a demo surface for the kind Quick Start (Step 4a). The ResourceSet renders two sibling resources — an OCIRepository pointing at the official flux-operator Helm chart and a HelmRelease that installs that chart with only the Web UI sub-chart enabled.

Property	Value
API version	fluxcd.controlplane.io/v1
Kind	ResourceSet
Name	flux-web
Namespace	flux-system
Chart URL	oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator
Version pin (input)	0.53.x — SemVer range locked to the minor track of FLUX_OPERATOR_VERSION in hack/deploy-infra.sh

Helm values on the nested HelmRelease:

Key	Value	Purpose
web.serverOnly	true	Render only the Web UI Deployment + Service; skip the operator Deployment, CRDs, and RBAC that the original install.yaml bootstrap already owns
installCRDs	false	The flux-operator CRDs (FluxInstance, ResourceSet, ResourceSetInputProvider, …) are already installed by the out-of-band install.yaml apply in hack/deploy-infra.sh — re-applying them here would fight the bootstrap on every reconcile
fullnameOverride	flux-web	Give the Web UI Deployment / Service / ServiceAccount a distinct identity so they do not collide with the operator's own flux-operator-* workload names

Version tracking. The spec.inputs[0].version SemVer range is updated automatically by a Renovate customManager entry in renovate.json that targets deploy/kind/base/flux-web.yaml and pulls release metadata from controlplaneio-fluxcd/flux-operator GitHub releases. The customManager shares the same packageRules as hack/deploy-infra.sh — major upgrades are disabled, minor/patch upgrades auto-merge after a three-day minimumReleaseAge cooldown.

Production opt-out. deploy/flux-system/kustomization.yaml deliberately does not list deploy/kind/base/flux-web.yaml. The flux-operator Web UI ships without token authentication, without TLS termination, and without an Ingress story — it is safe as a localhost port-forward demo on a single-node kind cluster, not as a shared-cluster surface. Production overlays can opt back in explicitly once upstream adds those prerequisites.

Access (kind Quick Start, Step 4a):

kubectl port-forward svc/flux-web -n flux-system 9080:9080

Browse http://localhost:9080 — no login required. The Web UI complements Headlamp by rendering the three flux-operator-specific CRDs (ResourceSet, ResourceSetInputProvider, FluxReport) that the generic Headlamp Flux plugin does not know about.

Chaos Mesh (kind-only opt-in)

File: deploy/kind/chaos-mesh/kustomization.yaml

Chaos Mesh ships as a separate opt-in kind overlay. The default make deploy-infra flow does not install it — first-run deployments skip the privileged chaos-daemon DaemonSet, the chaos-mesh namespace, and the upstream HelmRepository / HelmRelease pair so that developers who never run chaos E2E suites pay zero install cost. The production deploy/flux-system/ overlay also does not install Chaos Mesh.

The overlay is self-contained: the HelmRepository lives in deploy/kind/chaos-mesh/source.yaml and the HelmRelease in deploy/kind/chaos-mesh/release.yaml (both relocated from the former deploy/flux-system/{sources,releases}/chaos-mesh.yaml locations). The overlay bundles them with:

Property	Value
Target namespace	chaos-mesh (created inline with the privileged PodSecurity label required by chaos-daemon's host PID/network access)
Chart	chaos-mesh
Version constraint	>=2.6.0 <3.0.0
Source	chaos-mesh HelmRepository (deploy/kind/chaos-mesh/source.yaml)
Dependencies	cert-manager in cert-manager namespace

Kind-tuning patch (relocated here from deploy/kind/base/kustomization.yaml because kustomize requires the patch target to live in the same overlay):

Helm value	Override	Purpose
chaosDaemon.runtime	containerd	Match the kind node's container runtime
chaosDaemon.socketPath	/run/containerd/containerd.sock	Mount the kind containerd socket so chaos-daemon can attack pods
chaosDaemon.resources	25m / 64Mi requests	Reduce footprint on single-node kind
dashboard.create	false	Dashboard is unnecessary in CI
controllerManager.resources	25m / 64Mi requests	Reduce footprint on single-node kind

These overrides diverge intentionally from the upstream chart defaults (dashboard enabled, larger resource requests, auto-detected runtime), which target multi-node production clusters. Because the patch and the HelmRelease both live in the kind-only overlay, production environments that opt into Chaos Mesh start from the upstream defaults instead of inheriting the kind-tuning values.

No load-restrictor flag required. The overlay has no parent-directory ../../ references — every resource (namespace.yaml, source.yaml, release.yaml) lives under deploy/kind/chaos-mesh/. Kustomize's default LoadRestrictionsRootOnly security check is therefore satisfied without --load-restrictor=LoadRestrictionsNone, which matters because kubectl's embedded kustomize does not expose that flag (kubernetes/kubectl#948) and hack/deploy-infra.sh invokes the apply via kubectl apply -k.

Opt-in usage:

WITH_CHAOS_MESH=true make deploy-infra

This is the prerequisite for make e2e-chaos. See Chaos E2E Tests for the full workflow. The metal-stack lab deploys Chaos Mesh from its own overlay on top of this one, with a namespace filter and a pod that loads the kernel modules; see Lab Chaos Mesh.

kube-prometheus-stack (kind-only opt-in)

File: deploy/kind/prometheus/kustomization.yaml

kube-prometheus-stack ships as a separate opt-in kind overlay. The default make deploy-infra flow does not install it — the monitoring namespace stays absent, and Prometheus / Grafana / the prometheus-operator pods do not consume any of the kind node's CPU or memory budget unless a contributor explicitly opts in. The production deploy/flux-system/ overlay also does not install the stack: production clusters are expected to run their own Prometheus and widen its serviceMonitorSelector to pick up the keystone-operator chart's ServiceMonitor (see Enable Keystone Operator Metrics for that wiring path).

The overlay is self-contained: the Namespace and HelmRelease live in deploy/kind/prometheus/namespace.yaml and deploy/kind/prometheus/release.yaml, and the upstream prometheus-community HelmRepository in deploy/flux-system/sources/prometheus-community.yaml is reused (it is already present for the prometheus-operator-crds HelmRelease in the production base, so no new source manifest is added to the production tree). The overlay bundles the resources with:

Property	Value
Target namespace	monitoring (created inline; no PodSecurity label override required)
Chart	kube-prometheus-stack
Version constraint	>=65.0.0 <70.0.0
Source	prometheus-community HelmRepository (reused from deploy/flux-system/sources/)
Dependencies	cert-manager in cert-manager namespace

Kind-tuned values (deliberately too lean for a real workload — they exist so the stack fits in a single-node kind cluster alongside Flux, the operators, and the OpenStack control plane):

Helm value	Override	Purpose
crds.enabled	false	The monitoring.coreos.com CRDs are already installed by the production-base prometheus-operator-crds HelmRelease — re-installing them from the chart would fight that release on every reconcile
alertmanager.enabled	false	No alert routing in a developer cluster
nodeExporter.enabled	false	Single-node kind has no meaningful node-level metrics worth scraping
kubeStateMetrics.enabled	false	Kube-state-metrics adds noise the kind dashboards do not consume
prometheus.prometheusSpec.retention	6h	Short retention keeps the Prometheus PVC tiny on kind
prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues	false	Allow the operator chart's ServiceMonitor to be scraped without forcing a release: kube-prometheus-stack label on it
prometheus.prometheusSpec.serviceMonitorSelector	{}	Match every ServiceMonitor in the cluster (kind only — production overlays should use a tighter selector)
prometheus.prometheusSpec.serviceMonitorNamespaceSelector	{}	Match every namespace (kind only — see above)
prometheus.prometheusSpec.resources / grafana.resources	100m CPU / 256Mi mem caps	Hard cap on kind resource use

Dashboard provisioning. The overlay also adds a configMapGenerator that bundles the keystone-operator dashboard JSON (operators/keystone/dashboards/keystone-operator.json — the single source of truth, never forked into the overlay) with the grafana_dashboard: "1" and app.kubernetes.io/part-of: kube-prometheus-stack labels. Grafana's sidecar discovers the labelled ConfigMap on startup and imports it into the Dashboards → Keystone Operator entry without any manual API call. Because the dashboard JSON lives outside the overlay directory, hack/deploy-infra.sh performs an idempotent copy into deploy/kind/prometheus/keystone-operator.json immediately before kubectl apply -k runs — this satisfies kustomize's default LoadRestrictionsRootOnly constraint (the overlay has no ../ references) without requiring --load-restrictor=LoadRestrictionsNone.

Local validation (make stage-prometheus-dashboard). The staged deploy/kind/prometheus/keystone-operator.json is git-ignored — the canonical file lives only at operators/keystone/dashboards/keystone-operator.json. Developers who want to run kustomize build deploy/kind/prometheus/, kubectl apply -k deploy/kind/prometheus/, or chainsaw lint against the overlay without running WITH_PROMETHEUS=true make deploy-infra first must stage the dashboard manually:

make stage-prometheus-dashboard

The target performs the same cp -f that hack/deploy-infra.sh runs at deploy time, so local renders match CI exactly. make deploy-infra re-runs the copy on every invocation, so explicit staging is not needed when going through the full deploy path.

ServiceMonitor enablement. The keystone-operator chart defaults to monitoring.serviceMonitor.enabled=false so production overlays inherit the safe default. When WITH_PROMETHEUS=true, hack/deploy-infra.sh waits for the kube-prometheus-stack HelmRelease to become Ready, then runs:

kubectl patch helmrelease keystone-operator -n keystone-system --type=merge \
  -p '{"spec":{"values":{"monitoring":{"serviceMonitor":{"enabled":true}}}}}'

…and waits for the keystone-operator HelmRelease to reconcile back to Ready=True on the new values. The patch is only applied when WITH_PROMETHEUS=true — the chart values themselves are never modified, which keeps the production posture unchanged.

Opt-in usage:

WITH_PROMETHEUS=true make deploy-infra

This is the prerequisite for make e2e-prometheus (see CI / e2e-prometheus job for the workflow). For the kind UI walkthrough — port-forward, default Grafana credentials, the bundled Keystone Operator dashboard, and a Prometheus targets sanity-check — see Extended Quick Start — Step 4c.

Posture summary. Reviewers checking new kind-only opt-ins should treat this entry as a parallel of the Chaos Mesh (kind-only opt-in) example above: the production omission is explicit, the opt-in flag has a single documented name (WITH_PROMETHEUS), and the kind overlay is self-contained under deploy/kind/prometheus/ so the production kustomization root is untouched. The document-intentional-environment-divergence-in-overlays review pattern catalogues the full surface area.

The metal-stack lab runs the same stack with its metrics on a volume, the scrape jobs a Gardener shoot can feed, and a dashboard for the hypervisor operator; see Lab Prometheus stack.

metrics-server (kind-only opt-in)

File: deploy/kind/metrics-server/kustomization.yaml

metrics-server ships as a separate opt-in kind overlay. The default make deploy-infra flow does not install it — the kube-system metrics-server stays absent so the default Quick Start does not spend the kind node's budget on a component most tutorials do not need. The production deploy/flux-system/ overlay also does not install it: managed distributions ship their own metrics-server, and production clusters bring their own.

The overlay backs the Autoscaling (HPA) recipe and the e2e-autoscaling CI job, which deploys it with WITH_METRICS_SERVER=true to drive a Keystone HPA from one pod to its maximum (see e2e-autoscaling): the operator-generated HorizontalPodAutoscaler reads CPU/memory utilisation from the resource-metrics API, and without a metrics-server it reports unknown/80% and never scales.

The overlay is self-contained: the HelmRepository and HelmRelease live in deploy/kind/metrics-server/source.yaml and deploy/kind/metrics-server/release.yaml. Unlike the chaos-mesh and kube-prometheus-stack overlays it ships no Namespace — the chart defaults to priorityClassName: system-cluster-critical, which only resolves in the pre-existing kube-system Namespace, so the HelmRelease targets kube-system directly.

Property	Value
Target namespace	kube-system (pre-existing; no inline Namespace)
Chart	metrics-server
Version constraint	>=3.12.0 <4.0.0
Source	metrics-server HelmRepository (https://kubernetes-sigs.github.io/metrics-server/)
Dependencies	none

Kind-tuned values:

Helm value	Override	Purpose
args	["--kubelet-insecure-tls"]	kind's kubelets serve /metrics/resource with a self-signed certificate metrics-server cannot verify against the cluster CA; skipping verification lets scrapes succeed. This replaces the runtime kubectl patch the autoscaling recipe previously documented — never set it on a production cluster with properly issued kubelet certificates

When WITH_METRICS_SERVER=true, hack/deploy-infra.sh runs kubectl apply -k deploy/kind/metrics-server in Step 3 and appends metrics-server to the Phase 3 HelmRelease wait list. Both actions are gated strictly on the flag; the chart values are never modified, which keeps the production posture unchanged.

Opt-in usage:

WITH_METRICS_SERVER=true make deploy-infra

Posture summary. Same shape as the two entries above: the production omission is explicit, the opt-in flag has a single documented name (WITH_METRICS_SERVER), and the kind overlay is self-contained under deploy/kind/metrics-server/ so the production kustomization root is untouched.

VPA recommender (kind-only opt-in)

File: deploy/kind/vpa/kustomization.yaml

The VerticalPodAutoscaler recommender ships as a separate opt-in kind overlay. Neither the default make deploy-infra flow nor the production deploy/flux-system/ overlay installs it.

It has two consumers. The first is the sizing measurement. hack/ci-vpa-recommendations.sh creates a VPA with updateMode: "Off" for every workload in the openstack namespace and records the recommender's CPU and memory targets. The ci:measure-sizing label runs that measurement in CI (see the CI workflow label table).

The second is the e2e-autoscaling job, which always deploys the overlay. Its operators find the VerticalPodAutoscaler CRD at startup and create the VPAs that the ControlPlane's and the OVNCentral's verticalAutoscaling blocks opt into, and the e2e-autoscaling suite waits for the recommender to serve one of them. Without the updater and the admission controller, an opt-in with updateMode Initial, Recreate, InPlaceOrRecreate or Auto changes no pod on this overlay either, so the suite asserts the VPA objects and a recommendation, not rewritten requests.

Recommendation-only mode changes no pod, so the release installs the VPA CRDs and the recommender alone: the updater and the admission controller are disabled. At one replica the chart turns leader election off. The overlay ships no Namespace; the HelmRelease targets the pre-existing kube-system Namespace, and the HelmRepository lives in flux-system.

Property	Value
Target namespace	kube-system (pre-existing; no inline Namespace)
Chart	vertical-pod-autoscaler
Version constraint	>=0.13.0 <0.14.0 (0.13.0 ships VPA 1.8.0; the minor bound keeps the recommender's behaviour fixed between measurements)
Source	autoscaler HelmRepository (https://kubernetes.github.io/autoscaler)
Dependencies	metrics-server (the recommender reads the resource-metrics API)

Kind-tuned values:

Helm value	Override	Purpose
admissionController.enabled	false	No VPA of the measurement rewrites pod requests
updater.enabled	false	No VPA of the measurement evicts pods
recommender.replicas	1	One recommender; the chart disables leader election
recommender.extraArgs	--pod-recommendation-min-cpu-millicores=1, --pod-recommendation-min-memory-mb=1	The recommender divides a per-pod floor (25 millicores and 250 MiB by default) evenly among a pod's containers. With the default floor a container using less than its share reports the floor instead of its use
recommender.resources	requests 50m / 256Mi, limit 512Mi memory	Keeps the recommender itself small on the measured node
crds.enabled	true	Installs verticalpodautoscalers.autoscaling.k8s.io, which the collector checks for and the operators need to create opt-in VPAs

--memory-saver keeps its default false, so the recommender samples every pod from its start, whether or not a VPA selects it yet.

When WITH_VPA=true, hack/deploy-infra.sh sets WITH_METRICS_SERVER=true, runs kubectl apply -k deploy/kind/vpa in Step 3 right after the metrics-server overlay, and appends vertical-pod-autoscaler to the Phase 3 HelmRelease wait list. The banner prints the flag as VPA recommender. Once the MariaDB CRD is established, the script also removes the scale subresource from mariadbs.k8s.mariadb.com. The recommender only accepts a VPA on the topmost well-known or scalable controller, and the MariaDB scale subresource has no pod selector, so without the removal the database StatefulSet cannot be measured. The mariadb-operator-crds HelmRelease has no drift detection, so the change lasts until the next chart upgrade, and nothing in the kind stack scales a MariaDB through the subresource.

On an external cluster, which refuses WITH_VPA=true, hack/ci-vpa-recommendations.sh prepare removes the subresource for the lab measurement.

Opt-in usage:

WITH_VPA=true make deploy-infra

dizzy load/chaos stack (kind-only opt-in)

File: deploy/kind/dizzy/kustomization.yaml

dizzy is a scenario-driven load and consistency tester for OpenStack control planes. Its VictoriaMetrics + Grafana observability stack ships as a separate opt-in kind overlay. The default make deploy-infra flow does not install it — the dizzy namespace stays absent, and neither VictoriaMetrics nor Grafana runs unless a contributor opts in. The production deploy/flux-system/ overlay ships none of it.

The overlay is self-contained, seven tracked files under deploy/kind/dizzy/:

File	Purpose
namespace.yaml	The dizzy Namespace, declared inline so the overlay is self-contained
source-victoria-metrics.yaml	victoria-metrics HelmRepository
source-grafana.yaml	grafana HelmRepository
release-victoria-metrics.yaml	dizzy-victoria-metrics HelmRelease (chart victoria-metrics-single, NodePort 30428, emptyDir storage, 30d retention, OTLP ingest at /opentelemetry/v1/metrics)
release-grafana.yaml	dizzy-grafana HelmRelease (anonymous Viewer access, provisioned victoriametrics datasource, dizzy Overview home dashboard)
httproute.yaml	The static dizzy-grafana HTTPRoute attaching to the https-dizzy listener
kustomization.yaml	Ties the resources together and generates the dashboard ConfigMap

Both HelmRepositories (victoria-metrics, grafana) are declared into the flux-system namespace, where Flux resolves HelmRelease sourceRefs, even though the two source files live inside the kind-only overlay. That keeps deploy/flux-system/** untouched while the overlay stays self-contained.

Dashboard staging (hack/dizzy.sh stage-dashboards). The overlay's configMapGenerator wraps three dizzy dashboard JSONs, but deploy/kind/dizzy/dashboards/ is git-ignored; the dashboards are a version-pinned dizzy asset staged from the release tarball, so the tree tracks none of them. hack/dizzy.sh stage-dashboards copies them out of that tarball (cached under _output/dizzy/<version>/) before kubectl apply -k runs. WITH_DIZZY=true make deploy-infra performs the staging automatically; a raw kustomize build deploy/kind/dizzy on a fresh checkout fails until stage-dashboards has populated the directory.

Gateway wiring. Two pieces are present even without WITH_DIZZY: the https-dizzy listener on Gateway openstack-gw (deploy/kind/base/openstack-gateway.yaml) and the dizzy-nip-io-tls Certificate (deploy/kind/infrastructure/dizzy-nip-io-tls-certificate.yaml). The listener admits routes from the dizzy namespace through a namespace selector on the automatic kubernetes.io/metadata.name label, so no ReferenceGrant is required. This is the repo's first cross-namespace-admitting listener. The dizzy-grafana HTTPRoute itself ships only in the gated overlay; without it the dizzy.127-0-0-1.nip.io hostname answers 404.

Host port mapping. hack/kind-config.yaml maps host 127.0.0.1:8428 to the node's containerPort 30428, bridging host-side OTLP export to the VictoriaMetrics NodePort. A cluster created before this mapping existed keeps working, but the tooling's port probe warns; recreate the cluster (make teardown-infra && WITH_DIZZY=true make deploy-infra) to pick it up.

Opt-in usage:

WITH_DIZZY=true make deploy-infra

To drive the chaos soak against the ControlPlane, see dizzy Chaos Testing. The metal-stack lab runs the same stack with its metrics on a volume; see Lab dizzy stack.

Posture summary. Same shape as the entries above: the production omission is explicit, the opt-in flag has a single documented name (WITH_DIZZY), and the kind overlay is self-contained under deploy/kind/dizzy/ so the production kustomization root ships none of it.

NFS storage stack (opt-in)

File: deploy/kind/nfs/kustomization.yaml

The NFS storage stack, an in-cluster NFSv4 server plus the csi-driver-nfs mounter, ships as a separate opt-in kind overlay for the Cinder e2e suites of #979. The default make deploy-infra flow does not install it. Unlike the other opt-ins of this section it is not kind-only: the metal-stack lab builds its Lab NFS stack on this overlay. Production omits it for two reasons: users bring their own NFS server (a CinderBackend takes a server and a path per backend), and the mounter is a prerequisite of the target cluster. Shipping csi-driver-nfs from deploy/flux-system/releases/ would put a privileged hostNetwork DaemonSet into every production deployment for a service no production cluster runs yet.

One more reason keeps it opt-in on kind. The server, NFS-Ganesha, runs in a privileged container, and on a Linux host the deploy script loads nfs and nfsv4 for the mounter through sudo. An always-on manifest would put a privileged pod into the default Quick Start and ask every run for sudo.

The overlay is self-contained: source.yaml, release.yaml and nfs-server.yaml are all local to deploy/kind/nfs/. It ships noNamespace. The HelmRelease targets the pre-existing kube-system, because the chart defaults to priorityClassName: system-cluster-critical, which resolves in no other Namespace. The server lands in the pre-existing openstack, which deploy/flux-system/namespaces.yaml reserves for components that exist only to run the operators standalone. shared-services is not used: it is the trust zone holding openbao-init-keys, with Garage as its one accepted co-tenant.

Property	Value
Target namespaces	kube-system for the chart, openstack for the server (both pre-existing; no inline Namespace)
Chart	csi-driver-nfs
Chart version	4.13.4, an exact pin tracked by Renovate's flux manager (majors disabled, no automerge). Unlike the chaos-mesh, metrics-server and dizzy overlays this one carries no >=x <y range: a range would let Flux adopt a new chart on its next reconcile with no repo diff, and this release installs a privileged hostNetwork DaemonSet whose relied-on chart defaults it does not override
Source	csi-driver-nfs HelmRepository (https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts). That is the only place upstream publishes the chart; every index entry carries an absolute tarball URL back under master/charts/, so pinning the repository URL to a tag would freeze the index without making the downloaded chart more immutable. The version pin therefore controls which version Flux installs, not which bytes: Flux spec.verify is OCI-only and nothing records a checksum, so a rewrite of the pinned tarball upstream is adopted on the next reconcile. Accepted for the kind overlay and for the lab, which takes the release unchanged; a content pin means mirroring the chart into a registry this project controls and referencing it by digest
Server image	ghcr.io/kubernetes-sigs/nfs-ganesha:V6.5, NFS-Ganesha 6.5 on Fedora 41 with amd64 and arm64 manifests, from nfs-ganesha-server-and-external-provisioner. Digest-pinned and tracked by a Renovate customManager whose regex versioning reads the V-prefixed tags (majors disabled, no automerge)
Dependencies	none

Export layout. The server exports /exports with Pseudo = /, which makes it the NFSv4 pseudo-root. In the pod, /exports is the directory exports/ of the claim. The init container prepare-exports creates exports/volumes and exports/backups there as 42424:42424 with mode 0770. A client therefore mounts the two shares as:

nfs-server.openstack.svc.cluster.local:/volumes
nfs-server.openstack.svc.cluster.local:/backups

A path of :/exports/volumes does not exist on the server and fails. Those two share strings are what the CinderBackend and CinderBackupBackend of #979 carry. The server speaks NFSv4.1 and 4.2 over TCP alone (Protocols = 4, Minor_Versions = 1, 2, Transports = TCP), so 2049/TCP is the whole client surface and the Service exposes neither 111 nor a mountd port. Every mount in the tree asks for 4.1 or 4.2. A 4.0 client sends no RECLAIM_COMPLETE, so it would hold every restart of the server in its grace period for the full 90 seconds. Attr_Expiration_Time = 0 turns Ganesha's attribute cache off: a chown made with kubectl exec inside the server pod reached a client after 4 seconds, where the default of 60 took 94. Dir_Chunk = 0 turns its cache of directory entries off. The cinder-nfs-outage and multi-backend suites and the NFS backend guide remove a file or a directory of the export inside the server pod, behind Ganesha. With the cache on, every later listing of that directory failed with Stale file handle for every client, also on a fresh mount, and so did a mount of a directory that was removed and created again.

Client records. After a server restart an NFSv4 client reclaims its opens and locks, and the server admits a reclaim only from a client it recorded before. QEMU holds a lock on the file of every attached volume, so a refused reclaim leaves the guest's disk failing every request until its Nova server is hard-rebooted (#1245). The kernel's nfsd in a pod keeps no records. Ganesha writes them with RecoveryBackend = fs below RecoveryRoot = /var/lib/nfs/ganesha, a directory of the export claim, so they outlive the pod, also when it moves to another node. It announces the fixed server scope Server_Scope = "nfs-server.openstack". A server announces its hostname by default, a Deployment pod gets a new one at every restart, and the Linux client does not reclaim from a server whose scope changed. The pod spec sets no hostname.

The claim holds exports/ and ganesha/. The server container mounts it twice by subPath, exports at /exports and ganesha at /var/lib/nfs/ganesha, so a client that mounts the export does not see the records, and /exports stays the path a kubectl exec into the pod names. A forged file handle does reach the records, as the posture summary below states. prepare-exports mounts the claim root at /claim, creates both directories, and moves every other entry of the root but lost+found into exports/, printing prepare-exports: moved <name> into exports/ for each. A claim the kernel server wrote, with volumes/ and backups/ at its root, keeps its volume files that way. A name present on both levels stops the container with exit 1 and prepare-exports: /claim/<name> and /claim/exports/<name> both exist; move one of them away, unless it is an empty directory, which the container removes, and a second start moves nothing. The move keeps the files. A client that mounted the kernel server still holds file handles Ganesha does not know, so a stack that ran the kernel server switches only when it is redeployed from a bare cluster, or after every Nova server with a volume attached has had it detached and the cinder-volume and cinder-backup pods are deleted once Ganesha has rolled out: they mount both shares for as long as they run.

A deploy from a tree older than #1245, a revert or a branch that predates it, runs the kernel server against the claim root. It serves an empty volumes/ and backups/ there and leaves its writes at the root, while every existing volume and backup file stays unseen in exports/. The next Ganesha start removes the two directories while they are empty and stops on the conflict line once they hold a file. Delete the claim or rebase the branch before such a deploy.

The server's log is kubectl logs -n openstack deployment/nfs-server -c nfs-server. At start Ganesha logs CRIT lines about D-Bus and Cannot acquire credentials for principal nfs, and Cannot register NFS V4 on UDP. They are expected: the pod runs no D-Bus, no Kerberos and no rpcbind. With FSAL = INFO it logs Added filesystem ... /exports ... fsid=<id>, the id every file handle of the export carries, and Root fs for export /exports is /exports, which the nfs-health suite requires. A restart logs NFS Server Now IN GRACE and, once every recorded client has reclaimed or the 90 seconds are over, NFS Server Now NOT IN GRACE.

The server container requests 640Mi and is limited to 1Gi. Ganesha does the I/O in its own process. On the metal-stack lab it idled at 49 MiB, and its working set peaked at 547 MiB while four guests rewrote a volume of 2 GiB each for 300 seconds; the request is that peak plus 15 %, rounded up to a multiple of 64Mi (Lab NFS stack has the readings). Under writes the container's usage stays at the limit, because the page cache of the files Ganesha writes is charged to it. The kernel reclaims that cache at the limit, and no reading shows an OOM kill. In a local run before the lab reading, Ganesha peaked at 397 MiB resident under a 3 GiB write and four parallel 1 GiB writers, and a 256Mi limit got it OOM-killed.

Value overrides:

Helm value	Override	Purpose
controller.enableSnapshotter	false	The chart defaults it to true while externalSnapshotter.enabled defaults to false, so the csi-snapshotter sidecar would be deployed against snapshot.storage.k8s.io CRDs nothing in this stack installs. The Cinder NFS backend has snapshots off
storageClass.create	false	Already the chart default, set here with the reason: the cinder-operator mounts inline volumes, so no dynamic class is wanted, and an unwanted class in a kind cluster competes for is-default-class
feature.enableInlineVolume	true	The cinder-operator mounts every backend and backup share as an inline csi: volume, and the driver serves such a volume only when its CSIDriver lists the Ephemeral lifecycle mode. The chart adds that mode behind this flag. CSIDriver.spec.volumeLifecycleModes is immutable, so a kind cluster created before this value was set carries a CSIDriver the chart cannot patch; hack/deploy-infra.sh deletes that object before it applies the overlay and the chart recreates it (see below). A fresh cluster needs nothing

Everything else stays at the chart default: driver.name: nfs.csi.k8s.io, attachRequired: false, fsGroupPolicy: File and kubeletDir: /var/lib/kubelet.

When WITH_NFS=true, hack/deploy-infra.sh does four things. In kind mode it loads nfs and nfsv4, the modules the csi-driver-nfs node plugin mounts with, on the host before the cluster is created, best-effort through the same loader as WITH_OVN_KERNEL_MODULES (Linux only, root or passwordless sudo, otherwise a warning). The server needs no module. Under EXTERNAL_CLUSTER=true it loads nothing on the host and applies the overlay's nfs/ in place of deploy/kind/nfs, whose pods load the modules on the nodes; see Lab NFS stack. On a cluster whose CSIDriver/nfs.csi.k8s.io lists no Ephemeral lifecycle mode it deletes that object, because the field is immutable and the chart's patch would otherwise be rejected for the lifetime of the cluster: a reused cluster (a second run, or a runner keeping a warm one with SKIP_KIND_CREATE=true) would fail the csi-driver-nfs wait below with the cause buried in the HelmRelease status. Only a NotFound counts as "no such object" there; any other failed read (an API server still settling, a denied cluster-scoped read) aborts the run rather than skipping the check. A delete is followed by a forced csi-driver-nfs reconcile with the release's failure counts reset and a hard wait for the object to reappear, because the re-applied overlay can be identical to what the cluster already carries and helm-controller retries neither an unchanged release nor an upgrade whose remediation retries are spent: a cluster left with no NFS CSI driver at all is worse than the one the delete started from. A NotFound on a cluster that already carries the csi-driver-nfs HelmRelease enters that same recreate path, because it is not a fresh cluster but one an earlier run left driverless when it died between the delete and the forced reconcile. The wait is on the recreated object's spec.volumeLifecycleModes, not only on its existence: a remediation rollback racing the delete puts the pre-Ephemeral object back, which an existence-only check would accept. In kind mode it applies deploy/kind/nfs in Step 3 and waits for the nfs-server Deployment to roll out; a failed rollout is an error that stops the run and names the logs of prepare-exports and nfs-server, because a server that does not come up must not end in a green summary. It appends csi-driver-nfs to the Phase 3 HelmRelease wait list. All four actions are gated strictly on the flag; the default run is unchanged.

Opt-in usage:

WITH_NFS=true make deploy-infra

The nfs-health suite in E2E Deployment is the check that both shares mount under restricted PodSecurity.

Posture summary. Same shape as the entries above: the production omission is explicit, the opt-in flag has a single documented name (WITH_NFS), and the kind overlay is self-contained under deploy/kind/nfs/. The non-production posture is recorded in the header of nfs-server.yaml: a privileged server, because Ganesha reads the filesystem UUID behind its file handles from the device node, sec=sys with No_Root_Squash and no client list, and a third-party image pinned by digest. A client that dials 2049 reaches more than both shares: Ganesha checks a file handle against the filesystem alone, not against the export path, so a forged handle reaches every file of the filesystem that holds the claim, as root. On the lab that is the claim's volume with the client records in ganesha/, on kind the host disk behind the kind node's /var. The kernel server's no_subtree_check export reached as far. The Ephemeral lifecycle mode widens that posture by one step: reaching the export no longer needs a cluster-scoped PersistentVolume, so anyone who can create a Pod mounts both shares from the pod spec alone. PodSecurity does not bound that. In a namespace that is not restricted the pod mounts them as root. A restricted namespace admits a csi volume and the UID 42424, the owner of both exports, which is how the nfs-health probe mounts them. The metal-stack lab carries the same posture, as Lab NFS stack states.

Message bus (kind-only opt-in)

Files: deploy/kind/messaging/kustomization.yaml, deploy/kind/messaging/shared-rabbitmq.yaml

One RabbitmqCluster named shared-rabbitmq in openstack, for the standalone Cinder e2e suites of #988 and, later, the ControlPlane suites of #989. The RabbitMQ Cluster Operator that reconciles it is not part of the overlay: it reaches every cluster this script provisions through the Flux Kustomization in deploy/flux-system/releases/rabbitmq-cluster-operator.yaml. Only the broker instance is opt-in, and the default make deploy-infra flow creates none.

Production ships no equivalent object. A production ControlPlane declares spec.infrastructure.messaging, and the c5c3 operator projects a RabbitmqCluster for it into the ControlPlane's own namespace.

Property	Value
Target namespace	openstack (pre-existing; the overlay ships no inline Namespace)
API version	rabbitmq.com/v1beta1
Replicas	1
Requests	100m CPU, 1Gi memory
Limits	1Gi memory, no CPU limit
Dependencies	the RabbitMQ Cluster Operator (Phase 3b) and the rabbitmqclusters.rabbitmq.com CRD (the Step 5 wait_for_crds list)

Sizing. The cluster operator requests 1 CPU and 2Gi per pod by default. That request does not fit beside the rest of the stack on the self-hosted runner's kind node: it takes the last schedulable CPU and the next pod stays Pending on Insufficient cpu. tests/e2e-chaos/neutron-broker-outage records the same finding for its own broker. The e2e suites push little traffic through the bus, so 100m CPU carries them. The memory figure follows from the broker's alarm: the cluster operator alarms the broker at 0.6 of the limit minus a fifth, so 512Mi alarmed at 245.8 MiB against an idle footprint of about 236 MiB (#1298) and 1Gi alarms at 491.5 MiB. Memory is limited at the request. There is no CPU limit, so a busy moment goes unthrottled.

One vhost per suite. Every standalone Cinder e2e suite creates its own vhost, named after the Cinder CR, plus a Secret <cr-name>-messaging holding the matching transport_url. tests/e2e/cinder/broker-vhost.sh creates both. Two suites on one vhost would share the RPC topics cinder-scheduler and cinder-volume.<host>@<backend>, so a scheduler in one suite could hand a volume to the volume service of another. Managed mode (spec.messaging.clusterRef) always lands on the default vhost, so it is used only where a single Cinder runs alone on the cluster: the tempest legs.

The c5c3 full-controlplane-keystone suite is the broker's second consumer. It calls the same helper, naming its vhost after the ControlPlane rather than after a Cinder CR, and the ControlPlane copies the transport URL out of the resulting controlplane-keystone-messaging Secret into both of its bus-consuming children, Neutron and Cinder.

Deploy-infra wiring. With WITH_MESSAGING=true, hack/deploy-infra.sh applies deploy/kind/messaging after Step 5, where both prerequisites are settled: the openstack namespace from the Step 3 base overlay and the rabbitmqclusters.rabbitmq.com CRD from the Phase 3b operator wait. It then waits for rabbitmqcluster/shared-rabbitmq to report AllReplicasReady. The cluster operator publishes no Ready condition, so that is the condition to gate on. On timeout the run stops: it prints kubectl describe rabbitmqcluster/shared-rabbitmq and the events of shared-rabbitmq-server-0, then exits 1.

Opt-in usage:

WITH_MESSAGING=true make deploy-infra

Posture summary. Same shape as the entries above: the production omission is explicit, the opt-in flag has a single documented name (WITH_MESSAGING), and the kind overlay is self-contained under deploy/kind/messaging/.

Glance large-upload listener

Files: deploy/kind/base/openstack-gateway.yaml, deploy/kind/infrastructure/glance-upload-nip-io-tls-certificate.yaml

Gateway wiring. Gateway openstack-gw carries a fifth HTTPS listener, https-glance-upload, on the hostname glance-upload.127-0-0-1.nip.io. It shares port 443 with the keystone, horizon, glance, and dizzy listeners, routed by SNI, and terminates with its own glance-upload-nip-io-tls Certificate from the selfsigned-cluster-issuer ClusterIssuer — the per-listener pattern the four sibling nip.io certificates already follow. Unlike them it serves a test suite. tests/e2e/glance/gateway-large-upload streams 512 MiB through the public endpoint, while the Quick Start smoke suite already holds glance.127-0-0-1.nip.io; two HTTPRoutes on one hostname with the same / path prefix would race under chainsaw's parallelism, so the suite gets a hostname of its own.

The listener and its Certificate are unconditional; the HTTPRoute is not. Like the https-dizzy listener above, the hostname answers 404 until a route is projected onto it. The Glance operator creates that route from spec.gateway on the Glance CR the suite applies, and deletes it again when the suite cleans up.

Metal-stack lab

deploy/lab/metal-stack/ holds the manifests of the CobaltCore lab on a metal-stack cluster, planned in #1138. deploy/flux-system/kustomization.yaml does not reference the tree. hack/deploy-infra.sh applies its base/ and infrastructure/ under EXTERNAL_CLUSTER=true (see Lab overlay), its ceph/ and ceph/cluster/ when WITH_CEPH=true is set (see Lab Ceph), its nfs/ when WITH_NFS=true is set (see Lab NFS stack), its chaos-mesh/ when WITH_CHAOS_MESH=true is set (see Lab Chaos Mesh), its dizzy/ when WITH_DIZZY=true is set (see Lab dizzy stack), and its prometheus/ when WITH_PROMETHEUS=true is set (see Lab Prometheus stack); make dizzy-soak-start applies its dizzy-soak/ (see Lab dizzy soak); the probe is applied by hand, and so is controlplane/, once the deploy has finished (see Lab ControlPlane), and after it hypervisor-fixtures/ and hypervisor/ (see Lab hypervisors); on a lab deployed with WITH_NFS=true as well, Part 3 of the quick start applies controlplane-nfs/ by hand after those (see Lab ControlPlane). The ControlPlane's opt-in autoscaling blocks use the VPA and the metrics-server of the platform (see Lab autoscaling). The Quick Start (metal-stack) is the walkthrough that runs them in order, from a bare cluster to a migrated server and back.

The figure shows what those manifests add up to: the cluster-wide parts of base/, infrastructure/, ceph/ and controlplane/, and the parts hypervisor/ and migration-ports/ put on every worker.

Node probe

File: deploy/lab/metal-stack/probe/kustomization.yaml

The node probe is the prerequisite check of the lab. The Quick Start (metal-stack) has a reader run it first, against any metal-stack cluster, before anything else is deployed. It is one Job, node-probe, that prints the node facts the lab depends on under thirteen fixed headers. It exits 0 whatever it finds: a node that lacks something prints absent, none or NOT FOUND, and the Job still completes, so kubectl wait --for=condition=complete returns.

The container runs privileged, because only a privileged container sees the host's /dev/kvm. The host root is mounted read-only at /host, with recursiveReadOnly: IfPossible keeping its submounts read-only where the runtime supports it. The pod shares no host PID or network namespace and mounts no ServiceAccount token. The script loads no module, writes nothing and installs nothing, so the probe needs no egress beyond the image pull. tests/unit/deploy/metal_stack_probe_test.sh fails when an edit adds nsenter, chroot, modprobe, insmod, rmmod, mount, umount, sysctl, apt-get, apt, tee, dd, mknod or rm to the script, or a redirection other than 2>/dev/null.

kubectl apply -k deploy/lab/metal-stack/probe
kubectl wait --for=condition=complete job/node-probe -n default --timeout=5m
kubectl logs -n default job/node-probe
kubectl delete job -n default node-probe

The Job runs on the node the scheduler picks. On a cluster with more than one worker, pin it to each node in turn, with the same wait, logs and delete after every run. The delete is needed because a Job's pod template is immutable:

yq '.spec.template.spec.nodeName = "<node>"' deploy/lab/metal-stack/probe/node-probe.yaml | kubectl apply -f -

ttlSecondsAfterFinished: 3600 removes the Job, and its logs with it, an hour after it completes. A DaemonSet would cover every node in one run, but it never completes and has to be deleted by hand; the Job keeps each run bounded.

Property	Value
Target namespace	default. The probe runs before deploy/flux-system/namespaces.yaml has created openstack or shared-services, and the metal-stack shoot enforces no PodSecurity level on default
Image	docker.io/library/debian:bookworm-slim, digest-pinned. A Renovate customManager refreshes the digest (automerged after 3 days); Renovate never moves the bookworm-slim codename tag, so a Debian release change is a manual edit
Runs as	one Job with backoffLimit: 0, one privileged container, the host root read-only at /host
Dependencies	none

Header	What it answers for the lab
== kvm device	Whether the node has /dev/kvm. With == cpu it decides virtType: kvm
== cpu	CPU model, count and topology, the virtualization extension, and how many CPUs carry the vmx or svm flag
== loaded modules	Which KVM, vhost, Open vSwitch, Geneve, VXLAN, bridge, NBD, multipath, NVMe/TCP and NFS modules are loaded: the NFS server (nfsd), the NFS client (nfs, nfsv4) and sunrpc. It also lists the modules Chaos Mesh NetworkChaos uses: ip_set with every ip_set_* type module, which shows the ipset types calico-node has loaded, xt_set, sch_netem and sch_tbf
== module files for <kernel>	Whether the running kernel ships kvm, vhost_net, openvswitch, geneve and the other module files, so the OVN chassis and the libvirt DaemonSet can load what they need. The five NFS files, nfsd, nfs, nfsv4, lockd and sunrpc, decide whether the NFS server and clients of #1193 can use the node's kernel. The six Chaos Mesh files, ip_set, ip_set_hash_ip, ip_set_hash_net, xt_set, sch_netem and sch_tbf, decide D3 of #1219: whether a pod can load the modules NetworkChaos needs, and which ones it has to load. A module compiled into the kernel prints builtin
== filesystems	Whether the kernel has registered nfs4, the filesystem type of an NFSv4 mount, and nfsd, the NFS server's control filesystem. Each prints registered or not registered, whether the code is a loaded module or compiled into the kernel. The nfs module registers nfs4, not nfsv4: only the nfsv4 module file, or the load test's nfsv4: line, shows that the NFSv4 client code is there
== nested / iommu	The nested parameter of kvm_intel or kvm_amd, and the number of IOMMU groups
== memory	MemTotal and the hugepage reservations
== disks	The block devices, where /var/lib lives and how much it holds
== cgroup	The cgroup filesystem type, cgroup2fs on cgroup v2
== host os / binaries	The host OS, and that no libvirtd, qemu-system-x86_64, ovs-vswitchd or rpc.nfsd is installed on the host
== containerd socket	Where the containerd socket the Chaos Mesh daemon mounts lives. Two lines test containerd's default path, run/containerd/containerd.sock, and the k3s one, run/k3s/containerd/containerd.sock, under the host's /run; each prints socket, present, not a socket or absent. The third line prints the address of the [grpc] table in the host's /etc/containerd/config.toml, quotes included and a trailing comment dropped. not set means the probe read no address in a [grpc] table: the file is missing, is an absolute symlink, which resolves inside the pod, or sets no such key. An imports file or containerd's --address flag can still move the socket, so the two socket lines are the evidence
== nics	Every host interface with its MTU and state: the uplinks, and the host end of each pod's veth (cali*), which carries the pod network's MTU. Neutron's global_physnet_mtu must not exceed the MTU of the network the Geneve tunnels run on (see Lab ControlPlane)
== rook	Whether /var/lib/rook exists on the node, where the Lab Ceph keeps its state: /var/lib/rook: absent, or present followed by the directory's entries and the df -h lines of its filesystem. present on a bare cluster is the leftover of a Ceph that was not torn down with its cleanup policy, and a new monitor refuses to start over another cluster's data there. The probe mounts no ServiceAccount token and cannot see a StorageClass, so whether the default class serves volumeMode: Block claims shows in the deploy: Step 1 logs the default class, and the Ceph wait fails when the OSD claims stay Pending

The values a lab-ready node shows come from the 2026-09-29 survey in #1138. The header comment of deploy/lab/metal-stack/probe/node-probe.yaml lists them in the probe's own output format, from a run on the survey's node on 2026-10-04 that includes the NFS lines, the Chaos Mesh module lines and the containerd socket lines. The survey's NIC lines show the pod's own eth0; the probe reads the host's sysfs and lists the host's interfaces instead.

On the lab, the probe found the six Chaos Mesh module files on both workers on 2026-10-04, each with a path under /lib/modules/6.1.0-49-amd64. So D3 of #1219 stands: the DaemonSet chaos-mesh-modules of the Lab Chaos Mesh loads ip_set, ip_set_hash_ip, ip_set_hash_net, xt_set, sch_netem and sch_tbf on every node. Loaded at the time of the run were xt_set, ip_set_hash_ip, ip_set_hash_net and ip_set. This does not shorten the list: a reboot or a replaced node starts without them. Both workers printed run/containerd/containerd.sock: socket, so the lab's chaos-daemon mounts /run/containerd/containerd.sock, the kind overlay's value. The comment on #1219 holds the output of both workers.

File: deploy/lab/metal-stack/probe/nfs-module-load.yaml

The NFS module load test answers the one question of #1193 the probe cannot: whether a pod can load the NFS modules on a lab worker. It is one Job, nfs-module-load in default, that loads nfsd, nfs and nfsv4 with modprobe, prints each result and the nfs4 and nfsd filesystem lines, and removes what it loaded. A failed load prints <module>: FAILED: with modprobe's message. The Job exits 0 whatever it finds, so the result is read from the log.

Unlike the probe, the load test is not read-only. It loads up to three modules and their dependencies into the node's kernel and removes them again with rmmod. It removes the modules its own load added: those that are new since a /proc/modules snapshot taken before the load and are one of nfsd, nfs and nfsv4 it tried to load or one of their dependencies, as modprobe --show-depends lists them. A module that prints already loaded is not tried: one loaded before the run stays, and so does one another pod loaded since the snapshot. A module outside that list, such as vhost_net, is neither counted nor removed. Any other NFS module that another pod loads in the seconds between the snapshot and the unload can be taken for the run's own, so run the test on a node where nothing else loads NFS modules. A module it cannot remove stays loaded until the node reboots and is named on the last line, still loaded: ...; otherwise the last line is module list as before. The container mirrors the init container host-prepare of the lab hypervisors, which loads vhost_net the same way: the DaemonSet's pinned ghcr.io/c5c3/libvirt:<tag>@sha256:<digest>, pulled IfNotPresent, privileged, as root, with the node's /lib/modules mounted read-only as its one volume. Renovate moves the reference in the pull request that moves the DaemonSet's.

The probe's kustomization.yaml leaves the file out of resources, so kubectl apply -k deploy/lab/metal-stack/probe stays read-only. The load test is applied by file, pinned to one node:

kubectl delete job -n default nfs-module-load --ignore-not-found
yq '.spec.template.spec.nodeName = "<node>"' deploy/lab/metal-stack/probe/nfs-module-load.yaml | kubectl apply -f -
kubectl wait --for=condition=complete job/nfs-module-load -n default --timeout=5m
kubectl logs -n default job/nfs-module-load
kubectl delete job -n default nfs-module-load

On the lab, the probe found the five NFS module files on both workers on 2026-10-03, and the load test on shoot--df33f0b4c1--forge-group-0-666b6-qmhrg loaded all three modules, registered nfs4 and nfsd, and removed the eleven modules the load added. That settled the two open decisions of #1193: the NFS server is the kernel's nfsd in a privileged pod, as on kind (D1), and a privileged init container loads the modules from the node's /lib/modules (D2). The comment on #1193 holds the output of both runs; the header comment of nfs-module-load.yaml carries the load test's. #1245 replaced the kernel's nfsd with NFS-Ganesha, a userspace server, so the server needs no module, and the probe's nfsd lines describe the node alone.

Lab overlay

Files: deploy/lab/metal-stack/base/kustomization.yaml, deploy/lab/metal-stack/infrastructure/kustomization.yaml

hack/deploy-infra.sh applies these two kustomizations in Steps 3 and 5 when it deploys onto the lab. They take deploy/kind/base and deploy/kind/infrastructure as their base, so the lab inherits what the kind overlay patches:

• the NodePort EnvoyProxy on 31443, which keeps the Envoy Service from pending as a LoadBalancer on a MetalLB without an address pool, and the twelve-listener openstack-gw Gateway;
• headlamp and flux-web, the optional addons of the kind base;
• the standalone OpenBao, the suspended service-operator releases at one replica, and the single-replica MariaDB, Memcached and Garage;
• the Flux controller requests.

The figure shows that chain from the production manifests over the kind overlay to the lab, and who applies each directory. Kustomize Overlay Structure lists every directory with its base.

The patches remove the kind pin standard from the OpenBao, MariaDB and Garage volumes. The lab has a standard class too, so the pin would bind to it by coincidence. These volumes and the proving OpenBaoCluster name no class and bind to the cluster's default class, premium on forge. No metrics-server or VPA release is rendered, because the platform runs both.

The base overlay also labels every namespace it renders with apiserver-proxy.networking.gardener.cloud/inject: disable. The lab is a Gardener shoot, and Gardener's kubernetes-service-host webhook sets KUBERNETES_SERVICE_HOST in every new pod to the API server's DNS name, so that pods bypass the node-local apiserver-proxy. The stack's NetworkPolicies expect the address instead. openbao-operator 0.4.2 reads the variable as an IP address when it derives the API-server egress of the policy it renders over an OpenBaoCluster; a name makes the read fail, the fallback (get on Service default/kubernetes) is outside the operator's RBAC, and the instance stays at APIServerNetworkReady=False with reason APIServerNetworkConfigurationInvalid. The memcached-operator's chart policy allows DNS on port 53 alone, and the shoot's CoreDNS answers behind its Service on 8053, so that pod cannot resolve the name and every API request times out. The label is Gardener's opt-out: the webhook skips namespaces that carry it, so the pods keep the kubelet's value, the kubernetes Service's ClusterIP, and reach the API server through the apiserver-proxy, the same path as on kind. It is set on every namespace rather than on the ones known to break, so the stack's network posture is one thing on the lab.

The same patch annotates every namespace it renders with chaos-mesh.org/inject: enabled, the scope of the Lab Chaos Mesh. Its release selects pods only in a namespace that carries the annotation; without Chaos Mesh the annotation does nothing.

EXTERNAL_CLUSTER=true make deploy-infra
kubectl -n envoy-gateway-system port-forward \
  "$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=openstack-gw -o name)" 8443:443
curl -sk https://keystone.127-0-0-1.nip.io:8443/v3
EXTERNAL_CLUSTER=true make teardown-infra

The deploy runs against the current kubeconfig context and never switches it. It refuses four opt-ins that are bound to kind, WITH_VPA, WITH_METRICS_SERVER, WITH_REGISTRY_CACHE and WITH_OVN_KERNEL_MODULES (WITH_MESSAGING, which applies deploy/kind/messaging unchanged, and WITH_NFS, WITH_CEPH, WITH_CHAOS_MESH and WITH_DIZZY aside, which apply nfs/, ceph/ with ceph/cluster/, chaos-mesh/ and dizzy/), checks the cluster for a default StorageClass, for the absence of a node-local-dns DaemonSet (the instance's NetworkPolicy would need spec.network.dnsEndpointIPs for a host-networked resolver) and for a Ready node, and prints the port-forward command when it completes. Under WITH_PROMETHEUS=true it applies prometheus/ in place of deploy/kind/prometheus, and it refuses that flag for an overlay without one. WITH_CEPH=true has no kind overlay: the kind mode refuses it, and this mode accepts it only for an overlay with both ceph/ and ceph/cluster/. The teardown removes the stack in finalizer order and leaves the platform's namespaces and CRDs alone. Both are described in E2E Deployment, with every variable.

Property	Value
Storage class	the cluster's default class; no manifest names one
Access	kubectl port-forward to the Envoy Service on local port 8443; the *.127-0-0-1.nip.io hostnames are unchanged
Platform overlap	none: no metrics-server, VPA, MetalLB pool or DNS entry; the opt-in autoscaling and verticalAutoscaling blocks use the platform's metrics-server and VPA (see Lab autoscaling)
Gardener	apiserver-proxy.networking.gardener.cloud/inject: disable on every namespace of the base render
Chaos Mesh scope	chaos-mesh.org/inject: enabled on every namespace of the base render
Dependencies	a default StorageClass and no node-local-dns on the cluster

Lab NFS stack

Files: deploy/lab/metal-stack/nfs/kustomization.yaml, deploy/lab/metal-stack/nfs/client-modules-daemonset.yaml, deploy/lab/metal-stack/nfs/client-policy.yaml

The NFS server and the csi-driver-nfs mounter of NFS storage stack, for the metal-stack lab (#1196). The lab's Cinder runs on the Lab Ceph without it. The NFS variant of the Lab ControlPlane, which Part 3 of the quick start applies, adds the NFS volume backend nfs1 on the share /volumes; the share /backups stays unused, because the backups go to Ceph. hack/deploy-infra.sh applies the directory in Step 3 when WITH_NFS=true is set beside EXTERNAL_CLUSTER=true, in place of deploy/kind/nfs:

EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_NFS=true make deploy-infra

The kustomization takes deploy/kind/nfs as its base, the way the Lab overlay takes the kind base, and adds the DaemonSet nfs-client-modules. Its render holds six objects and no Namespace. The two differences from kind follow the decisions D2 and D3 of #1193; the load test of the Node probe settled D2. D1 chose the kernel's nfsd as the server, and #1245 superseded it with NFS-Ganesha, so the server pod is the kind one. A third difference sits outside the kustomization: the deploy script applies the NetworkPolicy of client-policy.yaml.

Property	Value
Export claim	nfs-server-exports, 100Gi, ReadWriteOnce (D3), with no storage class. The patch removes the kind pin standard, so the claim binds to the cluster's default class like the volumes of the Lab overlay. D3 named premium, which is the default class of forge. One volume holds both exports, so 100Gi bounds every volume of nfs1 together. The kind claim asks for 5Gi on standard
Server and shares	as on kind: the server pod with prepare-exports and NFS-Ganesha on ghcr.io/kubernetes-sigs/nfs-ganesha:V6.5 by digest, its client records on the export claim, the Service on 2049, and the shares nfs-server.openstack.svc.cluster.local:/volumes and :/backups. The server loads no module
Mounter	as on kind: the csi-driver-nfs HelmRelease in kube-system, chart 4.13.4 with its three values. The chart's kubeletDir, /var/lib/kubelet, is the lab's kubelet root
nfs and nfsv4	the DaemonSet nfs-client-modules in openstack, on every node, tolerating every taint as csi-nfs-node does (D2). A privileged init container load loads both, as root, with a read-only root filesystem and the node's /lib/modules mounted read-only, and an unprivileged container hold keeps the pod running, so the load repeats after a reboot. Both run ghcr.io/c5c3/libvirt:<tag>@sha256:<digest>, the pinned image of host-prepare in Lab hypervisors, pulled IfNotPresent; Renovate moves the reference in the pull request that moves the DaemonSet's. csi-nfs-node and nova-compute mount the shares through the node's kernel, and the chart and the nova-operator render them, so neither can carry an init container from this repository
Client policy	the NetworkPolicy nfs-server-clients in openstack, from the template client-policy.yaml. It selects the server's pods and admits one ipBlock, the cluster's node network, to TCP 2049. csi-nfs-node and nova-compute are host-network pods, so the node network names every client. The template carries the placeholder NODE_NETWORK; the deploy script replaces it with data.nodeNetwork of the ConfigMap kube-system/shoot-info, which Gardener writes into every shoot, 10.128.44.0/22 on forge
Namespaces	openstack for the server and nfs-client-modules, kube-system for the chart, flux-system for the HelmRepository
Gardener label	none added. The server and nfs-client-modules pods run in openstack, which the Lab overlay labels, and mount no ServiceAccount token. The chart's pods are host-network pods in kube-system, the platform's namespace, so the lab's gardener.cloud--deny-all NetworkPolicy there does not apply to them, and they reach the API server as the platform's own host-network pods do
Pinned by	tests/unit/deploy/metal_stack_nfs_test.sh

In this mode the deploy script's preflight accepts WITH_NFS=true only for an overlay with nfs/kustomization.yaml and refuses any other before it contacts the cluster. Step 1 then refuses a cluster whose CSIDriver/nfs.csi.k8s.io the HelmRelease kube-system/csi-driver-nfs did not install, read from its helm.toolkit.fluxcd.io labels: that cluster runs its own NFS CSI driver, which Step 3 would replace and the HelmRelease would adopt, and the teardown would uninstall. Step 1 also reads the node network for the client policy and logs it as NFS client network : <cidr>. It refuses a cluster whose ConfigMap kube-system/shoot-info is missing or names no IPv4 CIDR, and one where a node has no IPv4 InternalIP inside that network, because the policy would drop that node's mounts. The script loads no module on the machine it runs on; it logs Skipping the host-side NFS kernel modules (EXTERNAL_CLUSTER=true; ...) instead. Step 3 runs the CSIDriver lifecycle-mode guard of the kind mode, applies the client policy and then <overlay>/nfs, so the server never listens without the policy, and waits up to POD_TIMEOUT seconds for the nfs-server Deployment and then for the nfs-client-modules DaemonSet. A failed wait exits 1 and names the log to read. Phase 3 waits for the csi-driver-nfs HelmRelease, as on kind. The server's two containers and the loader log with:

kubectl logs -n openstack deployment/nfs-server -c prepare-exports
kubectl logs -n openstack deployment/nfs-server -c nfs-server
kubectl logs -n openstack -l app.kubernetes.io/name=nfs-client-modules -c load --prefix --tail=-1

prepare-exports prints a prepare-exports: moved <name> into exports/ line for each entry it moved and then prepare-exports: exports/volumes and exports/backups are 42424:42424 0770. The server's lines are those of NFS storage stack. A loader pod prints nfs-client-modules: nfs and nfsv4 are loaded. When modprobe fails, it prints nfs-client-modules: cannot load <module> from /lib/modules/<kernel> and exits 1, and its pod stays in Init.

Posture. The lab carries the kind posture, narrowed in one place by the client policy. The server container is privileged, the export is sec=sys with No_Root_Squash and no client list, and with the Ephemeral lifecycle mode every principal that can create a pod mounts both shares from the pod spec. PodSecurity does not bound that: in a namespace that is not restricted the pod mounts them as root, and a restricted namespace admits a csi volume and the UID 42424, the owner of both exports. kube-system enforces no PodSecurity level on the lab and already runs privileged host-network pods (calico-node, lb-csi-node), so it admits the privileged csi-nfs-node. The lab is one tenant's cluster, and the Service is a ClusterIP.

The client policy narrows one path. On kind every pod that dials 2049 reads and writes both shares as root. On the lab only the node network reaches the port, so a pod gets to the shares only through a volume the node mounts for it. The policy does not narrow who can ask for such a volume. Calico routes the lab's pod network without encapsulation, so the server sees a node's own address, also through the Service's ClusterIP. A probe on forge on 2026-10-04 confirmed it: host-network clients on both workers appeared as 10.128.44.1 and 10.128.44.3, and under a policy with the ipBlock10.128.44.0/22 they connected while pods on either worker timed out.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes the stack at the end of its step 2, once the ControlPlane and its Cinder are gone and while the helm-controller still runs (see E2E Deployment). While the HelmRelease csi-driver-nfs exists, it waits until no pod mounts an inline nfs.csi.k8s.io volume and no PersistentVolume of that driver is Bound, because the kubelet unmounts one through csi-nfs-node; a driver the platform runs has its pods and claims left alone. Then it deletes the overlay, whose HelmRelease finalizer has the helm-controller uninstall the chart, and the CSIDriver the release created, selected by the two labels the helm-controller sets on every object of a release. The client policy is deleted after the overlay, once the server it guarded is gone. The claim goes with the overlay. Where the default class has the reclaim policy Delete, as premium on forge has, its volume and every Cinder volume on it go too.

What the pods change on a node.

• nfs-client-modules loads nfs, nfsv4 and their dependencies on every node, and the server loads none. Nothing unloads them; they stay until the node reboots, also after a teardown.
• The server pod runs NFS-Ganesha as a process of the pod, which listens on port 2049 in the pod's network namespace and keeps its client records on the export claim.
• csi-nfs-node registers its socket under /var/lib/kubelet/plugins/csi-nfsplugin and /var/lib/kubelet/plugins_registry and mounts the shares below /var/lib/kubelet/pods; these go with the pods.
• While a server on a node has a volume attached, nova-compute holds the share /volumes mounted below /var/lib/nova/mnt/<md5> in the host's mount namespace, and unmounts it with the node's last detach.
• The deploy script itself runs no modprobe and writes no file on a node.

The stack ran on the lab on 2026-10-04 with the quick start's volume steps of that time, Part 1, Step 7 and Part 2, Steps 7 and 10, which then put Cinder's volumes and backups on its two shares; its two outages under Lab fault runs left an attached volume failing until the Nova server it was attached to was hard-rebooted (#1245).

A session on 2026-10-05 read the memory of the nfs-server container under load, on shoot newforge from commit d1f114f7, where the container requested 128Mi. The working set is memory.current less inactive_file, the figure the kubelet reports and evicts by. It was read from the kubelet every 10 seconds and from the container's cgroup every 30:

Load	Length	Largest working set	Largest anonymous memory
None	300 s	49 MiB	39 MiB
Four guests write 1 GiB each to an attached volume	43 s	276 MiB	229 MiB
Four volumes of 2 GiB are backed up at once while two guests write 1 GiB each	135 s	346 MiB	286 MiB
Four guests rewrite an attached volume of 2 GiB each for 300 seconds, 113 passes in all	356 s	547 MiB	460 MiB

The request follows from the last row: 1.15 × 547 is 629, which rounds up to 640Mi. The 15 % is the margin a VPA recommender adds to a memory peak, and the shoot's recommender gave the container a memory target of 641Mi over the same session. The limit stays at 1Gi. From the first write on, the container's usage stood at the limit with page cache, and memory.events counted 36,926 max events and no oom_kill by the end. Without the page cache the usage peaked at 545 MiB, and 1.25 times that is 681 MiB. The lab's volume was fast enough for each guest to write its first 1 GiB in about 5 seconds, so the 300-second row is the one the request rests on. The files of the session are in the comment on #1260.

Lab Ceph

Files: deploy/lab/metal-stack/ceph/kustomization.yaml, deploy/lab/metal-stack/ceph/namespace.yaml, deploy/lab/metal-stack/ceph/source.yaml, deploy/lab/metal-stack/ceph/release.yaml, deploy/lab/metal-stack/ceph/csi-clientprofile-crd.yaml, deploy/lab/metal-stack/ceph/cluster/kustomization.yaml, deploy/lab/metal-stack/ceph/cluster/cluster.yaml, deploy/lab/metal-stack/ceph/cluster/pools.yaml, deploy/lab/metal-stack/ceph/cluster/clients.yaml, deploy/lab/metal-stack/ceph/cluster/toolbox.yaml, deploy/lab/metal-stack/ceph/cluster/keys-push.yaml, deploy/lab/metal-stack/ceph/cluster/keys-pull.yaml

A Ceph deployed by Rook for the metal-stack lab (#1344, a package of #1338): one monitor, one manager and nine OSDs, three per worker, on raw block volumes, the RBD pools volumes and backups, and the Ceph users cinder and cinder-backup. The RBD backends of #1341 and #1346 attach to it, and the Lab hypervisors define the libvirt secret QEMU authenticates with from the key in openstack/ceph-client-cinder. The Lab ControlPlane puts the lab's Cinder on it: the RBD volume backend ceph1 and the RBD backup target cephbk, which the Quick Start (metal-stack) uses from Part 1 on. hack/deploy-infra.sh applies both kustomizations when WITH_CEPH=true is set beside EXTERNAL_CLUSTER=true, and its preflight refuses the Lab ControlPlane without the flag. The flag has no kind overlay, and the kind mode refuses it; Ceph on kind and in CI is #1339.

EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true make deploy-infra

The Rook chart installs the ceph.rook.io CRDs, so the Ceph comes in two kustomizations. Step 3 applies ceph/: the Namespace rook-ceph, the HelmRepository rook-release, the HelmRelease rook-ceph and the CRD clientprofiles.csi.ceph.io. Phase 3 waits for that release. Step 5 waits for the CRDs cephclusters, cephblockpools and cephclients (POD_TIMEOUT) and applies ceph/cluster/. After Step 8 wait_for_ceph_cluster polls the CephCluster every 10 seconds, bounded by CEPH_TIMEOUT (900), until its phase is Ready, its health HEALTH_OK and at least as many OSD Deployments report a ready replica as the device set counts (Rook removes no OSD when the count goes down), so the OSD preparation overlaps the OpenBao bootstrap. Then sync_ceph_client_keys makes ESO validate both SecretStores again and waits for them (POD_TIMEOUT), forces both PushSecrets and waits for them, then forces both ExternalSecrets and waits until they sync (EXTERNALSECRET_TIMEOUT). Step 5 applied the stores before Step 7 had initialised OpenBao and written their roles. Every resource of either kustomization is local, so kubectl apply -k needs no --load-restrictor.

Property	Value
Operator chart	rook-ceph v1.21.0 from https://charts.rook.io/release, pinned to one version: the chart installs a privileged operator and the ceph.rook.io CRDs. csi.installCsiOperator: false and csi.createCsiOperatorResources: false, because nothing in the lab mounts Ceph through CSI: Cinder's RBD backend talks librados, and the hypervisors attach through QEMU. The discovery daemon and the operator's monitoring are off. The operator keeps the chart's resources, 200m CPU and 128Mi requested, 512Mi limit. The operator of v1.21.0 still watches the CSI operator's ClientProfile kind without a condition and exits after about two minutes when its CRD is absent (timed out waiting for cache to be synced for kind source: *v1.ClientProfile), so csi-clientprofile-crd.yaml carries that one CRD, copied from ceph-csi-operator v1.1.0, the subchart version the chart depends on; no ClientProfile is ever created
Ceph image	quay.io/ceph/ceph:v20.2.4 (Tentacle) by tag and digest, for the CephCluster and the toolbox. Renovate moves both lines in one PR (ceph image), majors disabled, nothing automerged
Cluster	the CephCluster rook-ceph: one mon on a 10Gi claim, one mgr without modules, no dashboard, crash collector, log collector or metrics, msgr2 required, the daemons' state under /var/lib/rook. cephConfig.global.osd_pool_default_size: "2" gives the .mgr pool two replicas, so it fits two hosts. One mon is a single point of failure the lab accepts (D2 of #1338)
OSDs	one device set lightbits with count: 9, three OSDs per worker of newforge, each on a 100Gi volumeMode: Block claim of the default class (D1). A topology spread on the hostname puts the OSD prepare Jobs level across the nodes, three on each, and portable: false pins each OSD to the node of its Job and names that node as its CRUSH host, so the pools' failure domain host is a worker. (A portable OSD's CRUSH host is its claim's name: with nine portable OSDs, 17 of 64 placement groups had both replicas on one worker on 2026-10-12.) A worker Gardener replaces leaves its three OSDs down until they are purged and their claims deleted; Rook then prepares three new ones. 500m CPU and 2Gi requested, 2Gi limit per OSD, 6Gi per worker; with the limit equal to the request Rook derives osd_memory_target from it. A shoot of another size changes count to three times its worker count
Pools	the CephBlockPools volumes and backups, each with two replicas on two hosts and requireSafeReplicaSize: true. Nine OSDs of 100Gi give about 450 GiB at two replicas, which bounds the lab's volumes and backups together
Clients	the CephClients cinder (mon: profile rbd, osd: profile rbd pool=volumes, mgr: profile rbd pool=volumes) and cinder-backup (the same on backups), each with keyType: aes, keyRotationPolicy: KeyGeneration and keyGeneration: 1 (D5, D7). Rook writes the Secrets rook-ceph-client-cinder and rook-ceph-client-cinder-backup, with the raw key under userKey
Key hand-off	in rook-ceph, the PushSecrets ceph-client-cinder and ceph-client-cinder-backup write userKey to the OpenBao paths ceph/client-cinder and ceph/client-cinder-backup (kv-v2/data/ceph/client-*) as the role push-ceph-keys; in openstack, the ExternalSecrets of the same names read them as the role read-ceph-keys into the Secrets openstack/ceph-client-cinder and openstack/ceph-client-cinder-backup, with the one key userKey (D4). Each side has its own SecretStore openbao-ceph-store, client Certificate of openbao-ca-issuer and ServiceAccount, and both refresh every minute. OpenBao Bootstrap lists the two roles and policies
Toolbox	the Deployment rook-ceph-tools, Rook's deploy/examples/toolbox.yaml of v1.21.0 with the cluster's image and 10m CPU and 64Mi requested, 256Mi limit. It runs as UID 2016 with every capability dropped, no privilege escalation and the RuntimeDefault seccomp profile, and mounts no ServiceAccount token
Namespace	rook-ceph, with pod-security.kubernetes.io/enforce: privileged and apiserver-proxy.networking.gardener.cloud/inject: disable, and no chaos-mesh.org/inject annotation: Ceph faults are not a lab scope
The deploy waits for	the HelmRelease rook-ceph (Phase 3), the three CRDs (Step 5), the CephCluster Ready with HEALTH_OK and at least one ready OSD per device, then the stores, the PushSecrets and the ExternalSecrets of the hand-off (after Step 8)
Pinned by	tests/unit/deploy/metal_stack_ceph_test.sh, tests/unit/hack/deploy_infra_ceph_flag_test.sh

The health wait logs Ceph: phase Ready, HEALTH_OK, 9 of 9 OSDs ready, fsid <fsid>; the fsid is the default rbd_secret_uuid of #1341. On a timeout it exits 1 with ERROR: Ceph did not reach HEALTH_OK with <count> OSDs within <n>s (phase '<phase>', health '<health>', <ready> OSDs ready). and prints the CephCluster's health details and conditions, the pods of rook-ceph and the last 50 log lines of every OSD prepare pod and of the operator. A default class that serves no volumeMode: Block claim leaves the OSD claims Pending and ends here. A store that stays unready names the kubectl describe to read; a 403 there means setup-auth.sh has not written the role. The key sync logs Ceph client keys ceph-client-cinder ceph-client-cinder-backup synced into openstack.

The cluster, a block image and a key compared through the toolbox:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rbd create volumes/lab-probe --size 1G
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rbd info volumes/lab-probe
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rbd rm volumes/lab-probe
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth get client.cinder | grep caps
kubectl -n openstack get secret ceph-client-cinder -o jsonpath='{.data.userKey}' | base64 -d | sha256sum
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth get-key client.cinder | sha256sum

ceph status shows HEALTH_OK, one mon and nine OSDs up, and the pool list names volumes, backups and .mgr. The last two commands print the same digest. They hash the key rather than print it, so a transcript of the run can be pasted into an issue without publishing a live Ceph credential. A raised keyGeneration on a client has Rook rotate its key; the PushSecret and the ExternalSecret carry the new key to openstack within their refresh intervals:

kubectl -n rook-ceph patch cephclient cinder-backup --type merge \
  -p '{"spec":{"security":{"cephx":{"keyGeneration":2}}}}'

Posture. The namespace enforces the privileged PodSecurity level. The operator runs with the chart's ClusterRoles, the mon and OSD pods mount /var/lib/rook from the host, and the OSD pods run privileged to open their raw block devices. The client keys leave rook-ceph only through OpenBao: the push role writes ceph/* from rook-ceph, and the one read role is bound to openstack, because the Ceph paths are not namespace-templated. The two clients' aes keys raise three Ceph warnings: AUTH_INSECURE_CLIENT_KEY_TYPE for the keys, and, because Rook allows the aes cipher on the monitors to create them (mon set auth_allowed_ciphers aes,aes256k on every reconcile), AUTH_INSECURE_KEYS_ALLOWED and AUTH_INSECURE_KEYS_CREATABLE. The CephCluster mutes all three (healthCheck.muteHealthWarning), so the deploy can wait for HEALTH_OK; with the first mute alone the wait ran out on 2026-10-12. D7 of #1338 keeps aes until every client's librbd reads aes256k. When it is revisited, each mute is lifted by setting policy: unmute: an entry that is deleted sends Ceph no unmute, and the warning stays muted. Rook gives its own daemons aes256k keys.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes the Ceph at the end of its step 2, after the Lab NFS stack and while the operator, ESO, cert-manager and the helm-controller still run (see E2E Deployment), in six sub-steps:

1. It renders ceph/cluster/; a render that fails exits 1 before any delete.
2. While the CRD cephclusters.ceph.rook.io exists and the CephCluster answers, it reads the nodes the Ceph daemons run on, saves their number on the namespace rook-ceph as the annotation teardown.c5c3.io/ceph-nodes, sets spec.cleanupPolicy.confirmation to yes-really-destroy-data and marks the CephCluster for deletion without waiting. The manifest never carries the confirmation, because Rook refuses spec changes once it is set. Rook's pool and client controllers count a CephCluster marked for deletion under that policy as gone and release their CRs without a Ceph command, so neither a Ceph without mon quorum nor a pool that still holds RBD images blocks the next sub-step.
3. It deletes that render without its CephCluster: the clients, the pools, the toolbox and both halves of the hand-off. The ExternalSecrets take the two Secrets in openstack with them; the PushSecrets leave the values in OpenBao (deletionPolicy: None), which go with OpenBao's volumes in step 4. Then it waits for the CephCluster, which Rook deletes once no other Rook CR references it. A delete that runs out exits 1 with the hint that the operator has to clear the finalizer.
4. Rook then starts one Job cluster-cleanup-job-<node> per node, labelled rook-ceph-cleanup=true, which empties /var/lib/rook. The teardown waits until there is one finished Job for each node read in sub-step 2, logs each with its succeeded count, and exits 1 after TEARDOWN_TIMEOUT with the Jobs that are not done. A Job that failed is logged as a warning. A rerun after that exit finds the CRD without the CephCluster and waits for one Job per node of the annotation. Only the operator process that removed the CephCluster creates the Jobs. When fewer exist, the exit says how to go on once that process is gone: empty /var/lib/rook by hand on each node without a Job, then remove the annotation. Without the annotation, a rerun waits for the cleanup Jobs that exist. A wait that passes removes the annotation, and a rerun that reads a value other than a decimal count exits 1 before the wait.
5. It deletes the claims in rook-ceph, the mon's and the OSDs'. Where the default class reclaims with Delete, as premium does, the volumes go too.
6. It deletes the render of ceph/ without its Namespace, so the helm-controller uninstalls the chart. The chart keeps its CRDs (helm.sh/resource-policy: keep); step 8 deletes them with the ceph.rook.io, objectbucket.io and csi.ceph.io groups, and step 7 deletes the Namespace rook-ceph.

The operator also clears the finalizers of the Secret rook-ceph-mon and the ConfigMap rook-ceph-mon-endpoints when it deletes the CephCluster. An operator that is gone first leaves both, and they hold the namespace in Terminating until their finalizers are removed by hand.

What the pods change on a node.

• Every node that ran a mon, a mgr, an OSD or an OSD prepare Job keeps the cluster's configuration, logs and crash data under /var/lib/rook, until the cleanup Job of the teardown empties it. The node probe reports the directory under == rook.
• The OSDs hold their data on the Lightbits volumes, which the CSI driver of the platform attaches to the node as NVMe/TCP devices; no OSD uses a disk of the node itself.
• No pod loads a kernel module, and the deploy script writes no file on a node.

Lab Chaos Mesh

Files: deploy/lab/metal-stack/chaos-mesh/kustomization.yaml, deploy/lab/metal-stack/chaos-mesh/modules-daemonset.yaml

The fault injection of #1219 for the metal-stack lab (#1221): the Chaos Mesh of the kind overlay, scoped to the namespaces the lab declares. hack/deploy-infra.sh applies the directory in Step 3 when WITH_CHAOS_MESH=true is set beside EXTERNAL_CLUSTER=true, in place of deploy/kind/chaos-mesh. The flag composes onto the deploy command of the Quick Start (metal-stack), which does not need it:

EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_CHAOS_MESH=true make deploy-infra

The kustomization takes deploy/kind/chaos-mesh as its base and adds the DaemonSet chaos-mesh-modules. Its render holds four objects: the Namespace chaos-mesh, the HelmRepository, the HelmRelease and the DaemonSet. The first three rows below are its differences from kind; the namespace filter and the loader follow the decisions D2 and D3 of #1219.

Property	Value
Namespace label	apiserver-proxy.networking.gardener.cloud/inject: disable on chaos-mesh, the lab's rule for every namespace it declares (see Lab overlay), beside the privileged PodSecurity level of the kind overlay
Namespace filter	controllerManager.enableFilterNamespace: true: the controller selects a pod only when its Namespace carries the annotation chaos-mesh.org/inject: enabled. On kind the filter is off
NetworkChaos modules	the DaemonSet chaos-mesh-modules in chaos-mesh, on every node, tolerating every taint. A privileged init container load on the image of host-prepare in Lab hypervisors loads ip_set, ip_set_hash_ip, ip_set_hash_net, xt_set, sch_netem and sch_tbf from the node's /lib/modules, and an unprivileged container hold keeps the pod running, so the load repeats after a reboot. The chart renders chaos-daemon, so it cannot carry an init container from this repository (D3)
Inherited from kind	the HelmRepository; the HelmRelease in chaos-mesh with the chart range >=2.6.0 <3.0.0 and its dependency on cert-manager; chaosDaemon.runtime: containerd with the socket /run/containerd/containerd.sock, which the Node probe found on both workers; no dashboard; the reduced requests
Namespaces	chaos-mesh for the release and the loader, flux-system for the HelmRepository
Pinned by	tests/unit/deploy/metal_stack_chaos_mesh_test.sh

In this mode the deploy script's preflight accepts WITH_CHAOS_MESH=true only for an overlay with chaos-mesh/kustomization.yaml and refuses any other before it contacts the cluster. The script loads no module on the machine it runs on; it logs Skipping the host-side chaos-mesh kernel modules (EXTERNAL_CLUSTER=true; ...) instead. Step 3 applies <overlay>/chaos-mesh and waits up to POD_TIMEOUT seconds for the chaos-mesh-modules DaemonSet. A failed wait exits 1 and names the log to read. Phase 3 waits for the chaos-mesh HelmRelease, as on kind. The loader logs one line per pod:

kubectl logs -n chaos-mesh -l app.kubernetes.io/name=chaos-mesh-modules -c load --prefix --tail=-1

A pod prints chaos-mesh-modules: ip_set, ip_set_hash_ip, ip_set_hash_net, xt_set, sch_netem and sch_tbf are loaded. When modprobe fails, it prints chaos-mesh-modules: cannot load <module> from /lib/modules/<kernel> and exits 1, and its pod stays in Init.

Scope. The Lab overlay annotates every Namespace of its base render with chaos-mesh.org/inject: enabled, and deploy/lab/metal-stack/migration-ports/namespace.yaml annotates hypervisor-system, which Part 2 of the quick start applies, whether Chaos Mesh is deployed or not (D2 of #1219). Three groups carry no annotation, so no experiment selects their pods:

• chaos-mesh itself;
• flux-system, which the flux-operator install creates and no lab manifest declares;
• the platform's namespaces kube-system, firewall, metallb-system and default. An experiment that hit calico-node, vpn-shoot or apiserver-proxy would cut the shoot off from its control plane in the seed.

A selector that names only such namespaces matches nothing, and the experiment fails with no pod is selected. To take a lab namespace out of the scope, remove its annotation; the next deploy sets it again:

kubectl annotate namespace <name> chaos-mesh.org/inject-

NetworkChaos and hostNetwork. No NetworkChaos selects a pod that runs in the host's network namespace. On the lab these are libvirt, nova-compute, the pods of the OVNChassis and of the metadata agent, and migration-port-reservation. A delay or a loss there would act on the network of the node, and the shoot is shared. The chart's controllerManager.allowHostNetworkTesting defaults to false, and the controller then refuses such an experiment with the event It's dangerous to inject network chaos on a pod(<ns>/<name>) with `hostNetwork` . The overlay sets neither that value nor clusterScoped, and its test fails when the render names either.

Posture. chaos-daemon runs privileged with hostPID and the containerd socket on every untainted node. Whoever may create a chaos-mesh.org object in an annotated namespace therefore acts as root on every node, which the lab accepts as one tenant's cluster. The namespace filter bounds the pods an experiment selects; RBAC on the chaos-mesh.org kinds bounds who may write one. The chart's webhooks match chaos-mesh.org resources only, none matches pods.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes Chaos Mesh first, before its step 0, while chaos-controller-manager, chaos-daemon and the helm-controller still run (see E2E Deployment). It deletes the schedules and workflows, which create experiments, then every experiment, so the controller releases each fault before it clears the experiment's finalizer. The per-pod records podnetworkchaos, podiochaos and podhttpchaos carry no finalizer and go with their CRDs. An experiment that keeps its finalizer past TEARDOWN_TIMEOUT stops the teardown with exit 1 and a hint to read kubectl logs -n chaos-mesh deployment/chaos-controller-manager. Do not remove the finalizer by hand: the fault would stay injected. Then the teardown deletes the overlay without its Namespace, so the helm-controller uninstalls the chart while Helm's release Secret is still there. Its step 7 deletes the namespace chaos-mesh and the chart's cluster-scoped leftovers, its step 8 the chaos-mesh.org CRDs the chart installs from its crds/ (23 in chart 2.8.4), which Helm never deletes.

What the pods change on a node.

• chaos-mesh-modules loads ip_set, ip_set_hash_ip, ip_set_hash_net, xt_set, sch_netem, sch_tbf and their dependencies on every node. Nothing unloads them; they stay until the node reboots, also after a teardown. calico-node uses the ipset family as well.
• chaos-daemon mounts /run/containerd, /sys and /lib/modules from the node and goes with the release.
• The deploy script itself runs no modprobe and writes no file on a node.

make e2e-chaos is not run against the lab (D6 of #1219): the suites under tests/e2e-chaos/ are written for the kind stack. The lab's own faults, two NFS outages and killed hypervisor pods, are under Lab fault runs.

Proving run

The run starts from a bare forge, deploys with the command above, and runs Part 1 of the Quick Start (metal-stack) through Step 6, with the port-forward open. Then, from the root of the clone:

# 1. State: the chart version, chaos-daemon on both workers, one loader line
#    per node
kubectl get helmrelease chaos-mesh -n chaos-mesh -o jsonpath='{.status.history[0].chartVersion}'; echo
kubectl rollout status daemonset/chaos-daemon -n chaos-mesh --timeout=120s
kubectl logs -n chaos-mesh -l app.kubernetes.io/name=chaos-mesh-modules -c load --prefix --tail=-1

# 2. Baseline: five requests through the port-forward
for i in 1 2 3 4 5; do
  curl -sk -o /dev/null -w '%{time_total}\n' https://keystone.127-0-0-1.nip.io:8443/v3
done

# The two experiments on the Keystone API pods, kept for steps 4 and 6
run="$(mktemp -d)"
cat >"$run/kill.yaml" <<'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-keystone-kill
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [openstack]
    labelSelectors:
      app.kubernetes.io/name: keystone
      app.kubernetes.io/component: api
EOF
cat >"$run/delay.yaml" <<'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: lab-keystone-delay
  namespace: openstack
spec:
  action: delay
  mode: all
  selector:
    namespaces: [openstack]
    labelSelectors:
      app.kubernetes.io/name: keystone
      app.kubernetes.io/component: api
  delay:
    latency: 300ms
  # The fault ends by itself when the run is abandoned.
  duration: 10m
EOF

# 3. PodChaos: a Keystone API pod is killed and replaced
export sel='app.kubernetes.io/name=keystone,app.kubernetes.io/component=api'
export before="$(kubectl get pods -n openstack -l "$sel" -o jsonpath='{.items[*].metadata.uid}')"
kubectl apply -f "$run/kill.yaml"
kubectl wait podchaos/lab-keystone-kill -n openstack --for=condition=AllInjected --timeout=60s
timeout 300 bash -c 'until [ "$(kubectl get pods -n openstack -l "$sel" -o jsonpath={.items[*].metadata.uid})" != "$before" ]; do sleep 2; done'
kubectl wait pod -n openstack -l "$sel" --for=condition=Ready --timeout=300s
kubectl delete podchaos lab-keystone-kill -n openstack --wait --timeout=60s

# 4. NetworkChaos: 300 ms on every Keystone API pod, then released
kubectl apply -f "$run/delay.yaml"
kubectl wait networkchaos/lab-keystone-delay -n openstack --for=condition=AllInjected --timeout=60s
for i in 1 2 3 4 5; do
  curl -sk -o /dev/null -w '%{time_total}\n' https://keystone.127-0-0-1.nip.io:8443/v3
done
kubectl delete networkchaos lab-keystone-delay -n openstack --wait --timeout=120s
for i in 1 2 3 4 5; do
  curl -sk -o /dev/null -w '%{time_total}\n' https://keystone.127-0-0-1.nip.io:8443/v3
done

# 5. Scope: a pod in a namespace without the annotation is not selected
K="$(yq -r '.spec.template.spec.containers[0].image' deploy/lab/metal-stack/chaos-mesh/modules-daemonset.yaml)"
kubectl create namespace chaos-scope-probe
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: scope-probe
  namespace: chaos-scope-probe
  labels:
    app: scope-probe
spec:
  automountServiceAccountToken: false
  containers:
    - name: hold
      image: ${K}
      command: ["/bin/bash", "-c", "trap 'exit 0' TERM; sleep infinity & wait"]
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests: {cpu: 1m, memory: 8Mi}
        limits: {memory: 32Mi}
EOF
kubectl wait pod/scope-probe -n chaos-scope-probe --for=condition=Ready --timeout=120s
kubectl get pod scope-probe -n chaos-scope-probe -o jsonpath='{.metadata.uid}{"\n"}'
kubectl apply -f - <<'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-scope-probe
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [chaos-scope-probe]
    labelSelectors:
      app: scope-probe
EOF
sleep 60
kubectl get pod scope-probe -n chaos-scope-probe \
  -o jsonpath='{.metadata.uid} restarts={.status.containerStatuses[0].restartCount}{"\n"}'
kubectl get podchaos lab-scope-probe -n openstack \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
kubectl get events -n openstack --field-selector involvedObject.name=lab-scope-probe
kubectl delete podchaos lab-scope-probe -n openstack --wait --timeout=60s
kubectl delete namespace chaos-scope-probe --wait --timeout=120s

# 6. A fault left active, then the teardown and what it leaves
kubectl apply -f "$run/delay.yaml"
kubectl wait networkchaos/lab-keystone-delay -n openstack --for=condition=AllInjected --timeout=60s
EXTERNAL_CLUSTER=true make teardown-infra
kubectl get crd -o name | grep -c 'chaos-mesh\.org'
kubectl get namespace chaos-mesh
kubectl get clusterrole,clusterrolebinding,mutatingwebhookconfiguration,validatingwebhookconfiguration -o name | grep -c chaos-mesh
rm -r "$run"

What a run shows:

Step	Expected
1	the resolved chart version; chaos-daemon rolled out on both workers; one are loaded line per node
2	five times, the run's baseline; step 4 compares against the largest of them
3	AllInjected, a Keystone API pod with a new UID that becomes Ready, and the PodChaos deleted within 60 seconds
4	AllInjected and five times, each at least 0.2 s above the largest time of step 2; the delete returns within 120 seconds, then five times, each less than 0.1 s above the largest time of step 2
5	the pod's UID unchanged and restarts=0; the PodChaos reports that no pod is selected
6	the teardown exits 0, logs Deleting the Chaos Mesh experiments... and ends with Stack CRDs left: 0; stack namespaces left: 0; cluster-scoped chart objects left: 0; then 0, NotFound and 0

Steps 2 and 4 name no absolute time, because a request's time includes the round trip from the workstation to the shoot, and that differs from one workstation to the next. Take the largest of the five times of step 2. The delay of 300 ms counts as injected when each time of step 4's first loop is at least 0.2 s above it, and as released when each time of the second loop is less than 0.1 s above it. 0.2 s is 0.1 s below the delay, and 0.1 s is 0.1 s above no delay, which is the room for jitter of a request as slow as the largest time of step 2. Both limits count from that time, so for a request as fast as the fastest time of step 2 the room is 0.1 s less the spread of step 2's times under the delay, and 0.1 s plus that spread after the release. With a largest baseline time of 0.306 s the two limits are 0.506 s and 0.406 s.

The run of 2026-10-04 started from a bare newforge, the lab's shoot of three Xeon D-2141I workers on Kubernetes v1.35.6, and ran from commit 647736c1. It was made in the session of Lab fault runs: steps 1 to 5 ran after Part 1, Step 7 of the quick start, and step 6 an hour and a half later, after the fault runs and the quick start's deletes. Step 1 printed the chart version 2.8.4, rolled chaos-daemon out on all three workers and printed one are loaded line per node. Step 2 took 0.252, 0.261, 0.278, 0.303 and 0.306 seconds, so the largest baseline time is 0.306 s. Step 3 matched its row. In step 4 the five requests under the delay took 0.812, 0.779, 0.984, 1.084 and 0.905 seconds, each at least 0.473 s above 0.306 s; the delete returned within its 120 seconds, and the five requests after it took 0.325, 0.299, 0.317, 0.328 and 0.307 seconds, at most 0.022 s above it. Steps 2 and 4 match their rows. The table of commit 647736c1 expected a fixed 0.3 s in both steps, and six of the run's ten requests without the delay did not stay below it, because from that workstation the round trip to the shoot is itself near 0.3 seconds (#1246). Step 5 matched: the probe pod kept its UID with restarts=0, and the PodChaos reported Failed to select targets: no pod is selected. Step 6 matched: the teardown exited 0 after 5 minutes 10 seconds, logged Deleting the Chaos Mesh experiments... and ended with Stack CRDs left: 0; stack namespaces left: 0; cluster-scoped chart objects left: 0, and the three checks printed 0, NotFound and 0. No step needed a hand action. The outputs are on #1222.

Lab fault runs

The block below runs five faults that kind cannot show, each under the running server lab-0 of the Quick Start (metal-stack) (#1222). F1 to F3 are a PodChaos pod-kill of the libvirt pod, of the nova-compute pod and of the ovn-controller pod on lab-0's node. F4 is a pod-kill of the NFS server pod while lab-0 has a Cinder volume on NFS attached, and stands for a reschedule of that pod. F5 scales the NFS server to 0 for 300 seconds and back, and stands for a node replacement. Every fault is a pod fault: the three hypervisor pods run in the host's network namespace (see NetworkChaos and hostNetwork above), and D2 of #1219 rules out a network fault on them. A PodChaos kills with its default gracePeriod: 0. The graceful delete in Checks outside the quick start gives libvirt's start script its TERM handler and ovn-controller its preStop exit --restart; a kill cuts both short. The faults run on the node of lab-0, and lab-1 on another node is the peer of the guest's network probe.

The block starts on a lab where Part 1 ran with WITH_CHAOS_MESH=true and WITH_NFS=true added to the deploy command of its Step 3, Part 2 ran through Step 11 and Part 3 through Step 2, so lab-nfs on the NFS backend nfs1 is attached to lab-0 as its disk vdc. It runs in bash, in the shell of Part 3, Step 2, which holds nfs_volume, libvirt_pod, domain and the OS_* variables, with the port-forward of Part 1, Step 6 open in a second terminal.

Two probes in lab-0 show what the guest sees. Open lab-0's console as in Part 3, Step 2, log in, and type these two lines, with <lab-1> replaced by the address Part 2, Step 6 printed for lab-1:

sudo sh -c 'while :; do if ping -c 1 -W 1 <lab-1> >/dev/null 2>&1; then r=ok; else r=lost; fi; echo "lab-probe net $(date -u +%T) $r"; sleep 2; done >/dev/ttyS0 2>&1 &'
sudo sh -c 'i=0; while :; do i=$((i+1)); if echo "probe-$i" | dd of=/dev/vdc bs=512 seek=8 2>/dev/null && sync; then r=ok; else r=error; fi; echo "lab-probe disk $(date -u +%T) $r $i"; sleep 2; done >/dev/ttyS0 2>&1 &'

Each loop writes one line to the serial console about every two seconds, and Nova keeps that console as the server's console log, so openstack console log show reads the lines from outside. The network loop pings lab-1 through the Geneve tunnel to the other node. The disk loop writes its counter at byte 4096 of the NFS volume, so the marker of Part 3, Step 2 at byte 0 stays, and syncs it. A loop that blocks in sync writes no line, and the gap in the log is the stall the guest saw. Leave the console with Ctrl+]. Before F1, openstack console log show --lines 20 lab-0 shows both kinds of line with ok.

The block defines the helpers and reads the baseline, then runs F1 to F5 and a Cinder check, one section each. Paste it a section at a time, and read the output of each before the next. The baseline shows server=ACTIVE, a domain=running_ reason, compute=up, agents= with one or more True and nothing else, volume=in-use, cinder-volume=up, every kind with :True and hypervisor=True, and baseline.pids holds two PIDs. Otherwise the lab is not at the state the faults are measured against, and no fault runs. In each fault section, the kubectl get podchaos line prints the node the API kept and the pod Chaos Mesh selected: ${host} and the pod saved in <f>.before, and for F4 no node. Otherwise the selector did not hold, and the session ends. It ends as well when a PodChaos does not reach AllInjected within 60 seconds; kubectl describe then prints the experiment's state.

# bash; KUBECONFIG and the OS_* variables of Part 1, Step 7 set, the port-forward
# of Part 1, Step 6 running, lab-nfs attached to lab-0, both guest probes running
faults="$(mktemp -d)"
host=$(openstack server show lab-0 -c OS-EXT-SRV-ATTR:host -f value)
domain=$(openstack server show lab-0 -c OS-EXT-SRV-ATTR:instance_name -f value)
echo "lab-0 runs as ${domain} on ${host}"

# the libvirt pod of lab-0's node
node_libvirt_pod() {
  kubectl get pod -n openstack -l app.kubernetes.io/name=libvirt \
    --field-selector "spec.nodeName=${host}" -o name
}
# the host PIDs of lab-0's QEMU and of libvirtd (the libvirt pod shares the host's
# PID namespace); [t] keeps pgrep from matching the sh that runs it
pids() {
  kubectl exec -n openstack "$(node_libvirt_pod)" -c libvirtd -- \
    sh -c "echo qemu=\$(pgrep -f 'gues[t]=${domain},') libvirtd=\$(pgrep -x libvirtd)"
}
# one field: the output of "$@", and its error output only when it fails, so a
# warning the CLI prints on stderr stays out of the field
field() {
  local err
  { err=$("$@" 2>&1 1>&3 3>&-) || echo "${err}"; } 3>&1 | tr -s ' \n\r' '_'
}
# one line: what the APIs, the operators and libvirt report for lab-0 and its node
state() {
  echo "server=$(field openstack server show lab-0 -c status -f value)" \
    "domain=$(field timeout 10 kubectl exec -n openstack "$(node_libvirt_pod)" -c libvirtd -- virsh domstate --reason "${domain}")" \
    "compute=$(field openstack compute service list --service nova-compute --host "${host}" -c State -f value)" \
    "agents=$(field openstack network agent list --host "${host}" -c Alive -f value)" \
    "volume=$(field openstack volume show lab-nfs -c status -f value)" \
    "cinder-volume=$(field openstack volume service list --service cinder-volume --host controlplane-cinder@nfs1 -c State -f value)" \
    "ready=$(field kubectl get novacompute,ovnchassis,neutronmetadataagent,cinder,cinderbackend -n openstack -o jsonpath='{range .items[*]}{.kind}:{.status.conditions[?(@.type=="Ready")].status},{end}')" \
    "hypervisor=$(field kubectl get hypervisor "${host}" -o jsonpath='{.status.conditions[?(@.type=="LibVirtConnection")].status}')" \
    "nfs-server=$(field kubectl get pod -n openstack -l app.kubernetes.io/name=nfs-server -o jsonpath='{range .items[*]}{.spec.nodeName}/{.status.phase}/{.status.containerStatuses[0].ready},{end}')"
}
# appends "<UTC time> <state>" to $1 whenever the state changes; runs until killed
watch_state() {
  local previous="" current
  while :; do
    current=$(state)
    if [ "${current}" != "${previous}" ]; then
      echo "$(date -u +%T) ${current}" >>"$1"
      previous=${current}
    fi
    sleep 5
  done
}
# the last 120 lines of lab-0's console log, or a failure when the call fails or
# prints nothing; the serial console ends lines in \r\n
console_log() {
  local log
  log=$(openstack console log show --lines 120 lab-0 2>/dev/null) && [ -n "${log}" ] || return 1
  tr -d '\r' <<<"${log}"
}
# the newest line of the guest probe $1 (net or disk) in the console log on stdin
last_probe() {
  grep "lab-probe $1 " | tail -n 1
}
# true when both probes wrote a new "ok" line in the next 20 seconds
probes_ok() {
  local before after
  before=$(console_log) || return 1
  sleep 20
  after=$(console_log) || return 1
  [ "$(last_probe net <<<"${after}")" != "$(last_probe net <<<"${before}")" ] &&
    last_probe net <<<"${after}" | grep -q ' ok$' &&
    [ "$(last_probe disk <<<"${after}")" != "$(last_probe disk <<<"${before}")" ] &&
    last_probe disk <<<"${after}" | grep -q ' ok [0-9]*$'
}
# waits up to 300 s for pods of selector $1 without the UID(s) $2, on lab-0's node
# when $3 is "node", then up to 600 s for every pod of $1 to be Ready
wait_replaced() {
  local end=$((SECONDS + 300)) uids
  while [ "${SECONDS}" -lt "${end}" ]; do
    if [ "$3" = node ]; then
      uids=$(kubectl get pod -n openstack -l "$1" --field-selector "spec.nodeName=${host}" -o jsonpath='{.items[*].metadata.uid}')
    else
      uids=$(kubectl get pod -n openstack -l "$1" -o jsonpath='{.items[*].metadata.uid}')
    fi
    if [ -n "${uids}" ] && ! echo " ${uids} " | grep -q " $2 "; then
      kubectl wait pod -n openstack -l "$1" --for=condition=Ready --timeout=600s && return 0
      break
    fi
    sleep 2
  done
  echo "GATE FAILED: no Ready replacement for $1"; return 1
}
# the gate before the next fault: within $1 seconds lab-0 is ACTIVE, nova-compute
# on its node is up, Neutron lists network agents of its node and every one is alive,
# and both probes write "ok"
recovered() {
  local end=$((SECONDS + $1)) agents
  while [ "${SECONDS}" -lt "${end}" ]; do
    if [ "$(openstack server show lab-0 -c status -f value)" = ACTIVE ] &&
      [ "$(openstack compute service list --service nova-compute --host "${host}" -c State -f value)" = up ] &&
      agents=$(openstack network agent list --host "${host}" -c Alive -f value) &&
      [ -n "${agents}" ] && ! grep -qv True <<<"${agents}" &&
      probes_ok; then
      echo "recovered at $(date -u +%T)"; return 0
    fi
    sleep 5
  done
  echo "GATE FAILED: lab-0, nova-compute, the agents of ${host} or the guest probes did not recover in $1 s"; return 1
}
state | tee "$faults/baseline.state"
pids | tee "$faults/baseline.pids"

# F1: the libvirt pod of lab-0's node
sel=app.kubernetes.io/name=libvirt
pids >"$faults/libvirt.before"
before=$(kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o jsonpath='{.items[*].metadata.uid}')
kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o name >>"$faults/libvirt.before"
watch_state "$faults/libvirt.log" & watcher=$!
echo "applied at $(date -u +%T)" | tee -a "$faults/libvirt.log"
kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-libvirt-kill
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [openstack]
    nodes: [${host}]
    labelSelectors:
      app.kubernetes.io/name: libvirt
EOF
kubectl wait podchaos/lab-libvirt-kill -n openstack --for=condition=AllInjected --timeout=60s ||
  kubectl describe podchaos lab-libvirt-kill -n openstack
kubectl get podchaos lab-libvirt-kill -n openstack -o jsonpath='{.spec.selector.nodes}{" "}{.status.experiment.containerRecords[*].id}{"\n"}'
wait_replaced "${sel}" "${before}" node
recovered 600
kill "${watcher}"
pids >"$faults/libvirt.after"
state >"$faults/libvirt.state"
openstack console log show lab-0 | tr -d '\r' | grep 'lab-probe ' >"$faults/libvirt.console"
kubectl logs -n openstack "$(node_libvirt_pod)" -c libvirtd | head -n 20 >"$faults/libvirt.pod-log"
# typed on its own: press Enter, note whether a prompt answers, leave with Ctrl+]
kubectl exec -it -n openstack "$(node_libvirt_pod)" -c libvirtd -- virsh console "${domain}"
kubectl delete podchaos lab-libvirt-kill -n openstack --wait --timeout=60s

# F2: the nova-compute pod of lab-0's node, and the share it mounted
sel=app.kubernetes.io/name=novacompute,app.kubernetes.io/instance=lab,app.kubernetes.io/component=nova-compute
pids >"$faults/nova-compute.before"
before=$(kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o jsonpath='{.items[*].metadata.uid}')
kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o name >>"$faults/nova-compute.before"
kubectl exec -n openstack "$(node_libvirt_pod)" -c libvirtd -- grep ' /var/lib/nova/mnt/' /proc/mounts >"$faults/nova-compute.mount.before"
watch_state "$faults/nova-compute.log" & watcher=$!
echo "applied at $(date -u +%T)" | tee -a "$faults/nova-compute.log"
kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-nova-compute-kill
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [openstack]
    nodes: [${host}]
    labelSelectors:
      app.kubernetes.io/name: novacompute
      app.kubernetes.io/instance: lab
      app.kubernetes.io/component: nova-compute
EOF
kubectl wait podchaos/lab-nova-compute-kill -n openstack --for=condition=AllInjected --timeout=60s ||
  kubectl describe podchaos lab-nova-compute-kill -n openstack
kubectl get podchaos lab-nova-compute-kill -n openstack -o jsonpath='{.spec.selector.nodes}{" "}{.status.experiment.containerRecords[*].id}{"\n"}'
wait_replaced "${sel}" "${before}" node
recovered 600
kill "${watcher}"
pids >"$faults/nova-compute.after"
state >"$faults/nova-compute.state"
openstack console log show lab-0 | tr -d '\r' | grep 'lab-probe ' >"$faults/nova-compute.console"
kubectl exec -n openstack "$(node_libvirt_pod)" -c libvirtd -- grep ' /var/lib/nova/mnt/' /proc/mounts >"$faults/nova-compute.mount.after"
kubectl logs -n openstack "$(kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o name)" -c nova-compute |
  grep -E 'ERROR|Traceback' | head -n 40 >"$faults/nova-compute.errors"
kubectl delete podchaos lab-nova-compute-kill -n openstack --wait --timeout=60s

# F3: the ovn-controller pod of lab-0's node
sel=app.kubernetes.io/name=ovnchassis,app.kubernetes.io/instance=lab-chassis,app.kubernetes.io/component=ovn-controller
pids >"$faults/ovn-controller.before"
before=$(kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o jsonpath='{.items[*].metadata.uid}')
kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o name >>"$faults/ovn-controller.before"
openstack network agent list --host "${host}" -c 'Agent Type' -c Alive -f value >"$faults/ovn-controller.agents.before"
watch_state "$faults/ovn-controller.log" & watcher=$!
echo "applied at $(date -u +%T)" | tee -a "$faults/ovn-controller.log"
kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-ovn-controller-kill
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [openstack]
    nodes: [${host}]
    labelSelectors:
      app.kubernetes.io/name: ovnchassis
      app.kubernetes.io/instance: lab-chassis
      app.kubernetes.io/component: ovn-controller
EOF
kubectl wait podchaos/lab-ovn-controller-kill -n openstack --for=condition=AllInjected --timeout=60s ||
  kubectl describe podchaos lab-ovn-controller-kill -n openstack
kubectl get podchaos lab-ovn-controller-kill -n openstack -o jsonpath='{.spec.selector.nodes}{" "}{.status.experiment.containerRecords[*].id}{"\n"}'
wait_replaced "${sel}" "${before}" node
recovered 600
kill "${watcher}"
pids >"$faults/ovn-controller.after"
state >"$faults/ovn-controller.state"
openstack console log show lab-0 | tr -d '\r' | grep 'lab-probe ' >"$faults/ovn-controller.console"
openstack network agent list --host "${host}" -c 'Agent Type' -c Alive -f value >"$faults/ovn-controller.agents.after"
kubectl logs -n openstack "$(kubectl get pod -n openstack -l "${sel}" --field-selector "spec.nodeName=${host}" -o name)" -c ovn-controller |
  head -n 40 >"$faults/ovn-controller.pod-log"
kubectl delete podchaos lab-ovn-controller-kill -n openstack --wait --timeout=60s

# Before F4: the mount nova-compute holds, with its options, and the NFS server's pod
kubectl exec -n openstack "$(node_libvirt_pod)" -c libvirtd -- grep ' /var/lib/nova/mnt/' /proc/mounts >"$faults/nfs.mount"
kubectl get pod -n openstack -l app.kubernetes.io/name=nfs-server -o wide >"$faults/nfs.before"

# F4: the NFS server pod, wherever it runs (a reschedule)
sel=app.kubernetes.io/name=nfs-server
pids >"$faults/nfs-server-kill.before"
before=$(kubectl get pod -n openstack -l "${sel}" -o jsonpath='{.items[*].metadata.uid}')
kubectl get pod -n openstack -l "${sel}" -o name >>"$faults/nfs-server-kill.before"
watch_state "$faults/nfs-server-kill.log" & watcher=$!
echo "applied at $(date -u +%T)" | tee -a "$faults/nfs-server-kill.log"
kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: lab-nfs-server-kill
  namespace: openstack
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [openstack]
    labelSelectors:
      app.kubernetes.io/name: nfs-server
EOF
kubectl wait podchaos/lab-nfs-server-kill -n openstack --for=condition=AllInjected --timeout=60s ||
  kubectl describe podchaos lab-nfs-server-kill -n openstack
kubectl get podchaos lab-nfs-server-kill -n openstack -o jsonpath='{.spec.selector.nodes}{" "}{.status.experiment.containerRecords[*].id}{"\n"}'
wait_replaced "${sel}" "${before}" any
recovered 900
kill "${watcher}"
pids >"$faults/nfs-server-kill.after"
state >"$faults/nfs-server-kill.state"
openstack console log show lab-0 | tr -d '\r' | grep 'lab-probe ' >"$faults/nfs-server-kill.console"
kubectl get pod -n openstack -l "${sel}" -o wide | tee "$faults/nfs-server-kill.pod"
kubectl get events -n openstack --sort-by=.lastTimestamp | grep -E 'nfs-server|AttachVolume|Multi-Attach' >"$faults/nfs-server-kill.events"
kubectl logs -n openstack deployment/nfs-server -c nfs-server | grep -E 'IN GRACE|check grace|Added filesystem' >"$faults/nfs-server-kill.pod-log"
kubectl delete podchaos lab-nfs-server-kill -n openstack --wait --timeout=60s

# F5: the NFS server scaled to 0 for 300 seconds (a node replacement)
pids >"$faults/nfs-server-scale.before"
watch_state "$faults/nfs-server-scale.log" & watcher=$!
echo "scaled to 0 at $(date -u +%T)" | tee -a "$faults/nfs-server-scale.log"
kubectl scale deployment/nfs-server -n openstack --replicas=0
kubectl wait pod -n openstack -l app.kubernetes.io/name=nfs-server --for=delete --timeout=120s
sleep 300
echo "scaled to 1 at $(date -u +%T)" | tee -a "$faults/nfs-server-scale.log"
kubectl scale deployment/nfs-server -n openstack --replicas=1
kubectl rollout status deployment/nfs-server -n openstack --timeout=600s
echo "rolled out at $(date -u +%T)" | tee -a "$faults/nfs-server-scale.log"
recovered 900
kill "${watcher}"
pids >"$faults/nfs-server-scale.after"
state >"$faults/nfs-server-scale.state"
openstack console log show lab-0 | tr -d '\r' | grep 'lab-probe ' >"$faults/nfs-server-scale.console"
kubectl get events -n openstack --sort-by=.lastTimestamp | grep -E 'nfs-server|AttachVolume|Multi-Attach' >"$faults/nfs-server-scale.events"

# After F5: a new volume on the backend whose server came back, deleted whatever its status
openstack volume create --type nfs --size 1 lab-vol-after
timeout 300 bash -c 'until [ "$(openstack volume show lab-vol-after -c status -f value)" = available ]; do sleep 5; done'
openstack volume show lab-vol-after -c status -f value
openstack volume delete lab-vol-after
timeout 120 bash -c 'until ! openstack volume show lab-vol-after >/dev/null 2>&1; do sleep 5; done'

Every wait has a bound. A replacement pod with a new UID gets 300 seconds, and its Ready 600. The gate recovered gets 600 seconds after F1 to F3 and 900 after F4 and F5, because the restarted server starts in its NFSv4 grace period of up to 90 seconds and the client retries on its own back-off (see tests/e2e-chaos/cinder-nfs-outage/chainsaw-test.yaml). F5 holds the scale-down for 300 seconds, gives the server's pod 120 seconds to go and 600 to roll out, and the new volume gets 300. When a helper prints GATE FAILED:

1. Run openstack token issue -f value -c id >/dev/null. When it cannot connect to keystone.127-0-0-1.nip.io:8443, the port-forward has ended: start it as in Part 1, Step 6 and repeat the wait.
2. Otherwise repeat the wait once, with the same bound.
3. When it fails again, run openstack server reboot --hard --wait lab-0, open lab-0's console with the virsh console line of F1, because the libvirt_pod of Part 3, Step 2 names the pod F1 killed, start both guest probes again and run recovered 600.
4. When that holds, go on with the next fault. Otherwise the remaining faults do not run, and the session goes to its end.

A fault shows a defect when a gate fails, when the qemu= PID differs between <f>.before and <f>.after (the guest restarted), or when a field of <f>.state differs from baseline.state. domain= is compared by its running_ prefix, agents= by holding one or more True and nothing else, nfs-server= by its phase and ready flag, as its node may change, and every other field as text. An F1 console that does not answer, a lab-vol-after that ends in error or not available in time, and a failed check at the session's end are defects as well. Each gets its own issue. A status that does not move during a fault is a finding, and so is a guest stall that ends by itself within the bound.

At the session's end, open lab-0's console with the virsh console line of F1 and stop the disk probe. cirros has no pkill, so the line finds the loop's PID with ps, and [l] keeps awk from matching itself:

sudo kill $(ps | awk '/[l]ab-probe disk/ {print $1}')

Read its last line at once, because the network probe pushes it out of the 120 lines console_log reads within minutes. Then run Part 3, Step 4 and read the counter the disk probe wrote last from the volume file:

# after the disk probe stopped on lab-0's console
console_log | last_probe disk
# after Part 3, Step 4
kubectl exec -n openstack deployment/controlplane-cinder-volume-nfs1 -- sh -c "f=\$(ls /var/lib/cinder/mnt/*/volume-${nfs_volume}); dd if=\"\$f\" bs=512 skip=8 count=1 2>/dev/null | head -c 16; echo"

It prints probe-<n>. <n> is the counter of the last ok disk line or the one after it, because the kill can stop a write whose sync had not returned. The head -c 17 of Part 3, Step 4 still prints nfs-volume-marker. Then run the Teardown block without its last line, and step 6 of the Proving run in its place, which runs EXTERNAL_CLUSTER=true make teardown-infra with a fault active. Stop the port-forward, delete gateway-ca.pem, and check that kubectl get namespaces -o name lists the platform's namespaces alone: default, firewall, kube-node-lease, kube-public, kube-system and metallb-system.

The run of 2026-10-04 ran the block from commit 647736c1 on shoot newforge, three Xeon D-2141I workers on Kubernetes v1.35.6, with Chaos Mesh chart 2.8.4, in one session with the Proving run. lab-0 ran on shoot--df33f0b4c1--newforge-group-0-85bcf-7znw9 and lab-1 on shoot--df33f0b4c1--newforge-group-0-85bcf-hmw67, and the NFS server started on the third worker, shoot--df33f0b4c1--newforge-group-0-85bcf-rt6kn. The quick start then ran Cinder on the NFS stack alone, so the block started from Part 2, Step 7, which had attached lab-vol on nfs1 as the disk vdb, read the state of lab-vol and of the one cinder-volume, created lab-vol-after without a type and ended with Part 2, Step 10. The block has not run from Part 3 since Cinder moved to the lab Ceph. Nova mounted the share as nfs4 with vers=4.2, hard, proto=tcp, timeo=600 and retrans=2. The guest's clock was within 2 seconds of the workstation's. A script typed the console input and fed the page's blocks to one bash shell, a section at a time, and the port-forward ran without a restart. The block ran with two changes, both on this page since. field took the place of flat in state, because the workstation's OpenStack CLI (Homebrew openstackclient 8.2.0) prints a readline warning on stderr with every call, which 2>&1 | flat put into every field. The disk probe was stopped with kill and ps, because cirros 0.6.3 has no pkill. Since the run, F5 waits for the server's pod to go with kubectl wait --for=delete in place of a polling loop, with the same 120-second bound. Also since the run, recovered fails on an empty agent list, a failed agent call, or a console read that fails or prints nothing, which it passed before, and probes_ok reads the console log twice per check, where it read it up to six times. F4 and F5 each failed their gate twice, and lab-0 was hard-rebooted as the failed-gate steps order. The other hand actions only read: the empty and absent paths of the helpers after F1, whose pod gap fell between two samples of watch_state; sudo head -c 17 /dev/vdb on the console after F4's reboot; the guest's process tools before the kill; and the nodes' kernel logs, QEMU's block errors and the volume file, through the libvirt pods during F4 and F5. After each reboot the console was opened with $(node_libvirt_pod), as F1 does, because Step 6's libvirt_pod named the pod F1 killed. The times count from the applied at line or the scale-down, the pod times come from each new pod's Ready condition, and the API states from <f>.log, sampled about every 14 seconds. The outputs are on #1222.

Fault	Pod back Ready after	lab-0 and its QEMU	Guest network	Guest disk	Console	What the APIs and operators reported	Issue
F1 libvirt	5 s	ACTIVE; QEMU PID 202830 kept, libvirtd 196513 replaced by 214332	no gap, no lost line	no gap, no error line	the log kept growing; virsh console from the new pod answered with a prompt	no field moved; the gate held at 68 s	none
F2 nova-compute	4 s	ACTIVE; QEMU and libvirtd PIDs kept	no gap, no lost line	no gap, no error line	the log kept growing	no field moved, compute=up throughout; the share stayed mounted with the same options; the new pod logged no ERROR or Traceback; the gate held at 58 s	none
F3 ovn-controller	9 s	ACTIVE; QEMU and libvirtd PIDs kept	no gap, no lost line	no gap, no error line	the log kept growing	OVNChassis not Ready in the sample at 10 s and Ready again at 24 s; both agents of the node True before and after; the gate held at 52 s	none
F4 NFS server killed	24 s, moved from rt6kn to hmw67 after one Multi-Attach error	ACTIVE, domain running; QEMU PID 202830 kept until the hard reboot at 1867 s, after two failed gates	no lost line; the reboot cut it for 91 s	last ok 1 s before the kill; no line for 107 s, then error every 2 s from 106 s on, 875 lines, until the reboot; next ok at 1957 s	the log kept growing; the guest kernel logged I/O error, dev vdb	nothing moved: volume=in-use, cinder-volume=up, every kind Ready; the node's kernel logged lost 1 locks at 108 s	#1245
F5 NFS server scaled to 0	gone after 9 s; Ready 7 s after the scale-up at 303 s, on hmw67	ACTIVE, domain running; QEMU PID 317898 kept until the hard reboot at 2200 s, after two failed gates	no lost line; the reboot cut it for 69 s	last ok 1 s before the scale-down; no line for 414 s, with dd blocked past 120 and 241 s, then error every 2 s from 413 s on, 889 lines, until the reboot; next ok at 2269 s	the log kept growing; the guest kernel logged the blocked task and I/O error, dev vdb	nfs-server= empty during the hold, nothing else moved; the node's kernel logged not responding at 183 s, OK at 312 s and lost 1 locks at 415 s	#1245

A killed libvirt pod goes unnoticed by the servers on its node: QEMU keeps running, the console log keeps growing, and the console answers through the new pod within seconds. A killed nova-compute pod is back long before Nova misses its heartbeat, and the share it mounted stays mounted. A killed ovn-controller pod leaves the datapath in place: the guest lost no ping across the tunnel, and only the OVNChassis condition showed the restart. On the kernel's nfsd, the lab's NFS server in that run, a reschedule of the server pod stalled the attached volume for about two minutes, and then every request to it failed until the Nova server it was attached to was hard-rebooted, while every API reported the volume in-use and healthy: the restarted nfsd logged Unable to initialize client recovery tracking! (-22) and refused the reclaim of QEMU's lock. The longer outage stalled the guest's disk for as long as the server was away, with hung-task warnings in the guest, and then failed the same way once the server was back. #1245 replaced that server with NFS-Ganesha, which keeps its client records on the export claim (see NFS storage stack); F4 and F5 have not run on it yet. After F5, lab-vol-after was available within seconds and was deleted, so the backend served new volumes while the attached one failed. At the session's end last_probe disk printed ok 82, the volume file held probe-82, Part 2, Step 10 read lab-volume-marker and backed the volume up, and the deletes, proving step 6 and the namespace check left only the six platform namespaces.

Lab dizzy stack

File: deploy/lab/metal-stack/dizzy/kustomization.yaml

The metrics stack of a dizzy soak for the metal-stack lab (#1225, from the dizzy findings of #1219): the VictoriaMetrics and Grafana of the dizzy load/chaos stack, with the metrics on a volume. hack/deploy-infra.sh applies the directory in Step 3 when WITH_DIZZY=true is set beside EXTERNAL_CLUSTER=true, in place of deploy/kind/dizzy. The flag composes onto the deploy command of the Quick Start (metal-stack), which does not need it:

EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_DIZZY=true make deploy-infra

The kustomization takes deploy/kind/dizzy as its one resource, so its render holds the seven objects of the kind overlay: the Namespace dizzy, the HelmRepositories victoria-metrics and grafana, the HelmReleases dizzy-victoria-metrics and dizzy-grafana, the HTTPRoute dizzy-grafana and the ConfigMap grafana-dashboards. The first two rows below are its differences from kind; the storage follows the decision D7 of #1219.

Property	Value
Namespace label	apiserver-proxy.networking.gardener.cloud/inject: disable on dizzy, the lab's rule for every namespace it declares (see Lab overlay). The namespace carries no chaos-mesh.org/inject annotation: the stack measures a soak under faults, so no experiment of the Lab Chaos Mesh selects its pods
VictoriaMetrics	the StatefulSet dizzy-victoria-metrics-server keeps its metrics on the claim server-volume-dizzy-victoria-metrics-server-0, 10Gi on the cluster's default class, behind the headless ClusterIP Service dizzy-victoria-metrics-server on port 8428. On kind it keeps them in an emptyDir and publishes NodePort 30428
Inherited from kind	both HelmRepositories and both chart ranges; retentionPeriod: 30d and the two opentelemetry.* arguments of VictoriaMetrics; the Grafana release with anonymous Viewer access, the provisioned victoriametrics datasource and the three dashboards, without a volume; the HTTPRoute on the listener https-dizzy; the dashboard ConfigMap
Namespaces	dizzy for the two releases, the HTTPRoute and the ConfigMap, flux-system for the HelmRepositories
Pinned by	tests/unit/deploy/metal_stack_dizzy_test.sh

In this mode the deploy script's preflight accepts WITH_DIZZY=true only for an overlay with dizzy/kustomization.yaml and refuses any other before it contacts the cluster. Step 3 stages the three dashboards into deploy/kind/dizzy/dashboards/ with hack/dizzy.sh stage-dashboards, which the lab overlay reads as well, and applies <overlay>/dizzy. It calls no docker port. Phase 3 waits for both HelmReleases, as on kind; the VictoriaMetrics release is Ready once its claim is bound. The completion banner prints two dizzy: lines below its Access: lines.

Access. Nothing outside the cluster reaches the stack. A soak runs through two port-forwards from the workstation, each in a terminal of its own: the Gateway's on local port 8443, which serves Grafana at https://dizzy.127-0-0-1.nip.io:8443 beside Keystone, and one to VictoriaMetrics on local port 8428, which takes dizzy's OTLP export.

kubectl -n envoy-gateway-system port-forward \
  "$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=openstack-gw -o name)" 8443:443
kubectl -n dizzy port-forward svc/dizzy-victoria-metrics-server 8428:8428
EXTERNAL_CLUSTER=true make dizzy-keystone

With EXTERNAL_CLUSTER=true, hack/dizzy.sh takes the Keystone URL of the first port-forward, https://keystone.127-0-0-1.nip.io:8443/v3, and calls no docker. dizzy verifies Keystone against the Gateway's certificates, which the script collects from the Secrets openstack/*-nip-io-tls into _output/dizzy/gateway-ca.pem. When nothing answers on localhost:8428 it warns and names the second port-forward. Neither script opens a port-forward, so one that has ended is restarted by hand. dizzy Chaos Testing describes the soak and its dashboards.

Storage. The claim names no storage class and binds to the default class, which Step 1 of the deploy checks exists. A restarted pod mounts the same claim, so the metrics of a soak survive it. 10Gi holds the 30 days of retention many times over: a soak exports five metric families every 15 seconds. Grafana holds nothing a redeploy does not restore, so it has no volume.

Posture. Grafana answers anonymous Viewers, and VictoriaMetrics accepts writes and reads without authentication. Both sit behind ClusterIP Services that every pod of the cluster reaches and nothing outside it. The lab is one tenant's cluster.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes the stack at the end of its step 3, while the helm-controller still runs (see E2E Deployment). It deletes the two HelmReleases, so the helm-controller uninstalls both charts, then the two HelmRepositories, then the claims in dizzy, which Helm leaves behind. Where the default class has the reclaim policy Delete, the volume and the metrics on it go with the claim. Its step 7 deletes the namespace dizzy with the HTTPRoute and the ConfigMap. A HelmRelease delete that outlives TEARDOWN_TIMEOUT stops the teardown with exit 1 before any claim is deleted.

Lab dizzy soak

Files: deploy/lab/metal-stack/dizzy-soak/kustomization.yaml, identity.yaml, rbac.yaml, reports-pvc.yaml, job.yaml, scenario.yaml

The long-running dizzy soak of the lab (#1274): a Job in dizzy that runs dizzy mix chaos in the OpenStack project dizzy-soak and samples the platform beside it. make dizzy-soak-start applies the directory; hack/deploy-infra.sh never does. The soak needs the dizzy stack above, which declares the namespace dizzy, a Ready ControlPlane and the fixtures of Lab hypervisors. dizzy Chaos Testing describes the commands, the settings, the report and the verdict.

The kustomization renders the first nine objects below. start writes the Secrets and ConfigMaps and creates the Job from job.yaml, which the kustomization does not list; scenario.yaml is the default scenario, which start ships as a ConfigMap.

Object	Namespace	From	Purpose
K-ORC Domain dizzy-soak-domain	openstack	identity.yaml	Imports the domain Default, which stays on deletion
K-ORC Project dizzy-soak	openstack	identity.yaml	The project of the run, with the services' default quotas
K-ORC User dizzy-soak	openstack	identity.yaml	The run's Keystone user, its password from the Secret dizzy-soak-user-password
K-ORC Role dizzy-soak-admin-role	openstack	identity.yaml	Imports the role admin
K-ORC RoleAssignment dizzy-soak-admin	openstack	identity.yaml	admin for the user on the project, which the migration pre-check of dizzy's Legacy persona needs
ServiceAccount dizzy-soak	dizzy	rbac.yaml	The runner's identity in the cluster
ClusterRole and ClusterRoleBinding dizzy-soak-platform-reader	cluster	rbac.yaml	get and list on pods, namespaces, the pods of metrics.k8s.io, the ControlPlanes and the twelve service kinds
PersistentVolumeClaim dizzy-soak-reports	dizzy	reports-pvc.yaml	5Gi, ReadWriteOnce, on the default class; one report directory per run
Secret dizzy-soak-user-password	openstack	start, once	The key password, 32 characters
Secret dizzy-soak-clouds	dizzy	start	The clouds.yaml of the user, with interface: internal
ConfigMaps dizzy-soak-runner, dizzy-soak-scenario, dizzy-soak-config	dizzy	start	hack/dizzy-soak-runner.sh, the scenario and the settings
Job dizzy-soak	dizzy	job.yaml, by start	Its init container copies dizzy out of ghcr.io/b42labs/dizzy at the pin of hack/dizzy.sh; its container runner runs the runner on the digest-pinned docker.io/alpine/k8s
Pod dizzy-soak-reader	dizzy	report, for one copy	Mounts the claim read-only to copy a finished run

Reaching the APIs. The public endpoints of the lab's catalog resolve to 127.0.0.1, which leads nowhere from a pod. The co-located services also register an internal endpoint on their Service name, and Keystone answers in the cluster at http://controlplane-keystone.openstack.svc:5000/v3. The soak's clouds.yaml names that URL and interface: internal, so the run needs no CA file and no Gateway. The personas of dizzy mix build compute, network, block-storage and image clients, and each of the four services has an internal endpoint. dizzy exports its metrics to http://dizzy-victoria-metrics-server.dizzy.svc:8428/opentelemetry/v1/metrics.

Posture. The ClusterRole reads across the cluster with get and list alone and reads no Secret. The Keystone user holds admin on its own project, because dizzy's Legacy persona skips live migration without it. Its password lives in the Secret dizzy-soak-user-password, never in Git, an argument or a ConfigMap, and reaches Keystone over plain HTTP inside the cluster. The pods run as 65534 without privilege escalation or capabilities, and the reader pod mounts no ServiceAccount token.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes the soak after the Chaos Mesh step and before the hypervisors (see E2E Deployment). It first deletes the Job and the reader pod in the foreground, which ends a running soak with its report, then the render of the directory, while K-ORC still deletes the project and the user in Keystone. Its step 7 deletes the Secrets and ConfigMaps with their namespaces. A Job delete that outlives TEARDOWN_TIMEOUT stops the teardown with exit 1 before any hypervisor is touched.

Pinned by: tests/unit/deploy/metal_stack_dizzy_soak_test.sh.

Lab dizzy run

The run starts from a bare lab, where kubectl get namespace openstack dizzy flux-system answers NotFound for all three; a cluster that carries a stack is not deployed onto. It deploys with the command above, then runs Step 4, Step 5 and Step 6 of Part 1 of the Quick Start (metal-stack). The Gateway port-forward of Step 6 stays open, and step 3 below opens the VictoriaMetrics port-forward in another terminal. Steps 6 and 7 run within an hour of the soak's end, because their queries read the last hour. From the root of the clone:

# 1. A bare lab, then the deploy (then Part 1, Steps 4 to 6 of the quick start)
kubectl get namespace openstack dizzy flux-system
EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_DIZZY=true make deploy-infra

# 2. State: both releases with their chart versions, the label, the Service,
#    the claim and the default class
kubectl get helmrelease -n dizzy
kubectl get helmrelease -n dizzy \
  -o jsonpath='{range .items[*]}{.metadata.name} {.status.history[0].chartVersion}{"\n"}{end}'
kubectl get namespace dizzy -o jsonpath='{.metadata.labels}{"\n"}'
kubectl get svc dizzy-victoria-metrics-server -n dizzy -o jsonpath='{.spec.type} {.spec.ports[0].nodePort}{"\n"}'
kubectl get pvc -n dizzy
kubectl get storageclass

# 3. In another terminal:
#      kubectl -n dizzy port-forward svc/dizzy-victoria-metrics-server 8428:8428
curl -fsS http://localhost:8428/health; echo

# 4. The Keystone soak, the scenario's five minutes
EXTERNAL_CLUSTER=true make dizzy-keystone; echo "exit $?"

# 5. The dizzy_ metric names in VictoriaMetrics
curl -fsS http://localhost:8428/api/v1/label/__name__/values | jq -r '.data[]' | grep '^dizzy_'

# 6. Through Grafana on the Gateway port-forward: the dashboards, then the
#    expression each dashboard's panels read
curl -sk 'https://dizzy.127-0-0-1.nip.io:8443/api/search?type=dash-db' | jq -r '.[].uid' | sort
queries() {
  for metric in dizzy_iterations_total dizzy_operation_duration_seconds_count dizzy_resource_time_to_ready_seconds_count; do
    curl -sk -H 'Content-Type: application/json' https://dizzy.127-0-0-1.nip.io:8443/api/ds/query \
      -d "{\"queries\":[{\"refId\":\"A\",\"datasource\":{\"type\":\"prometheus\",\"uid\":\"victoriametrics\"},\"expr\":\"sum(last_over_time(${metric}[1h]))\",\"instant\":true}],\"from\":\"now-1h\",\"to\":\"now\"}" |
      jq -c --arg m "$metric" '{metric: $m, values: .results.A.frames[0].data.values}'
  done
}
queries

# 7. Storage: a new pod on the same claim, then the same queries
kubectl delete pod dizzy-victoria-metrics-server-0 -n dizzy
kubectl rollout status statefulset/dizzy-victoria-metrics-server -n dizzy --timeout=300s
queries

# 8. Stop both port-forwards, tear down, and wait up to 600 seconds for the
#    volume of the claim to go
EXTERNAL_CLUSTER=true make teardown-infra
kubectl get namespace dizzy
dizzy_pvs() {
  kubectl get pv -o json | jq '[.items[] | select(.spec.claimRef.namespace == "dizzy")] | length'
}
SECONDS=0
until [ "$(dizzy_pvs)" = 0 ] || [ "$SECONDS" -ge 600 ]; do sleep 5; done
echo "PersistentVolumes of dizzy: $(dizzy_pvs) after ${SECONDS}s"
rm _output/dizzy/clouds.yaml

What a run shows:

Step	Expected
1	three NotFound lines; the deploy exits 0, prints the two dizzy: lines and no predates the dizzy metrics port mapping warning
2	both HelmReleases Ready with their chart versions; the label apiserver-proxy.networking.gardener.cloud/inject: disable; ClusterIP and no node port; the claim server-volume-dizzy-victoria-metrics-server-0 Bound with 10Gi on the class kubectl get storageclass marks (default)
3	OK
4	exit 0; the log holds Keystone auth URL: https://keystone.127-0-0-1.nip.io:8443/v3, and neither a docker port line nor a WARNING
5	names of the five families of dizzy Chaos Testing: dizzy_operation_duration_seconds, dizzy_resource_time_to_ready_seconds, dizzy_iteration_duration_seconds, dizzy_iteration_operations_total and dizzy_iterations_total
6	dizzy-api-operations, dizzy-overview and dizzy-time-to-ready; three lines whose values is [[<ms>],[<number>]], none []
7	the rollout completes; the three lines hold the numbers of step 6
8	the teardown exits 0, logs Deleting the dizzy HelmReleases... and Deleting the PVCs in dizzy... and ends with Stack CRDs left: 0; stack namespaces left: 0; cluster-scoped chart objects left: 0; then NotFound, and 0 PersistentVolumes

No lab run of this stack is recorded yet.

Lab Prometheus stack

Files: deploy/lab/metal-stack/prometheus/kustomization.yaml, deploy/lab/metal-stack/prometheus/hypervisor-operator.json

The kube-prometheus-stack of the metal-stack lab (#1226, from the Prometheus findings of #1219): the Prometheus and Grafana of the kube-prometheus-stack (kind-only opt-in), with the metrics on a volume, the scrape jobs a Gardener shoot can feed, and a dashboard for openstack-hypervisor-operator (hvo). hack/deploy-infra.sh applies the directory in Step 3 when WITH_PROMETHEUS=true is set beside EXTERNAL_CLUSTER=true, in place of deploy/kind/prometheus. The flag composes onto the deploy command of the Quick Start (metal-stack), which does not need it:

EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_PROMETHEUS=true make deploy-infra

The kustomization takes deploy/kind/prometheus as its one resource, deletes its Namespace and patches its HelmRelease. Its render holds three objects in monitoring: the HelmRelease kube-prometheus-stack and the ConfigMaps keystone-operator-dashboard and hypervisor-operator-dashboard. The storage follows the decision D7 of #1219.

Property	Kind	Lab
Namespace monitoring	declared by the overlay, without labels	not in the render. The base overlay declares it with the label apiserver-proxy.networking.gardener.cloud/inject: disable and the annotation chaos-mesh.org/inject: enabled (see Lab overlay), and a second apply of the kind Namespace would remove both
Scrape jobs	the chart's defaults	kubeEtcd, kubeScheduler, kubeControllerManager and kubeProxy off. Gardener runs etcd, the scheduler and the controller manager in the seed. The chart's kube-proxy Service selects k8s-app: kube-proxy, while the shoot's kube-proxy pods carry app: kubernetes and role: proxy. Each of the four Services in kube-system would select nothing. apiserver, coredns and kubelet stay on
Retention	6h	7d, bounded by retentionSize: 8GB
Storage	emptyDir	a volume claim template of 10Gi, ReadWriteOnce, without a storageClassName
Rule selector	the chart's default, the label release: kube-prometheus-stack	ruleSelectorNilUsesHelmValues: false, which renders ruleSelector: {}: Prometheus loads every PrometheusRule, hvo's included, the posture the kind overlay takes for ServiceMonitors
Prometheus memory	request 256Mi, limit 512Mi, sized for one kind node	request 1Gi, limit 2Gi: the lab scrapes three kubelets with cAdvisor, and the request follows that working set
Dashboards	keystone-operator	keystone-operator and hypervisor-operator

Everything else is the kind overlay's: the chart range >=65.0.0 <70.0.0, the dependency on cert-manager, crds.enabled: false, Alertmanager, node-exporter and kube-state-metrics off, the open ServiceMonitor selectors, Prometheus's CPU request, Grafana's resources, and Grafana without a volume, since a redeploy restores everything it holds. With these values the chart renders one namespaced object outside monitoring, the Service kube-system/kube-prometheus-stack-coredns, which the uninstall removes. The Prometheus Operator writes a second Service there, kube-prometheus-stack-kubelet, with its Endpoints, for the kubelet job; no uninstall removes those two. tests/unit/deploy/metal_stack_prometheus_test.sh pins the render.

hypervisor-operator.json is a Grafana dashboard with the uid hypervisor-operator and four timeseries panels: reconciliation rate, reconciliation errors, reconciliation duration (p99) and workqueue depth. Their expressions are those of the controller-runtime panels of hvo's chart dashboard, each on job=~".*hypervisor-operator.*"; the job of hvo's target is its Service name, hypervisor-operator-controller-manager-metrics-service. The chart's own dashboard is a Perses dashboard, which Grafana cannot load, and in chart 1.2.3_sha-a2baf3f 12 of its 16 queries read kube_customresource_* series, which only a kube-state-metrics with the chart's custom-resource config exports. The file sits in the overlay directory, so the generator reads it without a staging step. The panels stay empty until Lab hypervisors are applied; hvo's release turns on its ServiceMonitor and its PrometheusRules on every lab. The pinned chart writes its version, which carries a +, into the label app.kubernetes.io/version of both rules, and the API server rejects that value. The release's post-renderer replaces the label (see the hvo rows of Lab hypervisors), so the rules exist and Prometheus loads the 10 alerts that step 4 of the run lists.

In this mode the deploy script's preflight accepts WITH_PROMETHEUS=true only for an overlay with prometheus/kustomization.yaml and refuses any other before it contacts the cluster. Step 3 stages the Keystone dashboard into deploy/kind/prometheus/keystone-operator.json, which the lab overlay reads through its resource, applies <overlay>/prometheus, and logs Prometheus overlay <overlay>/prometheus applied (WITH_PROMETHEUS=true).. Without the staged file the render fails, as on kind. Phase 3 waits for the HelmRelease with a timeout of at least 1200 seconds, as on kind. The deploy then turns on the ServiceMonitors of the nine service operators; under WITH_CONTROLPLANE=true their releases run by then, and it waits for each.

Access. Nothing outside the cluster reaches the stack. Prometheus and Grafana each take a port-forward from the workstation, in a terminal of its own:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Prometheus then answers on http://localhost:9090 and Grafana on http://localhost:3000, where it signs in admin with the chart's default password prom-operator, as on kind (see Extended Quick Start, Step 4c). A port-forward ends with the pod it reached and is restarted by hand.

Storage. The claim names no storage class and binds to the default class, which Step 1 of the deploy checks exists. A restarted Prometheus pod mounts the same claim, so the metrics survive it, up to 7 days or 8GB. A lab that already runs the release of the kind overlay gets a volume claim template added to a live Prometheus: the Prometheus Operator replaces the StatefulSet, and the metrics in the old emptyDir are lost.

Posture. Grafana keeps the chart's default admin password, and Prometheus answers without authentication. Both sit behind ClusterIP Services that every pod of the cluster reaches and nothing outside it, and the lab is one tenant's cluster. Prometheus sends its ServiceAccount token to hvo's metrics Service with insecureSkipVerify, the TLS setting of the chart's ServiceMonitor, so it does not check whom it hands the token to. The token reads nodes, Services, Endpoints, EndpointSlices, pods and Ingresses cluster-wide, as the ClusterRole kube-prometheus-stack-prometheus grants.

Teardown. EXTERNAL_CLUSTER=true make teardown-infra removes the stack in its step 3, after the base overlay and while the helm-controller still runs (see E2E Deployment). It deletes the HelmRelease by the file deploy/kind/prometheus/release.yaml, so the helm-controller uninstalls the chart, then the claims in monitoring, which Helm and the Prometheus Operator leave behind, then the Service and the Endpoints kube-prometheus-stack-kubelet in kube-system. Where the default class has the reclaim policy Delete, the metrics go with the claim. Its step 7 deletes the namespace monitoring with the two ConfigMaps. A release delete that outlives TEARDOWN_TIMEOUT stops the teardown with exit 1 before any claim is deleted. On a cluster deployed without the flag the three deletes find nothing.

Lab Prometheus run

The run starts from a bare lab, where kubectl get namespace openstack monitoring flux-system answers NotFound for all three; a cluster that carries a stack is not deployed onto. It saves the Services, Endpoints and EndpointSlices of kube-system, deploys with the command above, then runs Step 4, Step 5 and Step 6 of Part 1 and Step 1 to Step 4 of Part 2 of the Quick Start (metal-stack). Step 3 below runs at least ten minutes after the deploy and opens the Prometheus port-forward in another terminal; step 5 opens the Grafana one. From the root of the clone:

# 1. A bare lab, the objects of kube-system, then the deploy (then Part 1,
#    Steps 4 to 6 and Part 2, Steps 1 to 4 of the quick start)
kubectl get namespace openstack monitoring flux-system
kube_system_objects() {
  kubectl get service,endpoints,endpointslices -n kube-system -o name | sort
}
mkdir -p _output
kube_system_objects >_output/kube-system-before.txt
EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true WITH_PROMETHEUS=true make deploy-infra

# 2. State: the release with its chart version, the label and the annotation,
#    the claim and the default class, the chart's Services in kube-system, and
#    hvo's ServiceMonitor and PrometheusRules
kubectl get helmrelease kube-prometheus-stack -n monitoring
kubectl get helmrelease kube-prometheus-stack -n monitoring -o jsonpath='{.status.history[0].chartVersion}{"\n"}'
kubectl get namespace monitoring -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}'
kubectl get pvc -n monitoring
kubectl get storageclass
kubectl get service -n kube-system -o name | grep '^service/kube-prometheus-stack'
kubectl get servicemonitor,prometheusrule -n openstack

# 3. In another terminal, ten minutes after the deploy:
#      kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
#    Every active target: job, namespace, health, last error
targets() {
  curl -fsS 'http://localhost:9090/api/v1/targets?state=active' |
    jq -r '.data.activeTargets[] | [.labels.job, .labels.namespace, .health, .lastError] | @tsv' | sort
}
targets

# 4. The dashboard's four expressions as instant queries (the number of series
#    each returns), then hvo's alerts with their health
query() {
  curl -fsS http://localhost:9090/api/v1/query --data-urlencode "query=$1" --data-urlencode "time=${2:-$(date +%s)}" |
    jq '.data.result | length'
}
jq -r '.panels[].targets[].expr' deploy/lab/metal-stack/prometheus/hypervisor-operator.json |
  while IFS= read -r expr; do echo "$(query "$expr") ${expr}"; done
curl -fsS http://localhost:9090/api/v1/rules |
  jq -r '.data.groups[] | select(.file | test("openstack-hypervisor-operator-(operator|eviction)-alerts")) | .rules[] | [.name, .health] | @tsv'

# 5. In another terminal:
#      kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
curl -fsS -u admin:prom-operator 'http://localhost:3000/api/search?type=dash-db' | jq -r '.[].uid' | sort

# 6. Storage: note the time, replace the pod, restart the Prometheus
#    port-forward of step 3 once the rollout is done, then query `up` at the
#    noted time and read the new pod's restarts and memory
before="$(date +%s)"
kubectl delete pod prometheus-kube-prometheus-stack-prometheus-0 -n monitoring
kubectl rollout status statefulset/prometheus-kube-prometheus-stack-prometheus -n monitoring --timeout=300s
query up "$before"
kubectl get pod prometheus-kube-prometheus-stack-prometheus-0 -n monitoring \
  -o jsonpath='{range .status.containerStatuses[*]}{.name} {.restartCount}{"\n"}{end}'
kubectl top pod prometheus-kube-prometheus-stack-prometheus-0 -n monitoring --containers

# 7. Stop both port-forwards, tear down, compare kube-system with step 1, and
#    wait up to 600 seconds for the volume of the claim to go
EXTERNAL_CLUSTER=true make teardown-infra
kube_system_objects | diff _output/kube-system-before.txt - && echo "kube-system unchanged"
kubectl get namespace monitoring
monitoring_pvs() {
  kubectl get pv -o json | jq '[.items[] | select(.spec.claimRef.namespace == "monitoring")] | length'
}
SECONDS=0
until [ "$(monitoring_pvs)" = 0 ] || [ "$SECONDS" -ge 600 ]; do sleep 5; done
echo "PersistentVolumes of monitoring: $(monitoring_pvs) after ${SECONDS}s"
rm _output/kube-system-before.txt

What a run shows:

Step	Expected
1	three NotFound lines; the deploy exits 0 and logs Prometheus overlay <clone>/deploy/lab/metal-stack/prometheus applied (WITH_PROMETHEUS=true).
2	the HelmRelease Ready and its chart version; the label apiserver-proxy.networking.gardener.cloud/inject: disable and the annotation chaos-mesh.org/inject: enabled; one claim Bound with 10Gi on the class kubectl get storageclass marks (default); at most service/kube-prometheus-stack-coredns and service/kube-prometheus-stack-kubelet; the ServiceMonitor hypervisor-operator and the PrometheusRules openstack-hypervisor-operator-operator-alerts and openstack-hypervisor-operator-eviction-alerts
3	a target up in each of keystone-system, horizon-system, glance-system, placement-system, barbican-system, ovn-system, neutron-system, cinder-system and nova-system; the target of the job hypervisor-operator-controller-manager-metrics-service up without a last error; no target down
4	a count above 0 for the reconciliation rate and the workqueue depth; the 10 alerts of the two rules, each ok
5	hypervisor-operator and keystone-operator
6	the rollout completes; a count above 0 for up at the noted time; 0 restarts for every container; the memory of the prometheus container
7	the teardown exits 0, logs Deleting the PVCs in monitoring... and Deleting the kubelet Service and Endpoints the Prometheus Operator left in kube-system... and ends with Stack CRDs left: 0; stack namespaces left: 0; cluster-scoped chart objects left: 0; kube-system unchanged; then NotFound, and 0 PersistentVolumes

Three scrape jobs of the chart can only be judged on the lab: apiserver, whose target is the address of the shoot's kubernetes Endpoints, the apiserver-proxy address; coredns; and kubelet. A job with at least one target up ten minutes after the deploy stays. A job whose targets are all down is switched off in the overlay (kubeApiServer.enabled, coreDns.enabled, or kubelet.enabled together with prometheusOperator.kubeletService.enabled), the deploy runs a second time from the commit that switches it off, and the record names the job and its last error. A Prometheus container that was OOMKilled gets its memory limit doubled, and a memory in step 6 above the request becomes the request, rounded up to the next 256Mi; the record names the old and the new values. The overlay, its test and the table of the section above carry the values the run ends on.

No lab run of this stack is recorded yet.

Lab ControlPlane

Files: deploy/lab/metal-stack/controlplane/kustomization.yaml, deploy/lab/metal-stack/controlplane/ovncentral.yaml, deploy/lab/metal-stack/controlplane/controlplane-lab.yaml, deploy/lab/metal-stack/controlplane-nfs/kustomization.yaml

The kustomization is the OVNCentral of Step 3 and the ControlPlane CR of Step 4 of the Quick Start (ControlPlane), and the CR carries the cinder block of that page's optional # block-storage.yaml fragment with the Lab Ceph in place of its NFS shares: the RBD volume backend ceph1 and the RBD backup target cephbk. Apart from those two entries the data is unchanged except for two keys the lab adds to the ControlPlane. hack/deploy-infra.sh names the directory in its WITH_CONTROLPLANE=true completion hint and never applies it. Its preflight renders the directory and refuses it unless it holds exactly one ControlPlane, openstack/<CONTROLPLANE_NAME>, because Step 7 seeds the admin-password paths of that namespace and name only. The lab CR is openstack/controlplane, which the default name matches. Without WITH_CEPH=true the preflight refuses the directory as well, because the CR's ceph1 and cephbk name the monitor of the Lab Ceph, which only WITH_CEPH=true deploys; without the flag both would wait for their key Secrets, and the ControlPlane would never become Ready. tests/unit/deploy/metal_stack_controlplane_test.sh compares both files with three blocks of the page, the first # controlplane.yaml block, the # block-storage.yaml fragment without its backends and the # controlplane-ovn.yaml block, and fails when they drift apart. The CR file is not named controlplane.yaml, because .gitignore ignores that basename in every directory.

Setting	Value	Why
spec.sizing.profile	Minimal	The quick start's profile: one replica per component and a 512Mi database volume (operators/c5c3/api/v1alpha1/sizing_profiles.go), which fits a worker with 16 CPUs and 128 GiB. The Lightbits CSI driver rounds a request up to its 1 GiB granularity (getReqCapacity in pkg/driver/controller.go of LightBitsLabs/los-csi), so the 512Mi claim binds as a 1 GiB volume. A lab that needs a value between Minimal and Standard declares a cluster-scoped SizingProfile with spec.base: Minimal and selects it with spec.sizing.profileRef.name, which excludes spec.sizing.profile
spec.services.neutron.extraConfig.DEFAULT.global_physnet_mtu	"1460"	Tenant networks are Geneve. Neutron 27.0.3 computes their MTU as global_physnet_mtu minus 20 (the IPv4 header, get_mtu in neutron/plugins/ml2/drivers/type_tunnel.py) minus [ml2_type_geneve] max_header_size, which the Neutron operator owns at 38 (operators/neutron/internal/controller/reconcile_config.go): 1460 - 20 - 38 = 1402 for every tenant network. 1460 is the pod network's MTU (the cali* lines of the node probe). The chassis tunnels over the node network, whose uplinks lan0 and lan1 carry 9000, so 1460 is a bound, not a match: it holds without a path-MTU measurement between the two racks, and none exists yet. The value is a string, because extraConfig is map[string]map[string]string
spec.services.nova.hypervisorOperator	{}	Provisions the Keystone user hypervisor-operator (project service-hypervisor-operator, role admin) and writes the Secret controlplane-nova-hypervisor-operator-auth into openstack (see ServiceNovaHypervisorOperatorSpec), which Lab hypervisors feeds into the hypervisor operator's chart

Part 1 of the Quick Start (metal-stack) applies the directory and checks the ControlPlane, from the deploy in its Step 3 to the checks of its Step 7.

The Ceph entries name the pools, users and key Secrets of the Lab Ceph. A backup entry carries no secretUUID, because no hypervisor attaches a backup. The networks are the pod and service networks of forge and newforge, read from the ConfigMap kube-system/shoot-info on 2026-10-04 and 2026-10-11; another shoot takes its own two from there, which Part 1, Step 4 of the quick start reads before the apply. They open nothing on the lab. The c5c3-operator projects no spec.networkPolicy onto the Cinder, and the Cinder operator creates no NetworkPolicy for a Cinder without one. With one, the networks would open the Ceph ports to every pod and Service address of the shoot, which is where Rook's daemons answer. tests/unit/deploy/metal_stack_controlplane_test.sh pins both entries and checks their pools, users, key Secrets, monitor and libvirt secret UUID against the lab manifests they come from. Nothing checks the table below against the CR.

Field	ceph1	cephbk	Source
rbd.pool	volumes	backups	the CephBlockPools of ceph/cluster/pools.yaml
rbd.user	cinder	cinder-backup	the CephClients of ceph/cluster/clients.yaml
rbd.keySecretRef.name	ceph-client-cinder	ceph-client-cinder-backup	the targets of the ExternalSecrets of ceph/cluster/keys-pull.yaml, each with the one key userKey
rbd.monitors	rook-ceph-mon-a.rook-ceph.svc.cluster.local	the same	the Service Rook gives the first monitor in the CephCluster's namespace rook-ceph; ceph/cluster/cluster.yaml runs one (mon.count: 1)
rbd.networks	10.240.0.0/13, 10.248.0.0/18	the same	podNetwork and serviceNetwork of the kube-system/shoot-info of forge and newforge
rbd.secretUUID	090e4a3c-6c20-4e74-82dc-1a70382babe8	none: no hypervisor attaches a backup	the libvirt secret ceph-secret.sh defines in hypervisor/libvirt-configmap.yaml (see Lab hypervisors)
rbd.clusterName	unset	unset	the CinderBackend and CinderBackupBackend CRDs default it to ceph; the ControlPlane field has no default

controlplane-nfs/ is the NFS variant of the directory, for a lab deployed with WITH_NFS=true as well. It lists ../controlplane as its base and adds one JSON patch to the ControlPlane: the NFS volume backend nfs1 at the end of spec.services.cinder.backends, beside ceph1, on the share /volumes of the Lab NFS stack. nfs1 is the volume backend of the # block-storage.yaml fragment, which the test compares. A JSON patch keeps ceph1, because a strategic-merge patch on a custom resource is a JSON merge patch, which replaces the whole list. The backup target stays cephbk: the Ceph backup driver reads an NFS volume through the export the backup pod mounts, and the cinder-operator mounts the export of every NFS volume backend in that pod. Part 3, Step 1 of the quick start applies the variant with kubectl apply -k over the ControlPlane of Part 1, Step 4, behind a check that the NFS server and the CSIDriver nfs.csi.k8s.io exist. Without them the volume service of nfs1 and the backup pod wait for a mount nothing serves. Neither hack/deploy-infra.sh nor its preflight reads the variant.

The apply of controlplane/ is the way back: it replaces the backends list with ceph1 alone, the c5c3-operator prunes nfs1, and the webhook warns that every volume left on it can no longer be managed. Those volumes are deleted before that apply, and the volume type nfs of Part 3, Step 2 after them: its volume_backend_name names nfs1, so a volume created with it after the apply ends in error. The backups cephbk wrote of them stay in Ceph.

For the hypervisor package (#1142) the ControlPlane publishes the OVN central controlplane-ovn and three Secrets in openstack: the compute contract controlplane-nova-compute-config, the metadata proxy secret controlplane-nova-metadata-secret, and the hypervisor operator's credentials controlplane-nova-hypervisor-operator-auth. The auth Secret's auth_url is https://keystone.127-0-0-1.nip.io:8443/v3, the loopback URL the public catalog carries, and from inside a pod it resolves to the pod itself. The hypervisor operator of Lab hypervisors therefore takes the in-cluster URL http://controlplane-keystone.openstack.svc:5000/v3 for Keystone. With OS_INTERFACE=internal it takes the internal compute, placement, image and network endpoints of the catalog, which are the in-cluster Service URLs.

Property	Value
Namespace	openstack
Applied	by hand, after EXTERNAL_CLUSTER=true WITH_CONTROLPLANE=true WITH_CEPH=true make deploy-infra; the NFS variant by hand over it in Part 3 of the quick start, on a lab deployed with WITH_NFS=true as well
Storage	through the cluster's default class; neither CR names a storageClassName
Metadata gateway	none; the metadata API stays in-cluster at controlplane-nova-metadata.openstack.svc:8775
Block storage	Cinder with the RBD volume backend ceph1 on the pool volumes and the RBD backup target cephbk on the pool backups of the Lab Ceph. The NFS variant adds the NFS volume backend nfs1 on the share /volumes of the Lab NFS stack. CinderReady reports True with reason CinderReady
Removed by	EXTERNAL_CLUSTER=true make teardown-infra, which deletes the ControlPlane and then the OVNCentral in its first step
Pinned by	tests/unit/deploy/metal_stack_controlplane_test.sh

Lab autoscaling

The Gardener shoot under the lab runs a VerticalPodAutoscaler of its own, with the recommender, the updater and the admission controller, and a metrics-server, and the operators' opt-in blocks use both (#1223). These commands read what the platform provides; the comment above each gives what it printed on shoot newforge on 2026-10-05:

# the API server's version: v1.35.6; an in-place resize needs 1.33 or newer
kubectl version -o json | jq -r .serverVersion.gitVersion
# the update modes the VPA CRD admits: ["Off","Initial","Recreate","InPlaceOrRecreate","Auto"], no InPlace
kubectl get crd verticalpodautoscalers.autoscaling.k8s.io -o json |
  jq -c '.spec.versions[] | select(.name=="v1") | .schema.openAPIV3Schema.properties.spec.properties.updatePolicy.properties.updateMode.enum'
# the platform's own VPAs: six in kube-system, each in mode InPlaceOrRecreate
kubectl get vpa -A
# the resource metrics API: True under AVAILABLE, served by kube-system/metrics-server
kubectl get apiservice v1beta1.metrics.k8s.io
# the subresource an in-place resize writes: pods/resize
kubectl get --raw /api/v1 | jq -r '.resources[].name' | grep -x pods/resize

On kind, WITH_VPA=true installs the VPA CRDs with a recommender and WITH_METRICS_SERVER=true a metrics-server. On the shoot either would put a second copy beside the platform's, so make deploy-infra refuses both under EXTERNAL_CLUSTER=true, installs nothing, and names in its message the field that uses what the platform runs. The operators find the platform's VerticalPodAutoscaler CRD at startup, and a workload opts in on the ControlPlane:

• spec.sizing.<component>.<workload>.verticalAutoscaling renders a VPA for that workload (see VerticalAutoscalingSpec), which the platform's updater and admission controller act on.
• spec.sizing.<component>.api.autoscaling renders an HPA for that API (see AutoscalingSpec), which reads the pods' CPU from the platform's metrics-server.

The sizing measurement uses the platform's recommender as well, with VPAs in mode Off and neither block (see Lab measurement).

The Lab ControlPlane sets neither block. The block below patches the live ControlPlane, records what the platform did, and removes the patch again. It starts on a lab where Part 1 of the Quick Start (metal-stack) ran through Step 7, and runs in bash, in the shell of that step in the root of the clone, with the port-forward of Part 1, Step 6 open in a second terminal. Part 2 is not needed.

The VPA cases run on the Placement API, deployment/controlplane-placement with its one container placement-api. The Minimal profile gives it one replica and a CPU request of 15m, and an outage of Placement stops no reconcile of another operator, where an outage of Keystone would. Each case sets a minAllowed.cpu of its own, above the request the pod has. The recommender raises its target to minAllowed, so the pod is outside the recommendation at once, and the CPU request a pod ends up with names the case that set it. Each case also sets maxAllowed.cpu: 200m, the bound VPA Resource Mapping asks for whenever a mode lets the VPA change pods. A JSON merge patch merges objects, so every block names minReplicas, null where the case has none.

Case	Mode, applied through	minReplicas	minAllowed.cpu	Upstream's documented behaviour
V1	Off, the ControlPlane	unset	50m	the pod is unchanged, and its successor after a hand delete requests 15m
V2	Initial, the ControlPlane	unset	60m	the running pod is unchanged, and its successor requests 60m
V3	Recreate, the ControlPlane	unset	70m	no eviction: the updater's default floor is two replicas
V4	Recreate, the ControlPlane	1	80m	the pod is evicted, and its successor has a new UID and requests 80m
V5	InPlaceOrRecreate, a VPA applied by hand	unset	90m	no resize: the floor of two replicas holds for an in-place resize too
V6	InPlaceOrRecreate, the same VPA, patched	1	100m	the same UID and restart count, and 100m as the spec's and the running request

V5 and V6 apply the VPA by hand. The lab installs the operators from the charts published from main (deploy/flux-system/releases/c5c3-operator.yaml), and a release older than the mode refuses it in a ControlPlane block, so the two cases do not depend on the release the lab runs. The VPA lab-placement-inplace is the one BuildVPA renders for the mode, under a name of its own and without the CR's labels, so the operator's prune leaves it alone. On operators that offer the mode, a ControlPlane block with updateMode: InPlaceOrRecreate renders the same VPA under the name controlplane-placement.

watch_state writes a line to vpa.log every five seconds: the time, each Placement API pod that is not terminating (name, UID, restart count, the CPU request in its spec and the one the kubelet reports as running, and the memory request), the Deployment's ready replicas, the VPA's target for placement-api, and the reason of the Placement CR's VPAReady. Every wait is bounded. A target gets 600 seconds, a hold lasts 300 (five passes of an updater at upstream's default interval of one minute), VPAReady gets 120, and the ControlPlane's Ready 600. When await_target prints GATE FAILED, save kubectl describe vpa "$vpa" -n openstack and skip the case; when V1's gate fails, skip V2 to V6 as well. When kubectl patch or kubectl apply is refused, keep the API server's message and skip the case. When the ControlPlane is not Ready 600 seconds after a case, remove the block or the hand VPA and go on with the HPA part. A skipped case is recorded as not run, with its reason.

The HPA part runs on the Keystone API, deployment/controlplane-keystone, the API the e2e-autoscaling suite scales on kind. H1 sets the suite's autoscaling block without its processes and threads, and waits until the HPA reads a CPU utilization from the platform's metrics-server. H2 waits for one desired and one ready replica, because Keystone can scale out on its bring-up CPU, and applies the suite's load Job autoscaling-keystone-loadgen. It posts tokens from four threads for 900 seconds with the application credential of the Secret k-orc-clouds-yaml, whose auth_url is Keystone's in-cluster Service. H3 waits for three ready replicas and H4 for the Job's end. H5 waits for the scale-in to one replica, and before each poll deletes the Succeeded pods of Keystone's CronJobs: they carry the labels the Deployment selects on, and the HPA counts a pod without metrics at the full target. H6 removes the Job and the block.

Paste the block a section at a time, and read the output of each before the next.

auto="$(mktemp -d)"; echo "outputs in $auto"
ns=openstack
dep=controlplane-placement
sel='app.kubernetes.io/name=placement,app.kubernetes.io/instance=controlplane-placement,pod-template-hash'
flat() { tr '\n' ' ' | sed 's/ *$//'; }
millis() { jq -Rr 'if endswith("m") then (rtrimstr("m") | tonumber) else (tonumber * 1000 | floor) end'; }
pod() { # name uid restarts cpu(spec) cpu(running) memory(spec) of each Placement API pod that is not terminating
  kubectl get pod -n "$ns" -l "$sel" -o json 2>&1 | jq -r '.items[] | select(.metadata.deletionTimestamp == null)
    | [.metadata.name, .metadata.uid, (.status.containerStatuses[0].restartCount // "-"),
       (.spec.containers[0].resources.requests.cpu // "-"),
       (.status.containerStatuses[0].resources.requests.cpu // "-"),
       (.spec.containers[0].resources.requests.memory // "-")] | join(" ")' 2>&1 | flat
}
target() { # <vpa>: its capped CPU target for placement-api, nothing while there is none
  kubectl get vpa "$1" -n "$ns" -o json 2>/dev/null |
    jq -r '(.status.recommendation.containerRecommendations // [])[] | select(.containerName == "placement-api") | .target.cpu // empty'
}
state() { # the API omits readyReplicas at 0
  local ready
  ready="$(kubectl get deployment "$dep" -n "$ns" -o jsonpath='{.status.readyReplicas}' 2>&1 | flat)"
  printf '%s pod=[%s] ready=%s target=%s vpaready=%s\n' "$(date -u +%T)" "$(pod)" "${ready:-0}" "$(target "$vpa")" \
    "$(kubectl get placement "$dep" -n "$ns" -o jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}' 2>&1 | flat)"
}
watch_state() { while :; do state >>"$1"; sleep 5; done; }
vpa_block() { # <case> <verticalAutoscaling as JSON, or null>
  echo "$1 applied at $(date -u +%FT%TZ)" | tee -a "$auto/$1.log"
  kubectl patch controlplane controlplane -n "$ns" --type merge \
    -p "{\"spec\":{\"sizing\":{\"placement\":{\"api\":{\"verticalAutoscaling\":$2}}}}}"
}
await_target() { # <case> <floor in millicores>
  local t i secs=600
  for i in $(seq $((secs / 5))); do
    t="$(target "$vpa")"
    if [[ -n "$t" && "$(millis <<<"$t")" -ge "$2" ]]; then
      echo "$1 target $t at $(date -u +%FT%TZ)" | tee -a "$auto/$1.log"; return 0
    fi
    sleep 5
  done
  echo "GATE FAILED: $1 no target of ${2}m within ${secs} s" | tee -a "$auto/$1.log"; return 1
}
hold() { # <case> <seconds>
  local before after
  before="$(pod)"; sleep "$2"; after="$(pod)"
  echo "$1 before: $before" | tee -a "$auto/$1.log"
  echo "$1 after:  $after" | tee -a "$auto/$1.log"
  kubectl get vpa "$vpa" -n "$ns" -o yaml >"$auto/$1.vpa.yaml" 2>&1
  kubectl get pod -n "$ns" -l "$sel" -o yaml >"$auto/$1.pod.yaml" 2>&1
  kubectl get events -n "$ns" --sort-by=.lastTimestamp 2>&1 | grep -iE 'vpa|evict|resiz|placement' | tail -n 40 >"$auto/$1.events" || true
}
recreate() { # <case>: delete the pod by hand and read its successor
  kubectl delete pod -n "$ns" -l "$sel" --wait --timeout=120s
  kubectl wait pod -n "$ns" -l "$sel" --for=condition=Ready --timeout=300s
  echo "$1 recreated: $(pod)" | tee -a "$auto/$1.log"
}

# Baseline: one pod that requests 15m, and the timeline of every case
pod | tee "$auto/baseline"
vpa=controlplane-placement; watch_state "$auto/vpa.log" & watcher=$!

# V1: Off
vpa_block V1 '{"updateMode":"Off","minReplicas":null,"minAllowed":{"cpu":"50m"},"maxAllowed":{"cpu":"200m"}}'
kubectl wait placement/controlplane-placement -n openstack --for=jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}'=VPAReady --timeout=120s
await_target V1 50
hold V1 300
recreate V1
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# V2: Initial
vpa_block V2 '{"updateMode":"Initial","minReplicas":null,"minAllowed":{"cpu":"60m"},"maxAllowed":{"cpu":"200m"}}'
kubectl wait placement/controlplane-placement -n openstack --for=jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}'=VPAReady --timeout=120s
await_target V2 60
hold V2 300
recreate V2
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# V3: Recreate at the updater's own replica floor
vpa_block V3 '{"updateMode":"Recreate","minReplicas":null,"minAllowed":{"cpu":"70m"},"maxAllowed":{"cpu":"200m"}}'
kubectl wait placement/controlplane-placement -n openstack --for=jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}'=VPAReady --timeout=120s
await_target V3 70
hold V3 300
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# V4: Recreate with minReplicas 1
vpa_block V4 '{"updateMode":"Recreate","minReplicas":1,"minAllowed":{"cpu":"80m"},"maxAllowed":{"cpu":"200m"}}'
kubectl wait placement/controlplane-placement -n openstack --for=jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}'=VPAReady --timeout=120s
await_target V4 80
hold V4 300
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# V5: the block goes, and InPlaceOrRecreate comes as a VPA applied by hand
vpa_block V5 null
kubectl wait --for=delete vpa/controlplane-placement -n openstack --timeout=120s
kubectl wait placement/controlplane-placement -n openstack --for=jsonpath='{.status.conditions[?(@.type=="VPAReady")].reason}'=VPANotRequired --timeout=120s
kill "$watcher"; vpa=lab-placement-inplace; watch_state "$auto/vpa.log" & watcher=$!
echo "V5 lab-placement-inplace applied at $(date -u +%FT%TZ)" | tee -a "$auto/V5.log"
kubectl apply -f - <<'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: lab-placement-inplace
  namespace: openstack
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: controlplane-placement
  updatePolicy:
    updateMode: InPlaceOrRecreate
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        controlledValues: RequestsOnly
        controlledResources: [cpu, memory]
        minAllowed:
          cpu: 90m
        maxAllowed:
          cpu: 200m
EOF
await_target V5 90
hold V5 300
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# V6: InPlaceOrRecreate with minReplicas 1
echo "V6 applied at $(date -u +%FT%TZ)" | tee -a "$auto/V6.log"
kubectl patch vpa lab-placement-inplace -n openstack --type merge -p '{"spec":{"updatePolicy":{"minReplicas":1},"resourcePolicy":{"containerPolicies":[{"containerName":"*","controlledValues":"RequestsOnly","controlledResources":["cpu","memory"],"minAllowed":{"cpu":"100m"},"maxAllowed":{"cpu":"200m"}}]}}}'
await_target V6 100
hold V6 300
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s

# The VPA part's end: without a VPA, the next pod requests 15m again
kubectl delete vpa lab-placement-inplace -n openstack --wait --timeout=60s
kill "$watcher"
recreate end
pod

# The HPA part, on the Keystone API
ks=controlplane-keystone
hpa_line() {
  kubectl get hpa -n "$ns" -l "app.kubernetes.io/instance=$ks" -o json 2>&1 |
    jq -r '.items[0] | "hpa=\(.metadata.name) target=\(.spec.scaleTargetRef.name) desired=\(.status.desiredReplicas) current=\(.status.currentReplicas) cpu=\(.status.currentMetrics[0].resource.current.averageUtilization)"' 2>&1 | flat
}
hpa_state() { printf '%s %s ready=%s\n' "$(date -u +%T)" "$(hpa_line)" "$(kubectl get deployment "$ks" -n "$ns" -o jsonpath='{.status.readyReplicas}' 2>&1 | flat)"; }
drop_job_pods() { # the HPA counts a pod without metrics at the full target (tests/e2e-autoscaling/keystone-hpa.sh)
  kubectl delete pods -n "$ns" --ignore-not-found --field-selector=status.phase==Succeeded \
    -l "app.kubernetes.io/name=keystone,app.kubernetes.io/instance=$ks,job-name" >/dev/null
}
await_hpa() { # <label> <seconds> <extended regex the hpa_state line must match> [command to run before each poll]
  local i
  for i in $(seq $(($2 / 10))); do
    if [[ -n "${4:-}" ]]; then "$4"; fi
    if hpa_state | tee -a "$auto/hpa.log" | grep -Eq "$3"; then echo "$1 at $(date -u +%FT%TZ)" | tee -a "$auto/hpa.log"; return 0; fi
    sleep 10
  done
  echo "GATE FAILED: $1 not within $2 s" | tee -a "$auto/hpa.log"; return 1
}

# H1: an HPA on the Keystone API, reading the platform's metrics-server
echo "H1 applied at $(date -u +%FT%TZ)" | tee -a "$auto/hpa.log"
kubectl patch controlplane controlplane -n "$ns" --type merge \
  -p '{"spec":{"sizing":{"keystone":{"api":{"autoscaling":{"minReplicas":1,"maxReplicas":3,"targetCPUUtilization":150,"behavior":{"scaleDown":{"stabilizationWindowSeconds":15,"policies":[{"type":"Percent","value":100,"periodSeconds":15}]}}}}}}}}'
await_hpa metric 300 'cpu=[0-9]+ '
kubectl top pods -n openstack -l "app.kubernetes.io/instance=$ks"

# H2: one replica, then the token load
await_hpa idle 300 'desired=1 .* ready=1$'
kubectl apply -f tests/e2e-autoscaling/02-keystone-loadgen-job.yaml
echo "loadgen applied at $(date -u +%FT%TZ)" | tee -a "$auto/hpa.log"

# H3: the scale-out to three ready replicas
await_hpa scale-out 600 'desired=3 .* ready=3$'

# H4: the Job's end and its RESULT line
for i in $(seq 130); do
  done_count="$(kubectl get job autoscaling-keystone-loadgen -n openstack -o jsonpath='{.status.succeeded}{.status.failed}' 2>&1)"
  if [[ "$done_count" =~ ^[0-9]+$ ]]; then break; fi
  hpa_state >>"$auto/hpa.log"
  sleep 10
done
echo "loadgen ended at $(date -u +%FT%TZ): ${done_count}" | tee -a "$auto/hpa.log"
kubectl logs job/autoscaling-keystone-loadgen -n openstack --tail=60 >"$auto/loadgen.log" 2>&1
grep RESULT "$auto/loadgen.log"

# H5: the scale-in to one replica
await_hpa scale-in 600 'desired=1 .* ready=1$' drop_job_pods

# H6: the Job and the block go, and one ready replica stays
hpa="$(kubectl get hpa -n "$ns" -l "app.kubernetes.io/instance=$ks" -o jsonpath='{.items[0].metadata.name}')"; echo "hpa ${hpa}"
kubectl delete job autoscaling-keystone-loadgen -n openstack
kubectl patch controlplane controlplane -n "$ns" --type merge -p '{"spec":{"sizing":{"keystone":{"api":{"autoscaling":null}}}}}'
kubectl wait --for=delete "hpa/${hpa}" -n openstack --timeout=120s
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=600s
hpa_state | tee -a "$auto/hpa.log"

The last line prints ready=1 and no HPA. Then run the last line of the quick start's Teardown, EXTERNAL_CLUSTER=true make teardown-infra; its other lines delete objects of Part 2, which this run does not create. Stop the port-forward, delete gateway-ca.pem, and check that kubectl get namespaces -o name lists the platform's namespaces alone (default, firewall, kube-node-lease, kube-public, kube-system and metallb-system) and that kubectl get vpa -A lists no VPA outside kube-system.

A result that differs from the last column of the case table is a finding: the platform's updater runs in the seed, whose flags cannot be read from the shoot, so the record states what it did. Each of these is a defect and gets its own issue: more than one new pod UID within one hold; a successor in V2 or V4, or a resized pod in V6, whose CPU request is below the case's floor; a vpaready= other than VPAReady while a block is set, or other than VPANotRequired after it is removed; a Placement or ControlPlane CR that does not return to Ready; a pod that requests other than 15m after the VPA part's end; GATE FAILED: metric; an HPA or Deployment left after H6; anything the teardown leaves; and a hand action neither the block nor the quick start names. A failed load Job with a scale-out that held is a finding.

The run of 2026-10-05 ran the block unchanged on shoot newforge, on Kubernetes v1.35.6, from commit d34fde24 (d34fde24334c2809ba2225654c206af9f217eee9), in the session of #1224. The VPA CRD admitted ["Off","Initial","Recreate","InPlaceOrRecreate","Auto"]. The block started from Part 2 of the Quick Start (metal-stack) with its servers, volume, backup and network deleted by the first five lines of the Teardown, and it was pasted a section at a time, each case in two parts split at its hold line. The baseline pod, 4890bf21, requested 15m. No gate failed, and nothing was done by hand. The outputs are in two comments on #1223, the VPA cases and the HPA part. In the table a pod is named by the first eight characters of its UID, and the seconds with ready=0 count from the first vpa.log line with ready=0 to the next one with ready=1; the lines are about five seconds apart.

Case	Mode and minReplicas	Pod over the hold (UID, restarts, CPU request in the spec and running)	Successor after a hand delete	Seconds with ready=0	Finding or issue
V1	Off, unset	4890bf21, 0, 15m and 15m, before and after	fa529e0c, 15m	22, after the hand delete	none; as documented
V2	Initial, unset	fa529e0c, 0, 15m and 15m, before and after	1271bf3e, 60m, memory request 109814751	34, after the hand delete	finding: the successor's memory request is the target, 105 MiB in place of 368Mi
V3	Recreate, unset	5c28ca0b, 0, 70m and 70m, before and after; the updater had evicted 1271bf3e between 16:10:03 and 16:10:08, within 11 seconds of the patch	not run, no hand delete in V3	12, after the eviction	finding: an eviction at one replica, because Gardener's updater runs with --min-replicas=1 (computeUpdaterArgs, pkg/component/autoscaling/vpa/updater.go at gardener/gardener@92a252c0)
V4	Recreate, 1	before 5c28ca0b, 0, 70m and 70m; after 3653d9dd, 0, 93m and 93m	not run, no hand delete in V4	11, after the eviction	none; the successor requests the recommender's own target, 93m, above the floor of 80m
V5	InPlaceOrRecreate, unset	3653d9dd, 0, 93m and 93m, before and after	not run, no hand delete in V5	0	finding: 93m lay inside the VPA's range (lower bound 90m, upper bound 200m), so the case shows nothing about a replica floor
V6	InPlaceOrRecreate, 1	3653d9dd, 0, 100m and 100m, before and after; resized in place between 16:28:05 and 16:28:11, before the hold	not run, no hand delete in V6	0	none; as documented

The applied at and target lines of the cases: V1 at 15:57:46Z with 50m at 15:58:13Z, V2 at 16:03:58Z with 60m at 16:04:09Z, V3 at 16:09:57Z with 70m at 16:10:13Z, V4 at 16:15:44Z with 93m at the same second, V5 at 16:21:05Z, its hand VPA at 16:21:06Z, with 90m at 16:22:09Z, and V6 at 16:27:28Z with 100m at 16:28:10Z.

vpaready= read VPAReady from V1's patch to V5's, and VPANotRequired from vpa_block V5 null on; the operator's VPA controlplane-placement was gone before V5's hand VPA was applied. The pod after the VPA part's end, 0627f3e1, requests 15m again. What a deployer on a single-replica profile gets, case by case:

• V1: with Off the pod keeps its request, and a successor starts with the rendered request as well.
• V2: with Initial the running pod keeps its request, and the next pod starts with the recommendation for both resources, here 60m and 105 MiB, under the unchanged memory limit of 368Mi.
• V3: with Recreate and no minReplicas, the platform's updater evicts the only pod as soon as its request lies outside the recommendation, and the component has no ready pod until the successor is ready, 12 seconds here.
• V4: with minReplicas: 1 the same happens, 11 seconds here.
• V5: with InPlaceOrRecreate nothing happens while the request lies inside the recommended range.
• V6: once the request lies outside the range, the pod is resized in place, with the same UID, no restart and no time without a ready pod.

The HPA part read a CPU utilization at 16:34:32Z (metric), 31 seconds after H1's patch, and one ready replica at 16:34:39Z (idle), when the load Job was applied. The first line above one replica came 26 seconds after the apply and read desired=3; no line with desired=2 exists, so the HPA went from one to three in one step. Three pods were ready at 16:35:46Z (scale-out), 67 seconds after the apply. The peak reading was cpu=5693, in percent of the 15m request. The Job ended at 16:49:50Z with RESULT total=6289 ok=6289 fail=0, and the Deployment was back at one ready replica 133 seconds later, at 16:52:03Z (scale-in). After H6, hpa_state printed no HPA and ready=1. EXTERNAL_CLUSTER=true make teardown-infra then exited 0 and left the platform's namespaces alone and no VPA outside kube-system. The run found none of the defects listed above.

Lab hypervisors

Files: deploy/lab/metal-stack/hypervisor-fixtures/kustomization.yaml, deploy/lab/metal-stack/hypervisor/kustomization.yaml and the seven manifests it lists, deploy/lab/metal-stack/migration-ports/kustomization.yaml and the two manifests it lists

hypervisor-fixtures/ and hypervisor/ turn every labelled worker of the lab into a KVM hypervisor of the Lab ControlPlane, planned in #1142. They add libvirt in a DaemonSet, the OVN chassis, the metadata agent and a NovaCompute pool, and run openstack-hypervisor-operator (hvo) and kvm-node-agent (kna) from upstream with the settings that carry both on a Debian node under Gardener. Both are applied by hand once the ControlPlane is Ready. hypervisor/ pulls in migration-ports/, which reserves QEMU's migration ports on every node and which Part 2, Step 1 of the Quick Start (metal-stack) applies on its own before the node port check. hack/deploy-infra.sh applies none of the three. tests/unit/deploy/metal_stack_hypervisor_test.sh pins the three renders.

File	Content
hypervisor-fixtures/kustomization.yaml	The kind fixtures of deploy/kind/hypervisor-operator-fixtures/ without VolumeType/hvo-premium, Network/hvo-smoke-test and Subnet/hvo-smoke-test: only hvo's smoke test uses them, and every lab Hypervisor skips it through the node label cobaltcore.cloud.sap/node-hypervisor-lifecycle=skip-tests. The domain cc3test and the project test, which hvo scopes a token to at start, stay, and so do the flavor ID 1 (1 vCPU, 256 MiB, 1 GiB) and the image cirros-kvm the run boots
migration-ports/namespace.yaml	Namespace hypervisor-system, with the lab's Gardener opt-out label
migration-ports/reservation-daemonset.yaml	DaemonSet migration-port-reservation in hypervisor-system, on every node: it reserves QEMU's migration ports (see Migration port reservation)
hypervisor/kustomization.yaml	Takes ../migration-ports as a resource, so the hypervisor overlay applies the namespace and the reservation too and the teardown removes both with it
hypervisor/libvirt-ca.yaml	Certificate libvirt-migration-ca (ECDSA 256, three years, bootstrapped from selfsigned-cluster-issuer) and Issuer nova-hypervisor-agents-ca-issuer, hvo's default issuer name, in hypervisor-system. The CA signs nothing else
hypervisor/libvirt-configmap.yaml	ConfigMap libvirt-lab: host-prepare.sh, libvirtd.sh, ceph-secret.sh, libvirtd.conf and qemu.conf
hypervisor/libvirt-daemonset.yaml	DaemonSet libvirt in openstack
hypervisor/compute.yaml	OVNChassis/lab-chassis on controlplane-ovn, NeutronMetadataAgent/lab-metadata-agent on the in-cluster Nova metadata API, and NovaCompute/lab with virtType: kvm, cpuMode: custom, cpuModels: [Skylake-Server-IBRS] and imagesType: qcow2, all in openstack
hypervisor/sources.yaml	One digest-pinned OCIRepository per chart in flux-system
hypervisor/hvo-release.yaml	HelmRelease/openstack-hypervisor-operator in openstack
hypervisor/kna-release.yaml	HelmRelease/kvm-node-agent in hypervisor-system

The pods the overlay puts on a hypervisor node in openstack take their requests and limits from these sources:

Pod	Containers	Requests and limits
lab-nova-compute	nova-compute, create-instances-dir, wait-for-chassis	the Nova operator's default, 150m CPU request and 768Mi memory request and limit (Resources)
lab-metadata-agent-metadata-agent	metadata-agent, wait-for-chassis	the Neutron operator's default, 230m CPU request and 2Gi memory request and limit (Memory sizing)
libvirt	libvirtd, host-prepare, ceph-secret	the DaemonSet's own: 100m/256Mi request and 512Mi limit for libvirtd, 10m/16Mi request and 64Mi limit for host-prepare and for ceph-secret
lab-chassis-ovs, lab-chassis-ovn-controller	ovsdb-server, ovs-vswitchd, host-prepare; ovn-controller, apply-node	none of their own: the tenant LimitRange of openstack (OpenBao Proving Instance, Tenant) fills in a 100m CPU and 128Mi memory request and a 512Mi memory limit; the lab read 11 to 23 MiB per container (Recorded lab run)

OVNChassis renders no resources by design (OVNChassis CRD), so a lab that changes the tenant LimitRange changes the chassis pods with it.

The libvirt DaemonSet runs ghcr.io/c5c3/libvirt:<tag>@sha256:<digest> (see libvirt) on the nodes labelled openstack.c5c3.io/nova-compute-pool=lab, the pool's own label, as uid 0 in the host's network, PID and IPC namespaces; host-prepare and libvirtd run privileged, ceph-secret does not (below). <tag> is a keeper tag: <libvirt-version>-r<N>, such as 12.2.0-r57, which hack/ci-tag-libvirt-keeper.sh creates once on main and never moves, where N counts the commits under images/libvirt, nix/pkgs/libvirt.nix, nix/lib/image-defaults.nix and flake.lock (see Release-independent images), or <libvirt-package-version>-r<N>, such as 10.0.0-2ubuntu8.19-r2, a keeper of the Docker-built image the lab ran before #1407 and rolls back to. imagePullPolicy: IfNotPresent pulls the digest once per node. Renovate proposes each new keeper tag, a libvirt major included, as one pull request, never automerged; the lab never pins a branch build, whose <sha> tag the cleanup removes after 24 hours. The pull request moves all eight lines naming the image: the three containers here, the load test of the Node probe, both containers of nfs-client-modules in the Lab NFS stack, and both containers of chaos-mesh-modules in the Lab Chaos Mesh. The update strategy is OnDelete and the pod has no liveness probe: a rollout, or a restart on a slow answer, would interrupt running migrations. A merged bump therefore reaches a node when its libvirt pod is deleted. The readiness probe runs virsh -c qemu:///system version.

Host path	Why the pod mounts it
/run/libvirt	libvirtd's sockets, which nova-compute and kna open
/var/lib/libvirt, /var/lib/nova	The domains' state and the instance disks; /var/lib/nova with Bidirectional propagation, like the NovaCompute pod. The NFS share nova-compute mounts below /var/lib/nova/mnt for a Cinder volume appears in the libvirt pod through this propagation
/etc/pki/CA, /etc/pki/libvirt, /etc/pki/qemu	The TLS files kna writes, read-only
/dev, /sys/fs/cgroup, /lib/modules	/dev/kvm and the guests' devices, their cgroups, and the module tree for vhost_net, read-only
/run/systemd, /run/dbus	The host's systemd (its private socket and /run/systemd/system) and its D-Bus socket, the latter read-only

The init container host-prepare loads vhost_net and fails the pod with host-prepare: cannot load vhost_net from /lib/modules/<kernel> or host-prepare: /dev/kvm is missing on this node. The libvirtd container's libvirtd.sh then:

1. waits, without a timeout, for /etc/pki/CA/cacert.pem, /etc/pki/libvirt/servercert.pem and /etc/pki/libvirt/private/serverkey.pem, and logs libvirtd: waiting for the TLS files kvm-node-agent installs under /etc/pki every 10 seconds. A pod that is Running but not Ready waits here;
2. copies libvirtd.conf and qemu.conf to /etc/libvirt/, with the node's address in listen_addr;
3. stops cobaltcore-libvirtd.scope, which holds a libvirtd an earlier container left behind when it was killed after its grace period, and resets its failed state: a stop that runs out leaves the scope failed, and a failed unit keeps its name. It then starts systemd-run --collect --scope --slice=system --unit=cobaltcore-libvirtd libvirtd --listen. The scope moves libvirtd out of the pod's cgroup into the host's system.slice, so the QEMU processes it forks survive a restart of the pod. --collect lets systemd unload the scope even when it ends failed;
4. waits up to 60 seconds for /run/libvirt/libvirt-sock while libvirtd lives, exits with libvirtd's status when it dies, and exits 1 with libvirtd: /run/libvirt/libvirt-sock did not appear within 60s otherwise;
5. writes two runtime units into the host's /run/systemd/system, reloads the host's systemd and starts libvirtd.service;
6. on TERM stops that unit, deletes both, reloads, and stops libvirtd. The domains keep running. Every other end of the script, a failed step or libvirtd's own exit included, deletes the units as well and stops the scope.

kna reads libvirt's state from the host's systemd. It reports nothing about libvirt while libvirtd.service is not active, and it sets TLSCertificateInstalled=True only once starting virt-admin-server-update-tls.service succeeds. A Debian host without libvirt has neither unit, so the script writes stand-ins. Each starts with the line # Written by the cobaltcore lab libvirt DaemonSet (openstack/libvirt); removed when its pod stops.

Unit	Content
libvirtd.service	Type=simple, ExecStart=/usr/bin/tail --pid=<libvirtd PID> -f /dev/null: active as long as the containerized libvirtd lives
virt-admin-server-update-tls.service	Type=oneshot, ExecStartPre=/usr/bin/grep -q /cobaltcore-libvirtd.scope /proc/<libvirtd PID>/cgroup, ExecStart=/usr/bin/nsenter --target <libvirtd PID> --mount -- /usr/bin/virt-admin server-update-tls libvirtd: kna starts it after every certificate write. The check fails the start once the PID has left the scope, so a unit left behind never runs as host root in another process's mount namespace

This bends decision D1 of #1138, that the host is not modified, knowingly. /run is a tmpfs, so a reboot removes the units, and the script removes them whenever it ends. systemctl takes a container in the host's PID namespace for a chroot and ignores start and daemon-reload there, so the script sets SYSTEMD_IGNORE_CHROOT=1.

The third container, ceph-secret, defines the libvirt secret QEMU authenticates to Ceph with when it attaches a Cinder volume on RBD, and keeps it in step with the key in the Secret openstack/ceph-client-cinder of the Lab Ceph. It runs ceph-secret.sh from the same image pin as uid 0, which reads userKey, root-owned with mode 0400, and reaches the socket of the group +108 (below). It is not privileged, drops every capability and sets allowPrivilegeEscalation: false: virsh over the Unix socket needs no device, module or host namespace. It mounts the ConfigMap and /run/libvirt, and the Secret ceph-client-cinder read-only at /etc/ceph-client-cinder as a volume with optional: true, mode 0400 and no subPath. The pod starts without the Secret, and the kubelet writes userKey once the Secret appears, rewrites it on every change and removes it when the Secret is deleted, each on its periodic sync of about a minute. A watch through the API would need a ServiceAccount, a Role on Secrets in openstack and a token in a pod that shares the host's namespaces, so the pod keeps automountServiceAccountToken: false. The container has no probe, and the pod's readiness stays libvirtd's: on a lab without WITH_CEPH=true it waits and the pod is Ready.

The secret's UUID is 090e4a3c-6c20-4e74-82dc-1a70382babe8, its usage ceph with the name client.cinder. The UUID is a constant on both sides: ceph1, the RBD volume backend of the Lab ControlPlane, sets the same value as spec.rbd.secretUUID, so the rbd_secret_uuid Cinder sends names this secret (see CinderBackend CRD). A backend that leaves secretUUID unset sends the cluster FSID and finds no secret on the lab. Nova stores the UUID in the connection_info of every volume attachment; a start or a reboot of the server reuses it, and a migration or nova-manage volume_attachment refresh renews it. The UUID therefore stays fixed once a volume is attached: a change needs both UUIDs defined on every hypervisor until every attachment has been refreshed. The secret is private='yes', so libvirt refuses virsh secret-get-value on it, and ephemeral='yes', so libvirtd keeps the value in memory and writes no /etc/libvirt/secrets/<uuid>.base64. That file would hold the key in base64, readable by QEMU, which runs as root, and kept on the node's disk in the container's writable layer.

Once /run/libvirt/libvirt-sock exists, the script runs a pass every 10 seconds. It reads userKey with its whitespace removed; a missing or empty file counts as absent. With a key, it defines the secret when virsh secret-dumpxml finds none and sets its value with virsh secret-set-value --file whenever the key's digest differs from the one it set last. --file reads the base64 text and stores the decoded key, so the key never appears on a command line, which hostPID: true would show to every process on the node. Without a key, it undefines a secret libvirtd still has: the Secret goes when the WITH_CEPH=true stack is torn down or its ExternalSecret is deleted. The log names the first twelve hex characters of the key's SHA-256 and never the key, as the checks of the Lab Ceph do, and each line starts with ceph-secret::

When	kubectl logs -n openstack <pod> -c ceph-secret shows
At start	ceph-secret: libvirt secret 090e4a3c-6c20-4e74-82dc-1a70382babe8 for client.cinder follows the Secret openstack/ceph-client-cinder
Before libvirtd opens its socket, once	ceph-secret: waiting for /run/libvirt/libvirt-sock
The secret was defined	ceph-secret: defined libvirt secret 090e4a3c-6c20-4e74-82dc-1a70382babe8 for client.cinder
Its value was set, first and after each rotation	ceph-secret: set the value of libvirt secret 090e4a3c-6c20-4e74-82dc-1a70382babe8 from openstack/ceph-client-cinder (sha256 <first 12 hex characters>)
The Secret went and the secret was undefined	ceph-secret: undefined libvirt secret 090e4a3c-6c20-4e74-82dc-1a70382babe8 because openstack/ceph-client-cinder is gone
No key, once per change to that state	ceph-secret: waiting for the Secret openstack/ceph-client-cinder (deployed by WITH_CEPH=true); no libvirt secret is defined
A virsh call failed (libvirtd restarting, the socket gone), after virsh's error; the next pass tries again	ceph-secret: virsh <subcommand> failed; retrying in 10s

The digest of the set the value line equals kubectl get secret -n openstack ceph-client-cinder -o jsonpath='{.data.userKey}' | base64 -d | tr -d '[:space:]' | sha256sum | cut -c1-12. A rotated key reaches libvirtd within the kubelet's sync of about a minute and the next 10-second pass, and the log of each node's ceph-secret container is where it shows: a further set the value line with the new digest and no new defined line. On 2026-10-10 a scratch kind cluster with this container beside a libvirtd took 62 seconds from a new Secret to the set the value line, and 82 seconds each from a changed key to the next one and from a deleted Secret to the undefined line. A restart of the libvirtd container alone starts a libvirtd without the secret. The next pass finds none and logs the defined and set the value lines again. On a pod deletion libvirtd.sh stops libvirtd, ceph-secret exits 0 on TERM and leaves the secret as it is, and the next pod's container defines it again.

A lab deployed before the container existed gets it when its libvirt pods are deleted, as with every change to this OnDelete DaemonSet. The container carries the image pin of the other two. The Nix-built image compiles QEMU's rbd block driver in (see libvirt); a Docker-built image from before #1345 has bash and virsh for the container but no rbd block module, so its QEMU cannot open an RBD volume.

A server that holds an attached RBD volume keeps the key its QEMU authenticated with until the server is live-migrated or restarted (OSSN-0108), and a rotated key stops working for a new connection at once. The DaemonSet does not and cannot migrate or restart servers for it: it does not know which servers hold an RBD volume. The rotation of ceph1's user cinder in Part 2, Step 11 of the quick start migrates the server after the rotation.

libvirtd.conf listens for TLS on the node's address and port 16514 only (listen_tcp = 0, auth_tls = "none") and gives the socket the numeric group +108 with mode 0770, the group kna's chart gives its pod (supplementalGroups: [108]); the agent runs as root here (below), which opens the socket either way. qemu.conf runs QEMU as root:root without a security driver: the host's /dev/kvm is root:103, and the image's kvm group has another ID. QEMU's migration TLS reads /etc/pki/qemu and verifies the peer. The kna image of this repository writes all three private keys, libvirt/private/serverkey.pem, qemu/server-key.pem and ch/server-key.pem, with mode 0600 (see kvm-node-agent), and QEMU as root reads its key either way. PKI_KEY_GROUP therefore stays unset: it gives QEMU's key a group and mode 0640, which only a QEMU that runs as another user needs.

qemu.conf also sets stdio_handler = "file". libvirt's default, logd, sends QEMU's output, the console log Nova reads included, through a pipe to virtlogd. A virtlogd in the pod dies with the pod while QEMU lives on, and openstack console log show then stops at the restart (#1174). With file, libvirtd opens each log file and hands QEMU the descriptor, so the script starts no virtlogd. This gives up virtlogd's rollover, by default 2 MiB per file and three backups: /var/lib/nova/instances/<uuid>/console.log grows for as long as the guest writes to its console. A domain takes its handler when it starts. libvirtd reads qemu.conf when it starts too, and the script copies the file and starts libvirtd only when the container starts (steps 2 and 3). The DaemonSet updates OnDelete, so applying the change switches no node. On a lab with running servers, delete the libvirt pod of each node and wait until it is Ready. Only then run openstack server reboot --hard on each server of that node. A hard reboot before the pod delete starts the domain under the old pod's virtlogd again, and its console log stops once more at the delete. The status XML of a switched domain, /run/libvirt/qemu/<instance_name>.xml, carries no <chardevStdioLogd/>.

qemu.conf sets dynamic_ownership = 0 as well. libvirt adds its DAC driver whenever QEMU runs privileged, also under security_driver = "none" (qemuSecurityInit in src/qemu/qemu_driver.c). With dynamic ownership on, that driver hands every disk of a domain to QEMU's user, 0:0 here (virSecurityDACSetImageLabelInternal in src/security/security_dac.c). Both were read at libvirt v9.0.0; the image runs libvirt 12.2.0 from nixpkgs. Cinder's NFS backends keep each volume file 42424:42424 with mode 660 (nas_secure_file_permissions = true, see Rendered backend section), and the Cinder pods read it as uid 42424. After one attach the file would belong to root, and a backup of the volume would fail with Permission denied. With the setting off, QEMU as root opens every file it needs without a chown: the instance disks nova-compute creates as root, the console log libvirtd opens for it, /dev/kvm and the TLS keys. The owner change is read from libvirt's source, and no lab run has shown it yet. libvirtd reads qemu.conf when it starts, and the DaemonSet updates OnDelete, so on a running lab delete the libvirt pod of each node and wait until it is Ready before the first volume attach. An attach through a pod that predates the setting hands the volume file to 0:0 on the share, and no later detach or pod restart gives it back. Once every libvirt pod has been replaced, repair such a file in the NFS server pod with kubectl exec -n openstack deploy/nfs-server -c nfs-server -- chown 42424:42424 /exports/volumes/volume-<id>. A lab deployed after a teardown has the setting from the start.

The DaemonSet, the three compute CRs and hvo run in openstack. hvo sits there because its release reads the ControlPlane's auth Secret through valuesFrom, which reads only Secrets of the release's own namespace. The CA, its Issuer, the per-node Certificates libvirt-<node> with their Secrets tls-libvirt-<node>, and kna live in hypervisor-system: every CobaltCore operator reads Secrets in openstack, and a tls-libvirt-<node> Secret is root on its node's libvirtd (see Live migration).

Release	Setting	Value	Why
hvo	chart	1.2.3_sha-a2baf3f	The upstream chart of the pinned hvo commit, the rev key of images/openstack-hypervisor-operator/source.json. Its appVersion, sha-<commit>, is the image tag
hvo	fullnameOverride	hypervisor-operator	The chart names its metrics Service <fullname>-controller-manager-metrics-service, 64 characters with the release name as fullname, and the install fails
hvo	controllerManager.manager.image.repository	ghcr.io/c5c3/openstack-hypervisor-operator	The image built from that commit with its four patches: the Eviction patch, which leaves block migration to Nova, the two below that let onboarding finish, and patch 0004, which reads the catalog interface from OS_INTERFACE (see openstack-hypervisor-operator)
hvo	valuesFrom	six keys of controlplane-nova-hypervisor-operator-auth	The account the ControlPlane provisions: password into secret.servicePassword; username, user_domain_name, project_name, project_domain_name and region_name into the matching controllerManager.manager.env.os* values
hvo	env.osAuthUrl	http://controlplane-keystone.openstack.svc:5000/v3	The in-cluster Keystone URL, the spec.keystoneEndpoint of controlplane-nova. The auth Secret's auth_url is the public loopback URL
hvo	env.certificateNamespace	hypervisor-system	The Issuer's namespace
hvo	env.agentNamespaces	openstack	The namespace of the pool, chassis and metadata agent pods, which an offboarding waits for
hvo	serviceMonitor.enabled, prometheusRules.create	true	On every lab, with or without Prometheus: chart 1.2.3_sha-a2baf3f renders the ServiceMonitor hypervisor-operator and the PrometheusRules openstack-hypervisor-operator-operator-alerts (7 alerts) and openstack-hypervisor-operator-eviction-alerts (3 alerts) in openstack. Without a Prometheus Operator they are inert objects, and the Lab Prometheus stack picks them up without a second apply. The API server accepts the two rules only with the version-label patch below. Two alerts have data, HypervisorOperatorReconcileErrors and HypervisorOperatorDown. The other 8 read kube_customresource_* series, which the lab never has: HypervisorOnboardingStuck, HypervisorEvictionStuck, HypervisorEvictedTooLong, HypervisorTraitSyncFailed, HypervisorAggregateSyncFailed, EvictionFailed, EvictionMigrationFailing and EvictionOutstandingRamHigh
hvo	customResourceMetrics.create	false	Its ConfigMaps configure a kube-state-metrics to export the kube_customresource_* series of the Hypervisor and Eviction CRs, and the lab runs no kube-state-metrics
hvo	dashboards.create	false	The chart's dashboard is a Perses dashboard, which Grafana cannot load. The Lab Prometheus stack loads a Grafana dashboard with its four controller-runtime panels instead
hvo	post-renderer	OS_INTERFACE=internal on the manager	hvo takes the endpoints of compute, placement, image and network from the catalog interface OS_INTERFACE names (patch 0004), and the chart has no value for the variable. The internal endpoints are the in-cluster Service URLs over plain HTTP, so the pod needs neither a host alias nor the gateway certificates. Without the variable hvo reads the public endpoints, the *.127-0-0-1.nip.io:8443 URLs, which resolve to the pod itself
hvo	post-renderer	imagePullPolicy: Always on the manager	The chart sets no pull policy, so a node would keep the image it cached under sha-<commit>. build-images.yaml moves that tag to every main build, so its content changes whenever a patch does. An image built before patch 0002 refuses the flag below, and one built before patch 0004 ignores OS_INTERFACE
hvo	post-renderer	--default-high-availability=false appended to the manager's args	The flag of the image's patch 0002: hvo creates each Hypervisor with spec.highAvailability: false. While the field is true, onboarding waits for HaEnabled=True, which only SAP's kvm-ha-service sets. The chart has no value for the flag, so a JSON patch appends it to the argument list the chart renders, and controllerManager.manager.args stays unset
hvo	post-renderer	bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token on the ServiceMonitor's one endpoint	hvo defaults --metrics-secure to true and serves its metrics behind controller-runtime's authentication and authorization filter, and the chart's ServiceMonitor sends no token, so every scrape would answer 401. With the JSON patch Prometheus presents its own ServiceAccount token, and the chart's ClusterRole kube-prometheus-stack-prometheus grants get on /metrics, which the filter asks the API server about. The ServiceMonitor CRD marks the field deprecated in favour of authorization, which would need a token Secret in openstack. The endpoint keeps the chart's insecureSkipVerify
hvo	post-renderer	app.kubernetes.io/version: 1.2.3_sha-a2baf3f on both PrometheusRules	The chart's monitoringLabels helper writes .Chart.Version into this label unsanitized, where its labels helper writes .Chart.AppVersion. Upstream publishes the chart of every main commit as <last tag>+sha-<commit>, and helm-controller sets the build metadata of a chart pinned by digest to the digest's first 12 characters, so the label reads 1.2.3+a2442e53f92d. A label value may not contain +. The API server rejects both rules, and the install ends Stalled=True with RetriesExceeded after the Deployment and the RBAC exist. The hypervisors onboard, and the release wait of Part 2, Step 3 runs out (#1302). The base stack installs the PrometheusRule CRD, so this happens with or without the Lab Prometheus stack. The JSON patch writes the chart's ref.tag, the version with its + replaced by _ as the chart's own chart helper does, and tests/unit/deploy/metal_stack_hypervisor_test.sh fails while the two differ. The patch goes once the pinned chart carries the fix of cobaltcore-dev/openstack-hypervisor-operator#361
kna	chart	0.2.0_sha-1e4e4b8	The upstream chart of the pinned kna commit, the rev key of images/kvm-node-agent/source.json. Its appVersion, sha-<commit>, is the image tag. Upstream publishes no image under that tag, and the tag 0.2.0 would leave the private keys at 0644
kna	controllerManager.manager.image.repository	ghcr.io/c5c3/kvm-node-agent	The image built from that commit with the key mode patch, which writes the private keys with mode 0600 (see kvm-node-agent)
kna	controllerManager.manager.env.libvirtDefaultUri	qemu:///system	The chart's default ch:///system is Cloud Hypervisor
kna	controllerManager.manager.env.nodeLabelFieldPath	spec.nodeName	The chart renders it into the field reference of NODE_LABEL and defines no value
kna	controllerManager.manager.containerSecurityContext	every capability dropped but DAC_OVERRIDE, no runAsUser or runAsGroup	The image runs as 0:0: upstream's uid 42438 has no passwd entry on the host, and the host's dbus-daemon closes the connection of a uid it cannot resolve. Starting a unit over the system bus needs root too. The chart's init container hands the PKI directories to 42438, and root needs DAC_OVERRIDE to write there
kna	post-renderer	NAMESPACE=hypervisor-system	kna falls back to monsoon3 without it, and the chart sets none
kna	post-renderer	imagePullPolicy: Always on the manager	The chart has no value for the pull policy, so a node would keep the image it cached under sha-<commit>. build-images.yaml moves that tag to every main build

A lab whose hypervisors were applied with the PrometheusRules on but without the version-label patch carries the release Stalled=True with RetriesExceeded and no PrometheusRule in openstack. Applying deploy/lab/metal-stack/hypervisor again is enough: the release's new generation clears helm-controller's failure count, and helm-controller upgrades the failed release, which creates both rules.

Hypervisor.spec.createCertManagerCertificate stays at its default false, so each node's certificate is hvo's alone.

No manifest can patch a Node, so each node gets four labels and one annotation by hand:

Label	Why
openstack.c5c3.io/chassis=true	OVNChassis/lab-chassis selects it
openstack.c5c3.io/nova-compute-pool=lab	NovaCompute/lab and the libvirt DaemonSet select it
nova.openstack.cloud.sap/virt-driver=kvm	hvo creates a Hypervisor and a Certificate for such a node, and kna runs there
cobaltcore.cloud.sap/node-hypervisor-lifecycle=skip-tests	The only way a Hypervisor gets lifecycleEnabled and skipTests. The smoke test boots onto a 64 GiB volume of the type premium on the network hvo-smoke-test, which the lab fixtures leave out

Annotation	Why
nova.openstack.cloud.sap/aggregates= (empty)	hvo copies it into Hypervisor.spec.aggregates and appends the zone label, so each host stays in its zone's aggregate after onboarding. Without it the host leaves the zone's aggregate and tenant_filter_tests at Handover, and the zone has no host in Nova (#1314)

On a lab that onboarded its nodes without the annotation, the annotate command of Part 2, Step 1 is enough: hvo's list becomes the zone, which differs from status.aggregates, so hvo adds each host at once (The aggregates).

Onboarding waits at its Handover phase for two more conditions, and the hvo image of this repository patches the controller behind each. Upstream hvo sets TraitsUpdated only when a custom trait differs; patch 0003 sets TraitsUpdated=True when none differs, so a node without the nova.openstack.cloud.sap/custom-traits annotation onboards without a custom trait in Placement. Onboarding also waits for HaEnabled=True while spec.highAvailability is true, and only SAP's kvm-ha-service sets that condition. Upstream hvo creates every Hypervisor with the field true; patch 0002's flag, which the release sets to false, makes it create the field false. hvo reads the flag on create only, so a Hypervisor that exists keeps its value. EXTERNAL_CLUSTER=true make teardown-infra removes both annotations: nova.openstack.cloud.sap/aggregates, and nova.openstack.cloud.sap/custom-traits, which a lab deployed from an older version of the quick start carries.

Part 2 of the Quick Start (metal-stack) runs the sequence, from the node network to an Eviction with a Cinder volume attached and the volume's backup.

Checks outside the quick start

These checks need a lab on which Part 2 of the quick start has booted a server on every node. They confirm that the images the manifests name are published, read the value the manifests copy from the ControlPlane, show that a server survives a restart of its libvirt pod and that its console log keeps growing, and show that a live migration dials libvirt over TLS:

# a server runs on every node (Part 2, Step 5 of docs/quick-start-metal-stack.md),
# lab-0 on the first node, ${nodes[@]:0:1}
nodes=($(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'))

# the images; the libvirt keeper tag still names the digest the DaemonSet pins
libvirt_image="$(yq '.spec.template.spec.containers[0].image' deploy/lab/metal-stack/hypervisor/libvirt-daemonset.yaml)"
[ "$(docker buildx imagetools inspect "${libvirt_image%@*}" --format '{{json .Manifest.Digest}}' | tr -d '"')" = "${libvirt_image#*@}" ]
docker manifest inspect "ghcr.io/c5c3/openstack-hypervisor-operator:sha-$(hack/ci-resolve-hvo-commit.sh)" >/dev/null
docker manifest inspect "ghcr.io/c5c3/kvm-node-agent:sha-$(hack/ci-resolve-kna-commit.sh)" >/dev/null

# the Keystone URL hvo-release.yaml carries
kubectl get nova controlplane-nova -n openstack -o jsonpath='{.spec.keystoneEndpoint}{"\n"}'

# a libvirt restart under a running server, whose console log keeps growing
kubectl delete pod -n openstack -l app.kubernetes.io/name=libvirt \
  --field-selector "spec.nodeName=${nodes[@]:0:1}"
kubectl wait pod -l app.kubernetes.io/name=libvirt -n openstack --for=condition=Ready --timeout=10m
openstack --insecure server show lab-0 -c status -f value
libvirt_pod=$(kubectl get pod -n openstack -l app.kubernetes.io/name=libvirt \
  --field-selector "spec.nodeName=${nodes[@]:0:1}" -o name)
console_log="/var/lib/nova/instances/$(openstack --insecure server show lab-0 -c id -f value)/console.log"
kubectl exec -n openstack "${libvirt_pod}" -c libvirtd -- stat -c %s "${console_log}"
kubectl exec -n openstack "${libvirt_pod}" -c libvirtd -- \
  virsh reset "$(openstack --insecure server show lab-0 -c OS-EXT-SRV-ATTR:instance_name -f value)"
sleep 30
kubectl exec -n openstack "${libvirt_pod}" -c libvirtd -- stat -c %s "${console_log}"
openstack --insecure console log show lab-0 | tail -n 3

# the source libvirtd logs its migration steps
kubectl exec -n openstack "${libvirt_pod}" -c libvirtd -- \
  bash -c 'virt-admin daemon-log-filters 1:qemu.qemu_migration && virt-admin daemon-log-outputs 1:stderr'

The restart check prints the size of lab-0's console log before and after a reset of the guest, and the second number is larger.

Then run the live migration of Part 2, Step 8 and read the URIs the source libvirtd dialed, in the same shell:

kubectl logs -n openstack "${libvirt_pod}" -c libvirtd | grep -o 'qemu+tls://[^ ,]*' | sort -u

A lab that moves to the kna image of this repository in place keeps its 0644 keys: kna writes the files only when the node's Secret changes. Deleting tls-libvirt-<node> makes cert-manager reissue the certificate with a new key, which kna then writes with mode 0600. That also retires a key every local user of the node could read. Delete the Secrets only once every node runs the new image. kna records the resourceVersion it wrote in the host's /etc/pki/CA/.last_resource_version and reads it back at start, so a pod of the old image that handles the reissued Secret writes the new key with mode 0644, and the new pod then skips the Secret as unchanged:

image="ghcr.io/c5c3/kvm-node-agent:sha-$(hack/ci-resolve-kna-commit.sh)"
# the chart names the DaemonSet <release>-controller-manager
kubectl wait daemonset/kvm-node-agent-controller-manager -n hypervisor-system \
  --for=jsonpath="{.spec.template.spec.containers[?(@.name==\"manager\")].image}=${image}" \
  --timeout=10m
kubectl rollout status daemonset/kvm-node-agent-controller-manager \
  -n hypervisor-system --timeout=15m
for node in "${nodes[@]}"; do
  kubectl delete secret "tls-libvirt-${node}" -n hypervisor-system
done
kubectl wait certificate --all -n hypervisor-system --for=condition=Ready --timeout=10m
for pod in $(kubectl get pod -n openstack -l app.kubernetes.io/name=libvirt -o name); do
  kubectl exec -n openstack "${pod}" -c libvirtd -- \
    stat -c '%a %n' /etc/pki/libvirt/private/serverkey.pem /etc/pki/qemu/server-key.pem
done

Each key shows 600. A key that still shows 644 has not been rewritten by a pod of the new image; deleting its node's Secret again has that pod write it. These commands have not run on the lab. A lab deployed after a teardown gets new Secrets, so kna writes new keys anyway.

The block below, run on the lab on 2026-10-02, changes cpuMode under a running server, once with the rollout awaited and once with a pod that predates the change, and prints the pod's CPU keys and the server's <cpu> element at each point:

# after Part 2, Steps 1 to 5 of docs/quick-start-metal-stack.md, without
# the servers of Step 5; nodes and zone come from the opening block of Part 2

compute_selector=app.kubernetes.io/instance=lab,app.kubernetes.io/component=nova-compute
# prints the name of the nova-compute pod on the first node
compute_pod() {
  kubectl get pod -n openstack -l "${compute_selector}" \
    --field-selector "spec.nodeName=${nodes[@]:0:1}" -o name
}
# waits until the operator has rendered the current spec; kubectl wait skips a
# condition whose observedGeneration is older than metadata.generation
wait_seen() {
  kubectl wait novacompute/lab -n openstack --for=condition=ConfigReady --timeout=5m
}
# waits until the nova-compute pod on the first node mounts the ConfigMap of
# the current spec and its process has reported to Nova; until then a hard
# reboot keeps the old CPU or is lost. Updated At carries nova-conductor's clock
# and startedAt the node's, so it waits for Updated At to change twice while one
# container runs
wait_compute() {
  local i gen cfg pod have start updated seen base changes
  for i in $(seq 60); do
    gen=$(kubectl get novacompute lab -n openstack -o jsonpath='{.metadata.generation}')
    cfg=$(kubectl get novacompute lab -n openstack \
      -o jsonpath='{range .status.conditions[?(@.type=="ConfigReady")]}{.status} {.observedGeneration} {.message}{end}')
    pod=$(compute_pod)
    have=$(kubectl get -n openstack "${pod}" \
      -o jsonpath='{.spec.volumes[?(@.name=="pool-config")].configMap.name}')
    start=$(kubectl get -n openstack "${pod}" \
      -o jsonpath='{.status.containerStatuses[?(@.name=="nova-compute")].state.running.startedAt}')
    updated=$(openstack --insecure compute service list --service nova-compute \
      --host "${nodes[@]:0:1}" -c 'Updated At' -f value)
    if [[ "${cfg}" != "True ${gen} "* || "${cfg##* }" != "${have}" || -z "${start}" \
      || -z "${updated}" ]]; then
      seen=
    elif [[ "${seen}" != "${pod}@${start}" ]]; then
      seen="${pod}@${start}" base="${updated}" changes=0
    elif [[ "${updated}" != "${base}" ]]; then
      # the first change can be a report the old process sent before it died,
      # applied late by nova-conductor; the second is the new process's
      base="${updated}"
      (( ++changes >= 2 )) && return 0
    fi
    sleep 5
  done
  echo "wait_compute: ConfigReady '${cfg}', mounted '${have}', started '${start}'," \
    "Updated At '${updated}', changes ${changes:-0}" >&2
  return 1
}
# waits until every pod runs the current template (RollingUpdate only)
wait_pool() {
  wait_seen &&
    kubectl rollout status daemonset/lab-nova-compute -n openstack --timeout=15m &&
    kubectl wait novacompute/lab -n openstack --for=condition=DaemonSetReady --timeout=15m
}
# prints the start time and the cpu keys of the nova-compute pod on the first
# node
pool_state() {
  local pod
  pod=$(compute_pod)
  kubectl get -n openstack "${pod}" -o jsonpath='{.status.startTime}{"\n"}'
  kubectl exec -n openstack "${pod}" -c nova-compute -- \
    grep -E '^cpu_(mode|models)' /etc/nova/compute-pool.conf.d/compute-pool.conf
}
# prints the <cpu> element of the live and of the persistent definition of cpu-a
domain_cpu() {
  local pod name
  pod=$(kubectl get pod -n openstack -l app.kubernetes.io/name=libvirt \
    --field-selector "spec.nodeName=${nodes[@]:0:1}" -o name)
  name=$(openstack --insecure server show cpu-a -c OS-EXT-SRV-ATTR:instance_name -f value)
  kubectl exec -n openstack "${pod}" -c libvirtd -- sh -c \
    "virsh dumpxml ${name} | sed -n '/<cpu /,/<\/cpu>/p'; echo --inactive; virsh dumpxml --inactive ${name} | sed -n '/<cpu /,/<\/cpu>/p'"
}

# A. baseline: the pool on host-passthrough, one server
kubectl patch novacompute lab -n openstack --type merge \
  -p '{"spec":{"libvirt":{"cpuMode":"host-passthrough","cpuModels":null}}}'
wait_pool; pool_state
wait_compute && openstack --insecure server create cpu-a --image cirros-kvm --flavor 1 \
  --network lab-net --availability-zone "${zone}:${nodes[@]:0:1}" --wait
domain_cpu

# B. the cpuMode change with the rollout awaited
date -u +%FT%TZ
kubectl patch novacompute lab -n openstack --type merge \
  -p '{"spec":{"libvirt":{"cpuMode":"custom","cpuModels":["Skylake-Server-IBRS"]}}}'
wait_pool; pool_state
domain_cpu                                  # before the reboot
wait_compute && openstack --insecure server reboot --hard --wait cpu-a
domain_cpu                                  # after the reboot
pool_state                                  # unchanged

# C. a hard reboot handled by a pod that predates the change
kubectl patch novacompute lab -n openstack --type merge \
  -p '{"spec":{"updateStrategy":{"type":"OnDelete"},"libvirt":{"cpuMode":"host-passthrough","cpuModels":null}}}'
wait_seen; pool_state                       # the pod did not restart
kubectl get novacompute lab -n openstack \
  -o jsonpath='{.status.conditions[?(@.type=="DaemonSetReady")].status}{"\n"}'
openstack --insecure server reboot --hard --wait cpu-a
domain_cpu
kubectl delete pod -n openstack -l "${compute_selector}"
kubectl wait novacompute/lab -n openstack --for=condition=DaemonSetReady --timeout=15m
if wait_compute; then
  pool_state                                # a new pod, the new file
  openstack --insecure server reboot --hard --wait cpu-a
fi
domain_cpu

# D. back to the manifests, and away
openstack --insecure server delete --wait cpu-a
openstack --insecure server list --all-projects
kubectl apply -k deploy/lab/metal-stack/hypervisor
kubectl patch novacompute lab -n openstack --type merge \
  -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
wait_pool
EXTERNAL_CLUSTER=true make teardown-infra

The lab ran the block with looser waits. wait_seen waited for status.observedGeneration, which a pass that stops before the ConfigMap step sets too. wait_pool waited for Ready, which also needs every compute service of the pool up, and called wait_compute itself. wait_compute compared Updated At, from nova-conductor's clock, with the pod's start time, from the node's, and it did not check the mounted ConfigMap. Every command ran whether the wait before it held or not. The waits as written here have not run on the lab.

Property	Value
Namespaces	openstack (libvirt, the compute CRs, hvo), hypervisor-system (the CA, the node certificates, kna, the migration port reservation), flux-system (the chart sources)
Applied	by hand: migration-ports/, then the node labels, then hypervisor-fixtures/, then hypervisor/, after the Lab ControlPlane is Ready
Removed by	EXTERNAL_CLUSTER=true make teardown-infra, step 0, the servers and volumes in OpenStack, the labels and the maint-<node> objects included. The node state under /var/lib/nova, /var/lib/libvirt and /etc/pki stays, and so do the reserved ports in net.ipv4.ip_local_reserved_ports until a node reboots
Pinned by	tests/unit/deploy/metal_stack_hypervisor_test.sh; the hvo chart tag follows hack/ci-resolve-hvo-commit.sh, and the kna chart tag hack/ci-resolve-kna-commit.sh
Dependencies	the Lab ControlPlane with spec.services.nova.hypervisorOperator; /dev/kvm and vhost_net on every node; TCP 16514 and 49152 to 49215 open between the nodes (Node port check)

The run of 2026-10-01 on the lab ran upstream's kna v0.2.0 image and chart, with uid 0 set in the release, on two Xeon D-2141I workers on Debian 12 with kernel 6.1. It passed every step of the sequence that is now Part 2 of the Quick Start (metal-stack) and of the checks above, but the kna image check, which came with the image of #1178, and the console-log part of the restart check, which came with #1174. Both ran on 2026-10-02 (below). The run also created /var/lib/nova/instances by hand. The NovaCompute pod creates it since #1171. libvirtd kept its domains across a restart of its pod: the scope held libvirtd alone, and QEMU ran in the host cgroup /machine/qemu-<n>-<instance>.libvirt-qemu that libvirt created itself. The socket's group was 108, libvirtd listened on <node IP>:16514 only, and the live migration dialed qemu+tls://<destination IP>/system. A teardown with a server left stopped at the NovaCompute delete with the servers hint; after the servers were deleted, two teardowns exited 0 and left no kvm.cloud.sap CRD, hypervisor-system namespace, maint-<node> object, node label or host unit.

A run on 2026-10-02 applied stdio_handler = "file" (#1174) on the same two workers. Once lab-a was hard-rebooted, no virtlogd ran on its node and its status XML carried no <chardevStdioLogd/>. After a restart of the libvirt pod, a virsh reset of the guest left the QEMU PID unchanged, console.log grew, and openstack console log show returned the new boot output.

A second run on 2026-10-02 executed the Quick Start (metal-stack) from commit 715eafd3, from a bare cluster through the teardown, on the same two workers. Every block of the page exited 0, two of them after repeats: Part 1, Step 4, which named a repeat then, and Part 2, Step 3. The kna pods ran ghcr.io/c5c3/kvm-node-agent:sha-1e4e4b8e7050bbfa2d5a990aa21b849ba7b51700 with no runAsUser in their pod spec and restarted once each; the run did not read why. Both Hypervisor objects showed TLSCertificateInstalled True, and the key-mode loop above printed 600 for serverkey.pem and server-key.pem on both nodes. The NovaCompute pods carried the init container create-instances-dir, and both servers booted with no directory created by hand; the nodes still held /var/lib/nova from the earlier runs, so the run does not show the directory created on a fresh node. The two lab-metadata-agent pods had a restart count of 0 after both servers booted and after the eviction (#1173). The teardown exited 0 and left neither openstack nor hypervisor-system; it waited five minutes for the Keystone CR's backup PushSecrets (#1186).

hvo and kna come from SAP's own environment, and several of their defaults assume it. #1066 collects what would have to change upstream. Each item with what it did on the lab:

Item	Upstream	On the lab	Here
maint-<node> image	hvo's Gardener lifecycle controller creates Deployment maint-<node> in kube-system with keppel.global.cloud.sap/ccloud-dockerhub-mirror/library/busybox:latest	both Deployments stay 0/1; their pods are Pending in ImagePullBackOff	none; the teardown deletes the objects
kube-system interlock	the same controller keeps a PodDisruptionBudget maint-<node> with minAvailable: 1 on that Deployment, so a node drain waits until the hypervisor is offboarded	both PDBs allow 0 disruptions, on pods that never start	none
Operating system	kna fills Hypervisor.status.operatingSystem from the host's os-release and drives updates through systemd-sysupdate, both written for Garden Linux, the Gardener project's Debian-based host OS	the status carries the kernel, the firmware and the hardware, and no version, prettyVersion, variantID or Garden Linux field; the VERSION column is empty, and no update condition appears	none
BlockMigration: false	hvo's Eviction live-migrates without block migration, which Nova refuses for a server on local disks	the patched image's Eviction moved both servers off a node with local qcow2 disks, two completed live migrations	patched in the image (#1163)
HaEnabled gate	onboarding waits for HaEnabled=True while spec.highAvailability is true; only kvm-ha-service sets it	with highAvailability: false patched by hand, both Hypervisors reached Onboarding=False, reason Succeeded. The flag has not run on the lab yet	patched in the image: --default-high-availability=false (patch 0002), set by the release, creates each Hypervisor with spec.highAvailability: false
TraitsUpdated gate	onboarding waits for TraitsUpdated=True, which hvo sets only when a custom trait differs; it assigns the trait in Placement and does not create it	with the annotation, Placement answered 400 No such trait CUSTOM_C5C3_LAB until the trait existed. Patch 0003 has not run on the lab yet	patched in the image: TraitsUpdated=True when no custom trait differs (patch 0003), so the nodes carry no annotation
Host units	kna reads libvirtd.service and openvswitch-switch.service and starts virt-admin-server-update-tls.service through the host's systemd	with the stand-ins kna reports libvirtd.service, LibVirtConnection and TLSCertificateInstalled as True; openvswitch-switch.service stays False, because Open vSwitch runs in the chassis pod	runtime stand-ins written by the libvirt DaemonSet
Agent uid	kna's image runs as uid 42438 and authenticates to the system bus with it	the host has no such user, and its dbus-daemon drops the connection; kna exits at start. Run as uid 0 from the release, upstream's image started. The image of #1178 ran on 2026-10-02 with no runAsUser in the pod spec	the image of #1178 runs as uid 0; the release keeps DAC_OVERRIDE
Key mode	kna writes every TLS file with mode 0644, the private keys included (#1175, found in upstream's code)	not checked with upstream's image. With the image of #1178, both keys showed 600 on both nodes on 2026-10-02	patched in the image (#1178): 0600, or 0640 for QEMU's and Cloud Hypervisor's keys with PKI_KEY_GROUP
monsoon3 fallback	kna's namespace without NAMESPACE, which its chart does not set	with NAMESPACE kna installs the certificates of hypervisor-system	NAMESPACE set by a post-renderer
Catalog interface	hvo reads only the public endpoints and takes no CA	through host aliases onto an Envoy Service and SSL_CERT_DIR, hvo ran 35 minutes without a restart and without an x509 or connection refused line. OS_INTERFACE=internal has not run on the lab yet	patched in the image: OS_INTERFACE (patch 0004), set to internal by the release
Chart object names	the chart builds names from the fullname, the release name when it contains the chart name	openstack-hypervisor-operator makes the metrics Service name 64 characters, and the install fails	fullnameOverride: hypervisor-operator
Hand-set node labels	hvo flags changes made with kubectl	both Hypervisors reported Tainted=True, reason Kubectl. The taint controller reads kubectl from the managers of the Hypervisor's own managed fields (hypervisor_taint_controller.go), which the kubectl patch of spec.highAvailability wrote; the labels sit on the Nodes. No run without that patch has read the condition yet	none
Images per main commit	hvo and kna publish a chart for every main commit but no image under its tag; hvo pushes only latest	the chart of a2baf3f ran ghcr.io/c5c3/openstack-hypervisor-operator:sha-a2baf3f…; the kna chart of 1e4e4b8 ran ghcr.io/c5c3/kvm-node-agent:sha-1e4e4b8… on 2026-10-02	the images of #1163 and #1178

The run also found what this repository's own pieces owe a real hypervisor. The fake driver of the kind suites reaches none of it:

Item	On the lab	Here
Live-migration CPU check	with cpuMode: host-passthrough, and with host-model, every live migration ends in NoValidHost: Nova's pre-check on the destination fails with Unacceptable CPU info: CPU doesn't have compatibility, although virsh hypervisor-cpu-compare there accepts the guest CPU	cpuMode: custom with Skylake-Server-IBRS, the host-model of both workers
Console log after a libvirt restart	with libvirt's default stdio_handler = "logd", openstack console log show stopped at the restart: QEMU's log went through virtlogd, which ran in the old pod. The guest and its network kept running. With file the log grows across a restart	stdio_handler = "file" in qemu.conf (#1174)
CPU model change under a server	on 2026-10-01 a server booted with host-passthrough kept that CPU through a hard reboot after the pool moved to custom; the run recorded no time for the reboot. The cpuMode block above ran on 2026-10-02 on the first node. A, after the boot: the pod's file says cpu_mode = host-passthrough, the domain mode='host-passthrough'. B, before the reboot: a pod started 10 seconds after the patch, its file says cpu_mode = custom and cpu_models = Skylake-Server-IBRS, the domain is still host-passthrough. B, after the reboot: the same pod, and the live and the --inactive <cpu> say mode='custom' with the model Skylake-Server-IBRS. C, first reboot: the pod of B with its custom file, DaemonSetReady False, the domain mode='custom' while the CR says host-passthrough. C, second reboot: a new pod, its file says cpu_mode = host-passthrough and no cpu_models, the domain mode='host-passthrough'. A first pass of the same run sent C's second reboot 3 seconds after the new pods started, before nova-compute took requests: it cleared the reboot's task state at start-up, the reboot action ended in Error, --wait reported success and the domain stayed custom. wait_compute comes from that pass	Changing the libvirt settings of a pool with servers
NFS server restart under an attached volume	on 2026-10-04 a PodChaos pod-kill of the NFS server (F4) and its scale-down for 300 seconds (F5) restarted nfsd, which logged Unable to initialize client recovery tracking! (-22). The compute node logged lost 1 locks, and the guest got I/O error, dev vdb on every request to the attached volume until a hard reboot, while Nova, Cinder and the CRs reported nothing (see Lab fault runs)	#1245: the server is NFS-Ganesha, with its client records on the export claim and a fixed server scope, and cinder-nfs-outage checks that a lock outlives a restart; F4 and F5 have not run on it yet

libvirt 12 on the lab

The lab's libvirt image moves from the Docker build, libvirt 10.0.0 and QEMU 8.2.2 of Ubuntu noble, to the Nix build of #1407, libvirt 12.2.0 and QEMU 10.2.4 at the pinned nixpkgs-images. nova-compute moves with it, so its libvirt-python talks to both daemons while the lab rolls node by node. The roll runs on Renovate's pull request that moves the eight pins from the Docker-built keeper to the first Nix keeper, once main has minted it:

1. Set spec.image of NovaCompute/lab to ghcr.io/c5c3/nova-compute with the composite tag of the image under test, and wait for its DaemonSet to roll. The new nova-compute then talks to the old libvirt.
2. Boot a server on each node (Part 2, Step 5 of the Quick Start (metal-stack)) and live-migrate one (Step 8).
3. Apply the overlay with the pull request's pins, kubectl apply -k deploy/lab/metal-stack/hypervisor. The DaemonSet is OnDelete, so nothing restarts.
4. On one node, live-migrate its servers away and delete its libvirt pod. Once the pod is Ready, virsh version prints libvirt 12.2.0 and QEMU 10.2.4.
5. On each remaining node, delete the libvirt pod without moving its servers, since QEMU keeps running them while the pod restarts. Once the pod is Ready, run openstack server reboot --hard --wait on each server of the node, and compare virsh dumpxml <uuid> | grep machine= before and after: pc-i440fx-noble before, pc-i440fx-10.2 after. Then delete the pods of nfs-client-modules and chaos-mesh-modules, so they run the new pin.
6. Run Part 2, Steps 5 to 11, and Part 3, Steps 1 to 4.
7. Once main publishes the image, remove spec.image, so the pool follows nova-compute:<release> again.

A server started on QEMU 8.2.2 cannot be live-migrated to a rolled node. Nova asks for the machine type pc, which Ubuntu's QEMU 8.2.2 resolves to pc-i440fx-noble (and q35 to pc-q35-noble). libvirt records the resolved name in the domain XML that a live migration sends, and the libvirt of a rolled node refuses it: Emulator '/usr/bin/qemu-system-x86_64' does not support machine type 'pc-i440fx-noble'. A cold migration would write the XML anew, but the compute image has no ssh client to reach the other node with. A hard reboot writes it anew on the same node, where QEMU 10.2.4 resolves pc to pc-i440fx-10.2. Live migration works again between rolled nodes.

A migration from QEMU 10.2.4 back to 8.2.2 is not part of the roll. The prune keeps the Docker-built keeper tags as the rollback target. No lab run of the roll is recorded yet.

Node port check

File: hack/lab-node-ports.sh

Live migration between the lab's hypervisors (#1142) needs libvirt's TLS port 16514 and the QEMU migration range 49152-49215 open from every node to every other node on the node network. The script proves that on the current context's cluster. It starts one listener Pod per node (perl, one socket per port on 0.0.0.0) and one client Pod per ordered node pair on the source node (bash /dev/tcp to the destination's first InternalIP, the address status.hostIP and so the NovaCompute live_migration_inbound_addr resolve to), all host-networked and unprivileged: no capability, no privilege escalation, UID 65534 on a read-only root filesystem, no ServiceAccount token. It runs on the node probe's pinned debian image, which carries bash, perl and timeout, so it has no image or Renovate pin of its own. Every Pod is labelled app.kubernetes.io/name=lab-node-ports. A run first deletes the Pods an earlier run left behind, so their logs are never read as its results, and an EXIT trap deletes the Pods on every path.

hack/lab-node-ports.sh
NODE_PORTS_TCP=16514 hack/lab-node-ports.sh

It prints one line per ordered pair, then a summary:

worker-a (10.128.44.10) -> worker-b (10.128.44.11): 65/65 open
worker-b (10.128.44.11) -> worker-a (10.128.44.10): 63/65 open, closed: 16514 49215

A port is closed when the connect fails, listener bind failed when the destination's listener could not bind it, and no result when the client produced no line for it before the client wait ran out. The migration range sits in Linux's ephemeral port range, so an outgoing connection on the node can hold one of its ports. deploy/lab/metal-stack/migration-ports reserves the range on every node, which keeps new connections off it. The listener tries a port it cannot bind again every second for NODE_PORTS_BIND_TIMEOUT seconds before it reports it. A port a connection took before the reservation stays listener bind failed and fails the run, which then prints a NOTE: line saying the port was not tested; restarting the process that holds the port frees it.

Variable	Default	Description
NODE_PORTS_TCP	16514 49152-49215	Space-separated ports and inclusive ranges (65 ports by default)
NODE_PORTS_NAMESPACE	default	Namespace of the Pods
NODE_PORTS_IMAGE	the image: of deploy/lab/metal-stack/probe/node-probe.yaml	Image with bash, perl and timeout; the check exits 2 when it is unset and the probe manifest is missing
NODE_PORTS_NODE_SELECTOR	empty (every node)	Label selector limiting the nodes
NODE_PORTS_CONNECT_TIMEOUT	5	Seconds per connect
NODE_PORTS_BIND_TIMEOUT	10	Seconds the listener keeps retrying a port it cannot bind, once per second. The wait for the listeners' listening line adds it to NODE_PORTS_POD_TIMEOUT
NODE_PORTS_POD_TIMEOUT	120	Seconds for the listener waits and for removing an earlier run's Pods. The client wait adds NODE_PORTS_CONNECT_TIMEOUT per port, because a client probes its ports one after another and a firewall that drops the packets makes every probe take the full connect timeout

Exit code	Meaning
0	Every port of every ordered pair is open
1	At least one port is closed, has no result, or was not bound by the listener within NODE_PORTS_BIND_TIMEOUT
2	Usage or cluster error: a missing tool or image, an invalid variable, fewer than two nodes, a node without an InternalIP or whose first one is IPv6 (the listeners bind IPv4 only) or not an address, Pods of an earlier run that cannot be deleted, Pods that cannot be created, or listeners that never became Ready

Migration port reservation

Files: deploy/lab/metal-stack/migration-ports/kustomization.yaml, namespace.yaml and reservation-daemonset.yaml beside it

The kernel picks the local port of an outgoing connection from net.ipv4.ip_local_port_range, 32768 to 65535 on the lab's nodes, which holds the migration range. A host-network process can therefore get a migration port for a connection that lasts for days: on shoot forge, calico-node held 49152 for its connection to the API server, and the check answered listener bind failed: 49152 on every run. libvirt skips a migration port it cannot bind, so migrations go on; the check cannot test the port.

The DaemonSet migration-port-reservation in hypervisor-system runs on every node. Its init container reserve adds 49152-49215 to net.ipv4.ip_local_reserved_ports, which the kernel leaves out when it picks a local port; a bind to a named port, QEMU's and the check's, is not affected. It keeps a value the node already carries. The setting belongs to the host's network namespace and /proc/sys is writable only in a privileged container, so the pod uses the host's network and that container is privileged. The second container, hold, only keeps the pod running, as UID 65534 without a capability: the setting is gone after a reboot, and the init container then runs again. Both run the node probe's pinned debian image, and Renovate moves the pins in one group.

Part 2, Step 1 of the Quick Start (metal-stack) applies the directory in front of the check. The hypervisor overlay takes it as a resource, so EXTERNAL_CLUSTER=true make teardown-infra removes the DaemonSet with that overlay. The reserved ports stay on a node until it reboots.

The reservation closes no connection. One that held a port of the range before it keeps the port, and the check then still reports listener bind failed. The init container names such a socket and the process that owns it, which is what the pod's host PID namespace is for:

kubectl logs -n hypervisor-system -l app.kubernetes.io/name=migration-port-reservation \
  -c reserve --prefix --tail=-1

[pod/migration-port-reservation-5gm99/reserve] reserved: 49152-49215 (before: none)
[pod/migration-port-reservation-5gm99/reserve] in use: port 49152 by calico-node (pid 3798)
[pod/migration-port-reservation-bjjzt/reserve] reserved: 49152-49215 (before: none)
[pod/migration-port-reservation-bjjzt/reserve] in use: none

--tail=-1 keeps every line: with a label selector, kubectl logs prints only the last 10 lines of each pod, which can drop a pod's reserved: line and its first holders. kubectl get pod -n hypervisor-system -o wide maps each pod to its node. A socket no process holds prints by an unknown owner. Restarting the named process frees its port, and its next connection gets a port outside the range. For a process a DaemonSet runs, delete its pod on that node, as for calico-node on forge:

kubectl delete pod -n kube-system -l k8s-app=calico-node --field-selector spec.nodeName=<node>

The lines describe the moment the pod started. Delete the reservation's pod on the node to read them anew.