ControlPlane Reconciler Architecture

Reference documentation for the ControlPlaneReconciler, the CredentialRotationReconciler, and their sub-reconciler contracts. The ControlPlaneReconciler implements the control loop that drives a ControlPlane CR from desired state to a fully operational Keystone control plane: backing infrastructure, the projected Keystone service, the K-ORC admin application credential, and the OpenStack service-catalog entries.

For CRD type definitions and webhooks, see ControlPlane CRD API Reference. For the shared controller-manager bootstrap pattern (internal/common/bootstrap) the c5c3 operator reuses verbatim, see the Keystone Reconciler — Controller Registration section. For the library functions used by the sub-reconcilers, see Kubernetes-Interacting Packages. For the infrastructure stack (MariaDB, Memcached, K-ORC, OpenBao) the operator targets, see Infrastructure Manifests.

The c5c3 operator is intentionally a thin orchestrator: it provisions and owns child CRs (MariaDB, Memcached, Keystone, Horizon, K-ORC ApplicationCredential / Service / Endpoint) and aggregates their readiness. It does not re-implement the per-service logic those child operators already own. As a consequence the c5c3 API surface is deliberately smaller than the Keystone reconciler's: its reconcile chain is a blocking prefix plus a sequential, non-short-circuiting tail group (RunSequentialGroup) — not the keystone reconciler's goroutine-backed parallel groups (RunParallelGroup) — and it keeps no per-CR metric cardinality. It does install a single finalizer to sequence K-ORC teardown ahead of Keystone/infrastructure teardown on deletion — see Owner-ref / GC model.

The figure places the operator among what it creates and what it relies on: the ControlPlane CR, the infrastructure CRs, service CRs and K-ORC resources it projects, and the service operators that turn those CRs into workloads.

Controller Registration

The c5c3 operator registers two reconcilers and an optional webhook with the controller manager in operators/c5c3/main.go via the shared bootstrap package (github.com/c5c3/cobaltcore/internal/common/bootstrap). The bootstrap helper builds the manager, wires leader election, and invokes the operator's SetupFunc; the same pattern is documented in detail under Keystone Reconciler — Controller Registration.

import (
    "github.com/c5c3/cobaltcore/internal/common/bootstrap"
    c5c3v1alpha1 "github.com/c5c3/cobaltcore/operators/c5c3/api/v1alpha1"
    "github.com/c5c3/cobaltcore/operators/c5c3/internal/controller"
)

const leaderElectionID = "c5c3.openstack.c5c3.io"

bootstrap.Run(bootstrap.ManagerConfig{
    Scheme:           scheme,
    LeaderElectionID: leaderElectionID,
    TargetClusters:   true,
    SetupFunc: func(mcMgr mcmanager.Manager, opts bootstrap.SetupOptions) error {
        mgr := mcMgr.GetLocalManager()
        if opts.Controllers {
            if err := (&controller.ControlPlaneReconciler{
                Client:   mgr.GetClient(),
                Scheme:   mgr.GetScheme(),
                Recorder: mgr.GetEventRecorderFor("controlplane-controller"),
                Resolver: mcMgr,
            }).SetupWithManager(mcMgr); err != nil {
                return err
            }
            if err := (&controller.CredentialRotationReconciler{
                Client:   mgr.GetClient(),
                Scheme:   mgr.GetScheme(),
                Recorder: mgr.GetEventRecorderFor("credentialrotation-controller"),
            }).SetupWithManager(mgr); err != nil {
                return err
            }
        }
        if opts.Webhooks {
            return (&c5c3v1alpha1.ControlPlaneWebhook{Client: mgr.GetClient()}).
                SetupWebhookWithManager(mgr)
        }
        return nil
    },
})

Element	Value
LeaderElectionID	c5c3.openstack.c5c3.io (a package-level constant in main.go; referenced by the deploy-stack RBAC and asserted by main_test.go so a rename cannot silently break leader election)
Primary reconciler	ControlPlaneReconciler (event recorder controlplane-controller), completed through the multicluster builder: it takes mcMgr, and its Resolver turns a service's targetClusterRef into the client that service's children are written with
Secondary reconciler	CredentialRotationReconciler (event recorder credentialrotation-controller), on the local manager: it only ever touches the management cluster
TargetClusters	true: the binary engages the clusters registered in --clusters-namespace. The provider engages nothing while that namespace holds no registration Secret, which is the single-cluster default every existing install keeps
Webhook	ControlPlaneWebhook, registered only when bootstrap.Run passes opts.Webhooks == true to SetupFunc (the bootstrap layer resolves bootstrap.SetupOptions once from the flags). The reconcilers are registered only when opts.Controllers is true, which --enable-controllers=false clears for a standalone webhook deployment

Scheme Registration

The operator registers these schemes in main.go's init() so the reconcilers can interact with the typed child CRDs:

Module	Scheme	Types Used
k8s.io/client-go/kubernetes/scheme	clientgoscheme.AddToScheme	core Kubernetes types (Secret, Event)
github.com/c5c3/cobaltcore/operators/c5c3/api/v1alpha1	c5c3v1alpha1.AddToScheme	ControlPlane, CredentialRotation, SecretAggregate (own API)
github.com/c5c3/cobaltcore/operators/keystone/api/v1alpha1	keystonev1alpha1.AddToScheme	Keystone (projected and owned child)
github.com/c5c3/cobaltcore/operators/placement/api/v1alpha1	placementv1alpha1.AddToScheme	Placement (projected and owned child)
github.com/mariadb-operator/mariadb-operator	mariadbv1alpha1.AddToScheme	MariaDB (projected and owned child)
github.com/external-secrets/external-secrets	esov1alpha1.SchemeBuilder	PushSecret (admin-credential mirror)
github.com/external-secrets/external-secrets	esov1.SchemeBuilder	ExternalSecret, ClusterSecretStore, SecretStore (K-ORC clouds.yaml gate + per-ControlPlane store selection)
github.com/k-orc/openstack-resource-controller/v3	orcv1alpha1.AddToScheme	ApplicationCredential, Service, Endpoint

Note (Memcached is unstructured): memcached.c5c3.io ships no Go module, so the Memcached child is deliberately not registered in the scheme. reconcileInfrastructure builds and applies it as an *unstructured.Unstructured carrying the shared memcachedGVK (memcached.c5c3.io/v1beta1, kind Memcached), and SetupWithManagerOwns the same unstructured GVK. The GVK is resolved against the cluster RESTMapper at runtime, so no scheme entry is required.

Watches

The controller watches the primary ControlPlane CR, every child CR the sub-reconcilers project (including the owned ESO ExternalSecret and PushSecret), the admin-password Secret, and both OpenBao-backed store kinds (ClusterSecretStore and SecretStore):

Resource	Watch Type	Effect
ControlPlane	For()	Triggers reconciliation on CR changes
MariaDB	Owns()	Re-reconciles the owning ControlPlane when the managed MariaDB child status changes
Memcached (unstructured memcachedGVK)	Owns()	Re-reconciles when the managed Memcached child status changes; owned as *unstructured.Unstructured because the kind has no Go module
Keystone	Owns()	Re-reconciles when the projected Keystone child status changes
Horizon	Owns()	Re-reconciles when the projected Horizon dashboard child status changes
Glance	Owns()	Re-reconciles when the projected Glance image-service child status changes
GlanceBackend	Owns()	Re-reconciles when a projected GlanceBackend child status changes
Placement	Owns()	Re-reconciles when the projected Placement child status changes
Barbican	Owns()	Re-reconciles when the projected Barbican key-manager child status changes
BarbicanSecretStore	Owns()	Re-reconciles when the projected BarbicanSecretStore status changes
Neutron	Owns() + cross-namespace Watches()	Re-reconciles when the projected Neutron network-service child status changes. The neutron-operator is installed only for a ControlPlane that runs the network service, so both legs sit behind the discovery probe with the other sibling-operator kinds
Cinder	Owns() + cross-namespace Watches()	Re-reconciles when the projected Cinder block-storage child status changes. The cinder-operator is installed only for a ControlPlane that runs the block-storage service, so both legs sit behind the discovery probe with the other sibling-operator kinds
CinderBackend	Owns() + cross-namespace Watches()	Re-reconciles when a projected CinderBackend satellite status changes, under the same discovery guard as the Cinder kind
CinderBackupBackend	Owns() + cross-namespace Watches()	Re-reconciles when the projected CinderBackupBackend satellite status changes, under the same discovery guard as the Cinder kind
Nova	Owns() + cross-namespace Watches()	Re-reconciles when the projected Nova compute-service child status changes. The nova-operator is installed only for a ControlPlane that runs the compute service, so both legs sit behind the discovery probe with the other sibling-operator kinds
OpenBaoCluster, OpenBaoTenant	Owns()	Re-reconciles when the OpenBao instance provisioned for a dedicated Barbican secret store, or the tenant admitting its namespace, changes. The openbao-operator is installed only for that mode, so a ControlPlane without one runs on a cluster that never serves these kinds; both legs sit behind the discovery probe with the other sibling-operator kinds (probeOptionalWatches, which skips the leg and registers a leader-gated re-check that restarts the operator once the CRD appears)
RabbitmqCluster (unstructured rabbitmqClusterGVK)	Owns() + cross-namespace Watches()	Re-reconciles when the managed message-bus child status changes, so InfrastructureReady follows AllReplicasReady instead of waiting for the periodic requeue. Watched as *unstructured.Unstructured, since the c5c3 operator takes no dependency on the RabbitMQ Cluster Operator's Go module. Messaging is opt-in, so both legs sit behind the discovery probe with the openbao kinds: a cluster that does not serve rabbitmqclusters.rabbitmq.com starts without them, and crdWatchGate restarts the operator once the CRD appears
NovaCompute	Watches()	Per-CR fan-out via novaComputeToControlPlaneMapper, matching the Nova namespace and the Nova name a pool's spec.novaRef names. Pools are user-authored and only read, for the clusters the compute contract is mirrored to, so they carry no owner reference. The predicate admits a pool being created, deleted or starting to be deleted, the events that change the mirror targets. The leg sits behind the discovery probe with the other nova-operator kinds
NeutronMetadataAgent	Watches()	Per-CR fan-out via neutronMetadataAgentToControlPlaneMapper, waking every ControlPlane with services.nova and services.neutron whose OVN central namespace is the agent's namespace. Agents are user-authored and only read, for the clusters the metadata shared secret is copied to, so they carry no owner reference. neutronMetadataAgentDeliveryPredicate admits an agent being created, deleted or starting to be deleted, and a change of spec.novaMetadata.sharedSecretRef.name; status writes and generic events are dropped. The leg sits behind the discovery probe with the other neutron-operator kinds
OVNCentral	Watches()	Per-CR fan-out via ovnCentralToControlPlaneMapper. The central is deployed outside the plane and only named by spec.services.neutron.ovn.centralRef, so it carries no owner reference an Owns() could match; the leg re-runs reconcileOVN when the central's status moves instead of waiting for the periodic requeue. The ovn-operator is installed only for a plane that runs a network service, so the leg sits behind the discovery probe with the other sibling-operator kinds
SizingProfile	Watches()	Per-CR fan-out via sizingProfileToControlPlaneMapper, which lists every ControlPlane and enqueues those whose spec.sizing.profileRef.name names the event's profile. The kind is cluster-scoped and only referenced, so it carries no owner reference an Owns() could match. The watch is unconditional, because the CRD ships in the c5c3 chart; a List error is logged and drops the event.
K-ORC ApplicationCredential	Owns()	Re-reconciles when the minted admin credential's Available condition or status.id changes
K-ORC Service	Owns()	Re-reconciles when the identity catalog Service changes
K-ORC Endpoint	Owns()	Re-reconciles when the public identity Endpoint changes
K-ORC User, Domain	Owns()	Re-reconciles when the admin identity imports change. These, the ApplicationCredential, the identity Service/Endpoint and the adopted Region are the K-ORC CRs the ControlPlane itself owns (orcChildObjects)
K-ORC Region	Owns()	Re-reconciles when the adopted bootstrap Region ({controlplane.Name}-region) reports status.id or its Available condition moves, which is what advances the two-phase description push
K-ORC Project, Role, RoleAssignment	Owns()	Registered, but no CR the ControlPlane projects carries them any more. A registration's project, role imports and role assignments are claimed by its own KeystoneService (claimKeystoneServiceChild: a controller owner reference to the registration when co-located with it, ownership labels otherwise), never by the ControlPlane, so an Owns() leg keyed on a ControlPlane owner reference does not match them. Such a change wakes the registration's controller, and the ControlPlane sees it one step later when the registration's status moves, through the legs below
ExternalSecret	Owns()	Re-reconciles when an owned ESO ExternalSecret (DB credential, admin password, K-ORC clouds.yaml) syncs or fails, so the credential conditions track ESO promptly
PushSecret	Owns()	Re-reconciles when the owned admin-credential PushSecret status changes
KeystoneService (co-located child)	Owns()	Re-reconciles when a registration child projected for a built-in service changes. Both gates the service legs apply are status-only reads, and the standalone mapper leg below drops exactly those writes, so without this leg a child going Ready would only reach the ControlPlane at the next periodic requeue
KeystoneService (placed child)	Watches()	The label-predicate twin, for a registration child of a service placed in a namespace of its own, which can carry no owner reference
KeystoneService (standalone)	Watches()	Per-CR fan-out via keystoneServiceToControlPlaneMapper, behind watch.CRUpdatePredicate(): which allowlisted namespaces host a registration is what decides where reconcileRegistrationTenantStores provisions a store, so a CR appearing or being deleted has to re-drive the plane. Such a CR is no child of the ControlPlane, so it is mapped by its own controlPlaneRef, and the predicate drops the KeystoneService controller's status-only writes, which cannot move the provisioning set
Secret	Watches()	Maps Secret events to referencing ControlPlane CRs via the ControlPlaneSecretNameIndexKey field indexer (secretToControlPlaneMapper)
ClusterSecretStore	Watches()	Per-ref fan-out via storeToControlPlaneMapper (bound to the shared watch.StoreRefFanOut for the cluster kind): a status change on a cluster-scoped store enqueues only the ControlPlanes whose effective spec.secretStoreRef resolves to it
SecretStore	Watches()	The namespaced twin, scoped to the store's own namespace, so a ControlPlane pinned to a per-tenant SecretStore reacts to its backend health (storeToControlPlaneMapper for the namespaced kind)
projected service children + Namespace (cross-namespace)	Watches()	Label-predicate twin of the Owns() rows for a service placed in a namespace of its own: a cross-namespace child carries no owner reference (Kubernetes forbids one), so Keystone / Horizon / Glance / GlanceBackend / Placement / Barbican / BarbicanSecretStore / Neutron / Cinder / CinderBackend / CinderBackupBackend / Nova / OpenBaoCluster / OpenBaoTenant / RabbitmqCluster / MariaDB / Memcached / ExternalSecret / KeystoneService — and the Namespace itself — are watched a second time through the ownership labels the projections stamp (crossNamespaceChildHandler gated by crossNamespaceChildPredicate), so same-namespace children keep flowing through Owns() alone and neither leg double-enqueues the other's objects

The Secret watch uses Watches() with a MapFunc rather than Owns() because the admin-password Secret (spec.korc.adminCredential.passwordSecretRef) is typically ESO-managed — it is owned by the ExternalSecret controller, not by the ControlPlane CR — so an owner-reference filter would never match it. The index-backed namespace List is exactly what wakes the ControlPlane when its admin password rotates, so the re-mint chain (see K-ORC admin credential chain) converges on watch delivery instead of waiting for the next periodic requeue.

Both store watches are per-ref fan-outs rather than blanket enqueues: a status transition (for example ESO losing the backend connection) wakes only the ControlPlanes whose effective spec.secretStoreRef resolves to the changed store — the cluster watch lists every namespace, the namespaced watch only the store's own. A ControlPlane pinned to a different store stays untouched. This is why the DB-credential, admin-password, and admin-credential sub-reconcilers can flip their conditions to SecretStoreNotReady the moment the ControlPlane's own backend becomes unreachable instead of waiting up to a full ESO refresh interval (default 1h) for the next per-secret re-sync. The ExternalSecret/PushSecret children are owned (controller reference), so Owns() wires them directly.

Target-cluster watch legs

Every leg above is pinned to the management cluster, with EngageLocalCluster and EngageNoProviderClusters on each one. The ControlPlane CR, the K-ORC CRs, and the service CRs live there whatever a service names, so a provider-cluster leg would only watch objects this operator never wrote.

The kinds the ControlPlane writes into the namespace of a service placed on a target cluster take the opposite pinning: no local engagement, provider clusters only, and a per-cluster filter (ClusterServesKind) that skips the leg on a cluster not serving the kind. Eleven kinds are watched that way: MariaDB, Memcached, Certificate, ExternalSecret, PushSecret, VaultDynamicSecret, SecretStore, Secret, OpenBaoCluster, OpenBaoTenant, and Namespace. Memcached and Certificate are watched as *unstructured.Unstructured under the same memcachedGVK / certificateGVK values the local legs use, neither kind having a Go module the operator can type against. Secret and SecretStore carry two legs each: one maps a child back through its ownership labels, the other maps an input through the same mapper the management-side watch uses. The workqueue deduplicates what the two produce.

What keeps a registered cluster from reaching a ControlPlane that never named it is RemoteRequestsAmong. For every request a mapper produces it reads that ControlPlane on the management cluster and compares the cluster the event arrived from against TargetClusterNames(), the deduplicated set of the eight per-service refs; a cluster outside the set drops the event. The set comes from the CR, never from the object that raised the event, so an object planted in a shared namespace on any registered cluster cannot name a ControlPlane it does not belong to. A CR already deleted answers NotFound and the event is dropped in silence; any other read error is logged and the event dropped too. Surviving requests carry no cluster name, so the reconcile they trigger runs against the management cluster, where the CR is.

Secret Field Indexer

The controller registers a controller-runtime field indexer on the ControlPlane kind so that a Secret event resolves to the referencing ControlPlane CR(s) via an O(1) cache lookup instead of an unfiltered namespace-scoped List, mirroring the keystone operator's KeystoneSecretNameIndexKey.

Aspect	Value
Index key	ControlPlaneSecretNameIndexKey = "spec.korc.adminCredential.passwordSecretRef.name" (exported package-level constant in operators/c5c3/internal/controller/controlplane_controller.go)
Indexed fields	spec.korc.adminCredential.passwordSecretRef.name (the effective admin-password Secret), the CA bundle of the Keystone endpoint K-ORC dials, and spec.services.nova.remoteCompute.transportURLSecretRef.name while that block is set, deduplicated. The extractor (controlPlaneSecretNameExtractor) returns an empty slice when the name is unset so an unset field does not pollute the index, and returns nil if invoked with the wrong type rather than panicking.
Registration site	SetupWithManager → registerControlPlaneSecretNameIndex(ctx, local.GetFieldIndexer()), invoked before the Watches(Secret, …) chain. It goes on the local field indexer rather than the multicluster manager's: with a provider configured, that one registers its indexes against the target clusters, which hold no ControlPlane CR, and applying it while engaging a cluster would fail that engagement. Every request the legs emit is pinned to the management cluster, so a remote event still resolves its CR through this index. Any error from IndexField is wrapped with the index key and propagated, so manager startup aborts loudly if registration fails.
Lookup site	secretToControlPlaneMapper(local.GetClient()) — performs a namespace-scoped client.List with client.MatchingFields{ControlPlaneSecretNameIndexKey: secret.Name}. On List error the error is logged via log.FromContext and the mapper returns nil per the handler.MapFunc contract (it must not return errors).
Result	Each matching ControlPlane in the Secret's namespace is enqueued as a reconcile.Request; an event matching no ControlPlane returns nil.

Why no owner-ref fallback? Unlike the keystone operator, the c5c3 Secret mapper has a pure index-backed lookup with no owner-reference fallback branch — the ControlPlane projects no rotation-staging Secrets that are owned-but-unreferenced, so the union/owner-ref complexity of secretToKeystoneMapper is not needed here.

---

Reconciler Struct

type ControlPlaneReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

Field	Type	Purpose
Client	client.Client	Kubernetes API client for CRUD operations (embedded)
Scheme	*runtime.Scheme	Runtime scheme for owner-reference resolution
Recorder	record.EventRecorder	Records Kubernetes events for state transitions

The CredentialRotationReconciler has the identical three-field shape (client.Client embedded, Scheme, Recorder).

---

RBAC Permissions

RBAC markers on the two reconcilers generate the required ClusterRole. The ControlPlaneReconciler markers (in controlplane_controller.go):

API Group	Resources	Verbs
c5c3.io	controlplanes	get, list, watch, create, update, patch, delete
c5c3.io	controlplanes/status	get, update, patch
c5c3.io	controlplanes/finalizers	update
c5c3.io	credentialrotations	get, list, watch, create, update, patch, delete
c5c3.io	credentialrotations/status	get, update, patch
c5c3.io	secretaggregates, sizingprofiles	get, list, watch
scheduling.k8s.io	priorityclasses	get
k8s.mariadb.com	mariadbs	get, list, watch, create, update, patch, delete
memcached.c5c3.io	memcacheds	get, list, watch, create, update, patch, delete
rabbitmq.com	rabbitmqclusters	get, list, watch, create, update, patch, delete
keystone.openstack.c5c3.io	keystones	get, list, watch, create, update, patch, delete
placement.openstack.c5c3.io	placements	get, list, watch, create, update, patch, delete
barbican.openstack.c5c3.io	barbicans, barbicansecretstores	get, list, watch, create, update, patch, delete
neutron.openstack.c5c3.io	neutrons	get, list, watch, create, update, patch, delete
neutron.openstack.c5c3.io	neutronmetadataagents	get, list, watch
cinder.openstack.c5c3.io	cinders, cinderbackends, cinderbackupbackends	get, list, watch, create, update, patch, delete
nova.openstack.c5c3.io	novas	get, list, watch, create, update, patch, delete
ovn.openstack.c5c3.io	ovncentrals	get, list, watch
openbao.org	openbaoclusters, openbaotenants	get, list, watch, create, update, patch, delete
rbac.authorization.k8s.io	roles, rolebindings, clusterrolebindings	get, create, patch, delete
rbac.authorization.k8s.io	clusterroles (resourceNames: system:auth-delegator)	bind
openstack.k-orc.cloud	applicationcredentials, services, endpoints, regions, users, domains, projects, roles, roleassignments	get, list, watch, create, update, patch, delete
openstack.k-orc.cloud	applicationcredentials/status, services/status, endpoints/status, regions/status, users/status, domains/status, projects/status, roles/status, roleassignments/status	patch
external-secrets.io	externalsecrets, pushsecrets	get, list, watch, create, update, patch, delete
external-secrets.io	clustersecretstores, secretstores	get, list, watch
generators.external-secrets.io	vaultdynamicsecrets, passwords	get, list, watch, create, update, patch, delete
core	secrets	get, list, watch, create, update, patch, delete
core	events	create, patch

The sizingprofiles grant serves the Sizing step's read of the referenced profile and the watch on the kind. The ControlPlane and SizingProfile webhooks run under the same identity and look up the PriorityClasses a sizing names, which is what the priorityclasses grant is for.

The patch grant on the K-ORC status subresources lets the reconcilers clear a latched K-ORC transport error from a child's status (unlatchKORCTransportErrors in korc_unlatch.go).

The CredentialRotationReconciler markers (in reconcile_credentialrotation.go) are scoped tighter — it never mints, so it holds only update/patch (not create/delete) on K-ORC applicationcredentials and read-only access to controlplanes:

API Group	Resources	Verbs
c5c3.io	credentialrotations	get, list, watch, create, update, patch, delete
c5c3.io	credentialrotations/status	get, update, patch
c5c3.io	controlplanes	get, list, watch
openstack.k-orc.cloud	applicationcredentials	get, list, watch, update, patch
core	secrets	get, list, watch
core	events	create, patch

Blast radius and namespace scoping

By default the chart binds these markers to a cluster-wide ClusterRole, so the secrets rule lets a compromised operator pod read and write every Secret in every namespace; the Multi-Tenant Deployment → Security trade-off details that privilege-escalation path. Two specifics apply to this operator:

• It amplifies the exposure itself: reconcileAdminPassword and reconcileKORC project the OpenStack admin password in cleartext into a clouds.yamlSecret, so cluster-wide read access exposes every projected admin password.
• It writes RBAC. The dedicated Barbican secret store needs get, create, patch, and delete on roles, rolebindings, and clusterrolebindings to give the OpenBao instance its TokenReview and the barbican operator a bound provisioner token. What a forged binding can grant is still capped by the API server's privilege-escalation check: the operator holds escalate nowhere and bind only on system:auth-delegator (narrowed by resourceNames), so it can hand out no permission it does not already hold. The cluster-wide Secret read therefore remains the dominant risk.

The c5c3-operator chart refuses rbac.namespaceScoped: true: the render fails with rbac.namespaceScoped=true is not supported by c5c3-operator. The ControlPlane controller watches the cluster-scoped Namespace, SizingProfile and ClusterSecretStore kinds and creates ClusterRoleBinding objects. A namespaced Role grants none of them, so the manager would fail its cache sync at startup. Placing a service in a dedicated namespace needs the cluster-scoped namespaces verbs as well, which is why the markers add core/namespaces with get;list;watch;create;delete. See When cluster-wide RBAC is still required.

---

Reconciliation Flow

A pass runs a blocking prefix of seven steps and then one group of fourteen members that all run. The figure draws an arrow only where one condition gates another.

One pass, in order:

1. Fetches the ControlPlane. NotFound ends the pass with an empty result.
2. Hands a ControlPlane that carries a deletion timestamp to reconcileDelete (see Owner-ref / GC model).
3. Runs the duplicate guard: every ControlPlane but the oldest in a namespace is parked with Ready=False/DuplicateControlPlane and requeued after 30s, and none of its sub-reconcilers run (see Multi-instance).
4. Installs the finalizer c5c3.io/orc-teardown, and openstack.c5c3.io/remote-children once a target cluster the spec names resolves. A pass that installs one ends with a requeue.
5. Runs the blocking prefix through RunPipeline, rows 1 to 7 below. The first step that returns a requeue or an error ends the pass.
6. Runs the tail group through RunSequentialGroup, rows 8 to 21. Every member runs. Member requeues collapse to the shortest (ShortestRequeue), member errors are joined (errors.Join).
7. Calls updateStatus() on every exit path of steps 5 and 6: it recomputes the aggregate Ready as AllTrue(subConditionTypes), writes status.services, stamps status.observedGeneration, and skips the write when nothing changed.

The 21 sub-reconcilers, in call order. The Gate column names the conditions a step checks itself and, after them, what else it waits for. A prefix step also runs only after the step before it has converged.

#	Step	Sets	Gate	Does	Requeue
1	reconcileSizing	SizingReady	nothing	Resolves spec.sizing from the built-in profile or a SizingProfile	none: a missing or unreadable profile returns the error
2	reconcileNamespaces	NamespacesReady	nothing	Ensures or verifies the namespaces services are placed in. True/NoDedicatedNamespaces when no service declares one	15s while a namespace is unusable
3	reconcileInfrastructure	InfrastructureReady	nothing	Ensures the managed MariaDB, Memcached and RabbitMQ children	15s while a child is not Ready
4	reconcileESOTenantStore	ESOTenantStoreReady	nothing	Provisions ServiceAccount, client Certificate and the SecretStore openbao-tenant-store in every namespace the ControlPlane occupies. True/StoreRefOverridden when spec.secretStoreRef is set	10s while a store is not Ready
5	reconcileDBCredentials	DBCredentialsReady	the secret store, and in Dynamic mode the tenant onboarding a person runs (setup-database-tenant.sh)	Projects the DB-credential ExternalSecret of Keystone	10s while it has not synced (WaitingForDBCredentialSecret)
6	reconcileAdminPassword	AdminPasswordReady	the secret store	Projects the admin-password ExternalSecret	10s while it has not synced
7	reconcileKeystone	KeystoneReady	InfrastructureReady	Projects the Keystone child	5s gated, 15s while the child is not Ready
8	reconcileHorizon	HorizonReady	KeystoneReady	Projects the Horizon child	5s gated, 15s child
9	reconcileKORC	KORCReady	no condition. The admin-password Secret, and the Keystone CA bundle when one is referenced	Writes the password-based clouds.yaml, has K-ORC mint the admin ApplicationCredential, and drives a re-mint	10s
10	reconcileAdminCredential	AdminCredentialReady	KORCReady	Assembles the credential Secret, pushes it to OpenBao and waits until k-orc-clouds-yaml carries it	10s
11	reconcileCatalog	CatalogReady	AdminCredentialReady	Managed mode: registers the identity Service, its public Endpoint and the adopted Region. External mode: imports the catalog	10s
12	reconcileGlance	GlanceReady	KeystoneReady. Then its registration's AccountReady and its DB credential	Projects the KeystoneService registration, the DB credential, the Glance child and its backends	5s gated, 10s registration or DB credential, 15s child
13	reconcilePlacement	PlacementReady	KeystoneReady. Then its registration's AccountReady and its DB credential	Projects the registration, the DB credential and the Placement child	5s gated, 10s registration or DB credential, 15s child
14	reconcileBarbican	BarbicanReady	KeystoneReady. Then its registration's AccountReady, its DB credential, and on a dedicated secret store the OpenBao instance serving (WaitingForOpenBaoInstance)	Projects the registration, the DB credential, the dedicated OpenBao instance, the secret store and the Barbican child	5s gated, 10s registration or DB credential, 15s instance or child
15	reconcileOVN	OVNReady	nothing	Reads the OVNCentral that services.neutron.ovn.centralRef names and mirrors its readiness. Writes nothing	15s while the central is not usable
16	reconcileNeutron	NeutronReady	KeystoneReady, OVNReady. Then the message bus delivered into its namespace, its registration's AccountReady (and the notifier's while Nova is set) and its DB credential	Delivers the bus, projects the registrations, the DB credential and the Neutron child	5s gated, 15s bus or child, 10s registration or DB credential
17	reconcileCinder	CinderReady	KeystoneReady. Then the bus, its registration's AccountReady and its DB credential	Delivers the bus, projects the registration, the DB credential, the CinderBackend and CinderBackupBackend satellites and the Cinder child	5s gated, 15s bus or child, 10s registration or DB credential
18	reconcileNova	NovaReady	KeystoneReady, PlacementReady. Then the bus, its registration's AccountReady and its two DB credentials	Delivers the bus, projects the registration, the two DB credentials, the metadata shared secret and the Nova child, then the hypervisor operator's account when it is set	5s gated, 15s bus or child, 10s registration or DB credential
19	reconcileServiceAccounts	ServiceAccountsReady	nothing	Folds the Ready of the registrations the legs projected, up to eight	10s while one is not Ready
20	reconcileRegistrationTenantStores	RegistrationTenantStoresReady	nothing	Provisions the tenant-store trio in each allowlisted namespace that holds a registration. True/NoRegistrationNamespaces when there is none	10s while a store is not Ready
21	reconcileNamespaceAssignments	NamespaceAssignmentsReady	nothing	Reports every spec.namespaceAssignments entry in status.namespaceAssignments. True/NoNamespaceAssignments when there is none	1m while an entry is not Assigned

A service whose block is not set reports its condition True with a NotManaged reason, so the aggregate is not blocked: KeystoneNotManaged, HorizonNotManaged, GlanceNotManaged, PlacementNotManaged, BarbicanNotManaged, OVNNotManaged, NeutronNotManaged, CinderNotManaged, NovaNotManaged.

Execution Model

The chain runs in two phases over the shared scaffolding in internal/common/reconcile (the same building blocks the keystone controller uses). Every step is a commonreconcile.Step, and the whole table is driven by one commonreconcile.RunPipeline call:

Phase 1 — the blocking prefix. Sizing → Namespaces → Infrastructure → ESOTenantStore → DBCredentials → AdminPassword → Keystone run as seven named, short-circuiting Step entries. RunPipeline returns the pass at the first non-zero result or error, because each step genuinely feeds the next — a later step applying before its predecessor converged would fail or wedge. Each named step is wrapped in instrumenter.Instrument (see Metrics Instrumentation) so a duration sample and an error counter are emitted under a stable sub_reconciler label.

Phase 2 — the tail group. Horizon, KORC, AdminCredential, Catalog, Glance, Placement, Barbican, OVN, Neutron, Cinder, Nova, ServiceAccounts, RegistrationTenantStores and NamespaceAssignments are the fourteen named members of one commonreconcile.RunSequentialGroup, embedded as the pipeline's final bare (unnamed) Step. The group members self-instrument through instrumenter.Instrument — following the keystone self-instrumenting-group convention — which is why the enclosing group step carries no sub_reconciler name of its own. RunSequentialGroup attempts every member on every pass and never short-circuits: each member self-gates on the conditions it needs (Horizon on KeystoneReady; KORC until the admin-password Secret is readable; AdminCredential on KORCReady; Catalog on AdminCredentialReady; Glance, Placement, Barbican, Neutron, Cinder and Nova on KeystoneReady and on the AccountReady of the KeystoneService registration each projects for itself, see Built-in service registrations), so running all of them each pass is safe. Barbican carries one gate the others do not: on a dedicated secret store it holds the projection until the OpenBao instance it provisions serves requests. Neutron, Cinder and Nova also wait for the shared message bus to be delivered into their namespace. Neutron gates on OVNReady as well, and Nova on PlacementReady. OVN itself carries none, because the OVNCentral it mirrors is deployed outside the plane and nothing this chain produces can converge it.

Of the three members that close the group, the first depends on its order in it. ServiceAccounts carries no gate: it only folds the registration children the service legs wrote earlier in the same pass, so it must run after them and has nothing to defer. RegistrationTenantStores carries no gate either. It consumes no condition this chain produces — the trio it writes depends on cert-manager and OpenBao alone, exactly like its blocking-prefix twin — and sits in the group rather than in that prefix so a namespace the control plane does not own can never park DBCredentials, AdminPassword and Keystone behind it. NamespaceAssignments carries no gate and only reads (see reconcileNamespaceAssignments).

pipeline := []commonreconcile.Step{
    {Name: "Sizing", Fn: func(ctx context.Context) (ctrl.Result, error) {
        return r.reconcileSizing(ctx, &cp)
    }},
    {Name: "Namespaces", Fn: func(ctx context.Context) (ctrl.Result, error) {
        return r.reconcileNamespaces(ctx, &cp)
    }},
    // ... Infrastructure, ESOTenantStore, DBCredentials, AdminPassword, Keystone
    //     — the blocking prefix, each short-circuiting via RunPipeline.
    //
    // The independent tail projections run as one self-instrumenting,
    // non-short-circuiting group embedded as the final bare Step:
    {Fn: func(ctx context.Context) (ctrl.Result, error) {
        return commonreconcile.RunSequentialGroup(ctx, instrumenter.Instrument,
            []commonreconcile.Step{
                {Name: "Horizon", Fn: /* ... */},
                // KORC, AdminCredential, Catalog, Glance, Placement,
                // Barbican, OVN, Neutron, Cinder, Nova, ServiceAccounts,
                // RegistrationTenantStores, NamespaceAssignments
            })
    }},
}
result, err := commonreconcile.RunPipeline(ctx, instrumenter.Instrument, pipeline)
return r.updateStatus(ctx, &cp, statusBefore, result, err)

This guarantees:

1. Prefix (phase 1) — unchanged early-return semantics. A prefix sub-reconciler error propagates immediately and a non-zero result (RequeueAfter > 0) causes an early return; either way the subsequent prefix steps and the whole tail group are skipped for that pass.
2. Group (phase 2) — every member runs each pass. No member's non-zero result or error prevents a later member from running, so a still-converging or failing Horizon no longer parks KORC, the AdminCredential/Catalog identity bootstrap, Glance, Placement, Barbican, OVN, Neutron, Cinder, Nova, or the four members that close the group. Each member's condition therefore always persists.
3. Group result aggregation. When no member errors, the group result is the shortest member requeue (commonreconcile.ShortestRequeue) and the error is nil. When one or more members error, the group returns ctrl.Result{} and the errors.Join of every member error — member requeues are discarded, and controller-runtime's error backoff drives the retry.
4. Status always persisted. Whichever phase ends the pass, its outcome funnels through updateStatus(), so the conditions and the (result, error) pair are persisted by construction on every exit path.

Onboarding rule. A future service whose projection is independent of the others — it self-gates on the conditions it needs rather than being a hard prerequisite for a later step — joins the tail group, not the blocking prefix. Only a projection that genuinely feeds the next step belongs in the prefix.

Status Update Pattern

updateStatus() delegates to the shared commonreconcile.UpdateStatus: it stamps cp.Status.ObservedGeneration = cp.Generation, records the per-service status (setServicesStatus(), see below), persists all condition changes via r.Status().Update(), and returns the provided (result, error) pair.

Reconcile snapshots cp.Status immediately after the initial Get and threads that snapshot into updateStatus, which compares the computed status against it with equality.Semantic.DeepEqual and skips the write when a pass left status unchanged — no write means no watch event and no resourceVersion churn on a converged steady-state pass. Together with the watch.CRUpdatePredicate on the controller's For(...) watch (which filters the CR's own status-only updates), this closes the self-wake loop the previous always-write updateStatus and bare For() allowed.

When both a reconcile error and the status update fail, both errors are preserved via errors.Join so the original reconcile failure remains visible in controller-runtime logs:

reconcileErr	Status().Update()	Returned error
nil	succeeds	nil
non-nil	succeeds	reconcileErr (unchanged)
nil	fails	errors.Join(nil, fmt.Errorf("updating status: %w", statusErr))
non-nil	fails	errors.Join(reconcileErr, fmt.Errorf("updating status: %w", statusErr))

Because ObservedGeneration is stamped on every updateStatus call (early return or final), a stale status is always distinguishable from a current one.

Ready Condition Aggregation

On every status write, including a pass that ends early, setReadyCondition() evaluates whether every sub-condition type is True using aggregateReady(), which delegates to conditions.AllTrue(conds, subConditionTypes...):

All Sub-Conditions True	Ready Condition	Reason	Message
Yes	Status: True	AllReady	All sub-conditions are ready
No (any missing or False)	Status: False	NotAllReady	One or more sub-conditions are not ready

The aggregated sub-condition types (the source-of-truth subConditionTypes slice in controlplane_controller.go) are:

SizingReady, NamespacesReady, InfrastructureReady, ESOTenantStoreReady, DBCredentialsReady, KeystoneReady, HorizonReady, GlanceReady, PlacementReady, BarbicanReady, OVNReady, NeutronReady, CinderReady, NovaReady, KORCReady, AdminCredentialReady, AdminPasswordReady, CatalogReady, ServiceAccountsReady, RegistrationTenantStoresReady, NamespaceAssignmentsReady

The Ready condition carries ObservedGeneration = cp.Generation so clients can detect a stale aggregate.

One path bypasses the aggregation entirely: a ControlPlane parked by the duplicate guard (see Multi-instance) gets Ready=False with reason DuplicateControlPlane written directly — setReadyCondition() would otherwise overwrite the reason with NotAllReady on the next status update.

Services and Update Phase

setServicesStatus() runs on every updateStatus call and populates two status fields that the schema declared but the reconciler previously never wrote:

Field	Value
status.updatePhase	Fixed at Idle — the release-update state machine is not implemented and the other UpdatePhase values are reserved, so "no update in progress" is the current state
status.services	one entry per managed service, in a stable order: keystone (present when spec.services.keystone is set), then horizon (present when spec.services.horizon is set), then glance (present when spec.services.glance is set), then placement (present when spec.services.placement is set), then barbican (present when spec.services.barbican is set), then neutron (present when spec.services.neutron is set), then cinder (present when spec.services.cinder is set), then nova (present when spec.services.nova is set). Each entry's ready mirrors the matching KeystoneReady / HorizonReady / GlanceReady / PlacementReady / BarbicanReady / NeutronReady / CinderReady / NovaReady sub-condition (via conditions.AllTrue) and release is spec.openStackRelease; an unmanaged service is omitted rather than reported

---

Sub-Reconciler Contracts

Each sub-reconciler owns exactly one Ready sub-condition. The tables below give each one's gate, what it projects/owns, and the condition reasons it sets on the True, requeue, and error paths. All condition constants are the exported source-of-truth strings in controlplane_controller.go; sub-reconcilers reference the constants (never inline literals) so a rename is a compile error and is caught by the no-inline-literals drift guard.

Every condition is stamped with ObservedGeneration = cp.Generation on every path.

Which cluster a sub-reconciler writes to. A sub-reconciler that projects into a service namespace does not reach for the reconciler's own client. It resolves one per namespace through childrenClientFor, which maps the namespace to the target cluster of the service placed there and hands back that cluster's client. The ControlPlane's own namespace answers with the local client, and so does any namespace no placed service declares. Co-located services cannot disagree about the answer, because admission rejects a shared namespace whose services name different clusters, so the namespace alone decides. An ensemble that spans several namespaces resolves all of them through childrenClientsFor before its first write: a cluster that does not resolve then leaves the whole ensemble unwritten rather than half of it, under TargetClusterUnavailable on the sub-reconciler's own condition, carrying the resolver's message verbatim (cluster not found for a name that was never registered).

A nil Resolver resolves nothing. The reconciler is then running without a multicluster manager, which is how the unit tests and the single-cluster envtest fixtures construct it, and every namespace answers with the local client whatever the spec names.

External keystone mode and the chain

When spec.services.keystone.mode is External, the ControlPlane manages identity against a pre-existing, externally-operated Keystone. The chain keeps its order; External mode changes what each link does. Four sub-reconcilers short-circuit — reconcileInfrastructure, reconcileDBCredentials, reconcileAdminPassword and reconcileKeystone — each reporting its own condition with Status=True and reason ExternallyManaged, and a message naming spec.services.keystone.external.authURL.

Skipped sub-reconcilers keep their condition types. The condition schema is therefore identical across modes, so subConditionTypes, setReadyCondition and the condition_type drift guard need no mode awareness.

Every skip is keyed on the mode discriminator cp.IsExternalKeystone(), never on the database shape: an External-mode ControlPlane has no spec.infrastructure block at all, so "no managed database" and "no database" are different states. The vocabulary keeps three "nothing was projected" reasons deliberately apart:

Reason	Meaning
ExternallyManaged	identity is managed against a pre-existing Keystone; this sub-reconciler has nothing to project
KeystoneNotManaged	spec.services.keystone is unset: there is no identity plane at all (staged adoption)
BrownfieldUserSuppliedCredential	the ControlPlane owns a Keystone, but the database is brownfield, so the user supplies its credential Secret

services.horizon is forbidden in External mode, so reconcileHorizon always takes its HorizonNotManaged early-exit. The duplicate-ControlPlane parking guard is mode-agnostic and applies unchanged — External-mode ControlPlanes count towards the one-per-namespace contract.

reconcileCatalog neither skips nor behaves as it does in Managed mode: it forks. The catalog belongs to the external installation, so External mode is import-first — the existing identity service and its endpoints are imported read-only, zero catalog entries are created by default, and an import that resolves to nothing fails loud rather than waiting forever. See reconcileCatalog.

Egress and TLS posture

K-ORC — not the c5c3-operator — is what dials the external Keystone. It is installed by the Flux Kustomization deploy/flux-system/releases/k-orc.yaml into the orc-system namespace, and the operator never opens an OpenStack connection itself: everything stays K-ORC-mediated.

• Egress. No NetworkPolicy exists anywhere under deploy/, and none scopes orc-system, so nothing restricts K-ORC's egress on the shipped stack — out-of-cluster Keystone worked first-pass in the phase-1 spike. A cluster that applies a default-deny egress policy to orc-system must add an explicit allow rule to the external endpoint's host and port, or every mint, import and catalog call fails as EndpointUnreachable.
• TLS. An IP-based authURL requires an IP SAN in the external Keystone's server certificate — a CN or DNS SAN alone will not verify. Hostnames resolve through cluster DNS via the upstream forwarder, so no extra DNS wiring is needed. A privately-signed certificate needs spec.services.keystone.external.caBundleSecretRef; without it K-ORC reports an x509 failure, classified onto KORCReady=False/TLSVerificationFailed.
• CA-cache aliasing. K-ORC's provider-client cache keys on the parsed cloud struct only — cacert is not part of the key (internal/scope/provider.go) — and the entry lives for the token lifetime / 2 (≈30 min at Keystone defaults). A rotated or removed CA bundle therefore converges the Secrets immediately but the trust store only after cache expiry. Nothing in this operator can shorten that window; an upstream fix would have to fold cacert into the cache key.

Reaching a placed service

An in-cluster Service DNS name resolves only inside the cluster the Service runs on. Every URL the ControlPlane composes therefore depends on where the two ends sit, and the rule is the same everywhere: the in-cluster URL while both resolve to the same cluster (sameTargetCluster compares the two targetClusterRefs, with "no ref" meaning the management cluster), the public URL as soon as they do not.

• The seven dependents of Keystone. horizonKeystoneEndpoint, glanceKeystoneEndpoint, placementKeystoneEndpoint, barbicanKeystoneEndpoint, neutronKeystoneEndpoint, cinderKeystoneEndpoint and novaKeystoneEndpoint compare their own service's ref against Keystone's. A service that shares Keystone's cluster keeps the conventional http://{controlplane.Name}-keystone.<keystone-namespace>.svc:5000/v3; one on another cluster gets keystonePublicEndpoint — services.keystone.publicEndpoint when set, else the gateway-derived https://{gateway.hostname}/v3.
• K-ORC and the service accounts. korcAuthURL takes the cluster the document is CONSUMED on and answers by the same rule. The two admin documents are read by K-ORC, which always runs on the management cluster wherever the ControlPlane places its services, so they render the public URL as soon as Keystone names a cluster — a placed Keystone's Service DNS name is an address K-ORC cannot dial. A registration's clouds.yaml is rendered with no target-cluster ref, because a registration is delivered on the management cluster. It resolves like a document read there. External mode is untouched, and a co-located Keystone renders the in-cluster URL byte for byte as before.
• The catalog. A placed service registers its public URL on its internal interface as well as its public one (internalCatalogURL), because that entry is what K-ORC and every consumer outside the service's cluster resolve. The identity row is unaffected: it registers a public interface only.

Admission is what keeps those public URLs from being empty. A placed catalog service (keystone, glance, placement, barbican, neutron, cinder, nova) must declare a publicEndpoint or a gateway, and Keystone must declare one as soon as ANY other service is placed away from it — the per-service rule only reaches a service carrying a ref of its own, so an unplaced Keystone would otherwise leave every dependent child with an empty spec.keystoneEndpoint its own CRD refuses. A placed Keystone's endpoint must additionally use https, because placement promotes it to the auth_url the credential documents render passwords next to. Composing those URLs is where this ends: making them routable between clusters is the deployment's job, a gateway or a load balancer per target cluster.

Projected satellites and the shared bus

Two shapes recur across services: a satellite child projected per spec entry, and the ControlPlane-wide message bus delivered into a service's namespace. Each is implemented once, and a service opts in by naming itself.

projectedChildren names one projected satellite kind of one service: an empty typed list, the kind name every wrapped error carries, the namespace the children live in, the name prefix every projected child carries, and a Keep set of names never deleted and never reported (nil keeps nothing). ownedProjectedChildren lists the kind in that namespace on the management client and returns, in list order, the objects this ControlPlane owns (isControlPlaneChild) whose name carries the prefix and is not in Keep.

pruneProjectedChildren is the reconcile-time arm. It deletes those objects with background propagation and ignores NotFound, so a child the projection no longer declares is removed on the same pass. sweepProjectedChildren is the teardown arm: a child already Terminating is reported without a second delete, and every owned child is reported as namespace/name for teardownDedicatedNamespaces to wait on. An absent satellite CRD (meta.IsNoMatchError on the List) is nothing to sweep. That is the reading deleteORCResources applies to an absent K-ORC stack, and it lets a ControlPlane whose service operator was uninstalled first finish deleting.

Each service supplies a constructor: glanceBackendChildren for the GlanceBackend kind under the {controlplane.Name}-glance- prefix (see reconcileGlance), barbicanSecretStoreChildren for the BarbicanSecretStore kind under {controlplane.Name}-barbican- (see reconcileBarbican), and cinderBackendChildren / cinderBackupBackendChildren for the two satellite kinds of the block-storage service (see reconcileCinder).

Site	Helper	Keep
reconcileGlanceBackends	pruneProjectedChildren	the backend names the pass declared
deleteOrphanedGlance	pruneProjectedChildren	nil
reconcileBarbicanSecretStore	pruneProjectedChildren	the store the projection names
reconcileCinderBackends	pruneProjectedChildren	the backend names the pass declared
reconcileCinderBackupBackend	pruneProjectedChildren	the backup-backend name the projection declares, nil when the block is unset
deleteOrphanedCinder, both satellite kinds	pruneProjectedChildren	nil
deleteServiceChildrenIn, Glance namespace	sweepProjectedChildren	nil
deleteServiceChildrenIn, Barbican namespace	sweepProjectedChildren	barbicanSecretStoreName(cp)
deleteServiceChildrenIn, Cinder namespace, both satellite kinds	sweepProjectedChildren	nil

The Barbican teardown keeps the projected store because crossNamespaceServiceChildren already holds it in the wait set, so the sweep reports it once.

The two Cinder constructors carry Prefix: "", so ownership alone selects their satellites. A CinderBackend is named after its services.cinder.backends entry rather than after the ControlPlane, because cinder keys every volume by the backend it was created on and a rename would strand the volumes already there, so there is no prefix to recognise it by. The bare name is also what a person naming a hand-made object of that kind picks, which is why every satellite write goes through ensureProjectedSatellite instead of ensureUnownedOrOwned: the adoption pre-check runs in every namespace, the ControlPlane's own included, and refuses a same-named object this ControlPlane did not create. Without it the foreign object would become a child of the plane, and the prune that follows each projection would delete it as soon as the entry was dropped.

serviceMessagingTarget names a bus consumer by four fields: Service (the display name in the <Service>MessagingError reason and in the wrapped texts), ChildName (what the two Secrets are named after), Namespace (where they are written, on the client childrenClientFor resolves for it), and ConditionType (the ControlPlane condition every arm reports on). reconcileServiceMessaging resolves spec.infrastructure.messaging in the ControlPlane's own namespace on the management cluster (messaging.ResolveTransportURL) and writes the transport-URL Secret, plus the CA mirror on a TLS bus. serviceMessagingSpec renders the child's brownfield spec.messaging, messagingCAMirrorReleasable is the reap gate for the mirror, pruneServiceMessagingCA performs the reap, and serviceMessagingSecrets returns the two Secret stubs both teardown paths delete. Three services name themselves as targets. neutronMessagingTarget(cp) names Neutron, the {controlplane.Name}-neutron child, cp.NeutronNamespace() and NeutronReady, so the delivery lands as {controlplane.Name}-neutron-messaging and {controlplane.Name}-neutron-messaging-ca (see reconcileNeutron). cinderMessagingTarget(cp) names Cinder, the {controlplane.Name}-cinder child, cp.CinderNamespace() and CinderReady, so its delivery lands as {controlplane.Name}-cinder-messaging and {controlplane.Name}-cinder-messaging-ca (see reconcileCinder). novaMessagingTarget(cp) names Nova, the {controlplane.Name}-nova child, cp.NovaNamespace() and NovaReady, for {controlplane.Name}-nova-messaging and {controlplane.Name}-nova-messaging-ca (see reconcileNova). Each Secret carries the ControlPlane's own name rather than the one the service operator claims for the Secret it derives from spec.messaging.secretRef, so no two controllers rewrite one object. In the Nova namespace that rule is what keeps the ControlPlane's delivery apart from the {controlplane.Name}-nova-transport-url Secret the nova operator derives for itself.

Admission carries the same two shapes. projectedChildNameBound bounds a child name composed as {controlplane.Name} + infix + entry name against the apiserver's 253-byte metadata.name cap, and validateGlanceBackends calls it with the -glance- infix and the backend entry noun the remediation tells the operator to shorten. validateMessagingConsumers requires spec.infrastructure.messaging once per declared service whose child CRD requires spec.messaging; Neutron, Cinder and Nova are the three such services. The CinderBackend name carries a composed bound of its own that projectedChildNameBound does not express: validateCinderBackends keeps len(controlplane.Name) + 7 + len(entry name) at or below 47, the budget the satellite shares with its cinderRef in the <cinder>-<backend>-service-remove Job name a detach spawns.

reconcileSizing

Aspect	Value
File	reconcile_sizing.go
Condition	SizingReady
Gate	none; the first step of the pipeline
Projects / Owns	nothing; resolves spec.sizing
Requeue	none; a failure returns the error, which stops the pass and requeues with backoff

reconcileSizing reads the SizingProfile that spec.sizing.profileRef names through the cached client (the kind is watched) and reports the resolved base. A built-in profile needs no read.

Status	Reason	When
True	SizingResolved	sizing resolved from built-in profile "<base>", or sizing resolved from SizingProfile "<name>" (base "<base>").
False	SizingProfileNotFound	The profile is gone: SizingProfile "<name>" not found; the children keep their last projected sizing. The step returns the wrapped NotFound error of the read.
False	SizingProfileError	Any other read error; the step returns it.

Returning the error follows the InvalidRotationInterval precedent in reconcileKeystone: no later step runs, so no child is projected from a sizing that could not be resolved, and each keeps what it was last projected with.

Every consumer resolves the sizing for itself through effectiveSizing (c5c3v1alpha1.ResolveSizing over the read profile) rather than reading a value this step stored, because the members of a parallel group keep only the conditions and metadata they write. sizing_projection.go holds the rules they share: the replica fallback, the placement fallback to the top-level spec.sizing values, the spread completion with the child's pod selector, and the uWSGI, Job and Glance apiServer helpers. See SizingSpec → Projection.

reconcileNamespaces

Aspect	Value
File	reconcile_namespaces.go
Condition	NamespacesReady
Gate	none (second in the prefix, after reconcileSizing)
Projects / Owns	Namespace objects for every service placed in a namespace of its own under the Managed lifecycle, on the management cluster and on the target cluster of a service that names one
Requeue	namespaceRequeueAfter = 15s while a namespace is unusable

reconcileNamespaces ensures the namespaces the ControlPlane's services are placed in outside its own (see Service Namespaces), and runs ahead of every sub-reconciler that projects, because each of them projects into one of them — applying into a namespace that does not exist fails with an error naming neither the ControlPlane nor the assignment behind it. A ControlPlane with no assignments (the default) has nothing to ensure and reports NamespacesReady=True immediately, so the step costs nothing on the common path.

The two lifecycles are asymmetric. Under Managed the operator creates the namespace and stamps it with the ownership labels plus app.kubernetes.io/managed-by, and the annotation c5c3.io/controlplane-uid, which holds the ControlPlane's UID. Adoption requires it on a target cluster. A namespace that already exists without the labels is never adopted — the condition fails loud rather than taking over a namespace it did not create. On a target cluster the mark is required on top of them, because the labels alone are forgeable there: both are derived from the CR's name and namespace, so anyone holding patch on a namespace of that cluster can write them, while the UID is minted by the management cluster's API server. A mark naming a different ControlPlane disowns the namespace — and the condition message names the recorded UID, since such a namespace was created by a ControlPlane of this name, on a cluster several management clusters can register. A missing mark refuses adoption too, since nothing on the object separates a stripped mark from labels somebody else wrote; that message names both remedies, restoring the annotation or picking a free name. The read behind the verdict goes through the target cluster's live reader, as the teardown side does — which draws the missing-mark line one notch lower, because it is the last pass anything makes over that namespace. Under External the operator only verifies the namespace exists; a missing one parks the condition and requeues.

A namespace whose services carry a targetClusterRef is ensured on that cluster as well as on the management cluster, under whichever lifecycle it declares. Both copies are needed: the workload CR stays at home, in the namespace the service is assigned to, and the service operator that picks it up writes its own children into the namespace of the same name on the target. One reason vocabulary covers both sides, so a namespace missing on the target reports NamespaceNotFound as one missing at home does. The cluster is resolved per namespace before anything is written, and a name that does not resolve creates nothing on either side.

Status	Reason	When
True	NoDedicatedNamespaces	No service declares a namespace of its own.
True	NamespacesReady	Every declared service namespace is present (and, for Managed, owned).
False	NamespaceNotFound	An External namespace does not exist; requeue.
False	NamespaceNotOwned	A Managed namespace exists but does not carry the operator's ownership labels, or — on a target cluster — its c5c3.io/controlplane-uid annotation is missing or names a different ControlPlane, so it is never adopted. The message distinguishes the three, because the remedies differ.
False	NamespaceTerminating	The namespace is being deleted; wait and requeue.
False	TargetClusterUnavailable	A service placed the namespace on a target cluster that does not resolve; the resolver's own message, cluster not found for a name that was never registered. On deletion the same reason marks a placed namespace whose cluster has not answered yet: the teardown waits for it, and gives up on its children only past the abandon window (see Owner-ref / GC model).
False	FinalizingNamespaces	On deletion, waiting for cross-namespace children to be torn down (see Owner-ref / GC model).
False	NamespaceError	A create/get against the namespace failed.

reconcileInfrastructure

Aspect	Value
File	reconcile_infrastructure.go
Condition	InfrastructureReady
Gate	none
Projects / Owns	Managed-mode MariaDB (k8s.mariadb.com) and Memcached (unstructured memcached.c5c3.io/v1beta1) children, each named after its clusterRef and created in the namespace of the service that resolves to it (cp.KeystoneNamespace() / cp.HorizonNamespace(), the ControlPlane's own unless a namespace assignment places the service elsewhere), on the cluster that namespace lives on; plus the managed-mode RabbitmqCluster (unstructured rabbitmq.com/v1beta1), always in the ControlPlane's own namespace on the local cluster
Requeue	infraRequeueAfter = 15s while a managed child is not yet Ready

Backing services follow the service. managedInfraInstances adds each service's effective database and cache at that service's namespace and deduplicates on (kind, namespace, name), so the one shared spec.infrastructure block materializes once per namespace that consumes it — two instances when Keystone and Horizon are placed apart, one when they are co-located, exactly one (today's behavior) when neither is assigned a namespace. A child in a service namespace carries no owner reference (Kubernetes forbids a cross-namespace one) — it is stamped with the ownership labels and cleaned up by the finalizer instead; a same-namespace child keeps its controller owner reference. The dashboard's cache is enumerated only when the dashboard is declared, so a ControlPlane that places Keystone apart never provisions a phantom cache for an absent Horizon.

Sizing. reconcileInfrastructure resolves the sizing (effectiveSizing) and passes it to managedInfraInstances; every managed instance of a class takes that class's block, with the top-level spec.sizing placement folded in for the database and the bus. ensureMariaDB takes an unset replicas / storageSize from spec.sizing.database (the webhook normally stored them at admission) and writes spec.resources, spec.nodeSelector, spec.tolerations and spec.priorityClassName; an owned MariaDB is re-asserted with one Update when any of them differs (updating owned MariaDB "<name>" on failure), and an adopted one is never written. ensureMemcached takes an unset replicas from spec.sizing.cache on every pass, floors a zero count to 3, and writes or removes spec.resources; the Memcached CRD has no placement. ensureRabbitMQ takes an unset replicas from spec.sizing.messaging, writes spec.resources and spec.tolerations, and writes the node selector and priority class through spec.override.statefulSet.spec.template.spec (containers: [] included, which the upstream schema requires); unset resources that equal the operator's default count as matching, so a Standard bus is not rewritten on every pass. The shrink gate on c5c3.io/allow-messaging-recreate is unchanged. The teardown enumerates the instances with the zero sizing, since it only reads their names.

The message bus stays home. addMessaging is called once, at childNamespace(cp) with the declared-at path spec.infrastructure.messaging, and it is called whether or not a service consumes the bus. That makes messaging the single class enumerated at the ControlPlane's own namespace regardless of consumers: a bus is shared across services by nature, so declaring the block is what asks for the broker. Brownfield messaging (secretRef) enumerates nothing, the same way a brownfield database does. Consumers reach the managed bus at <clusterRef.name>.<cp.Namespace>.svc. A service placed on a target cluster cannot reach it from there; that path is open until the first consumer settles it (issue #906).

A service that names a targetClusterRef takes its backing services with it: the instance is created on that cluster, where the mariadb-operator that acts on the CR runs and where the service's own pods connect to it. The cluster of every enumerated instance is resolved before the first one is written, so an unresolvable name leaves the whole set unprovisioned rather than the database on one cluster and the cache nowhere.

reconcileInfrastructure provisions the backing services the ControlPlane owns. That set is the instances its services actually resolve to, not the set of declared blocks: the shared instances in spec.infrastructure, and the per-service dedicated instances under services.<svc>.dedicatedBackingServices (see DedicatedBackingServices) that a service opted into instead. managedInfraInstances enumerates them by walking the effective-instance resolvers per service and deduplicating on the identity of the child CR they resolve to — so several services on one shared instance ensure it exactly once, and a shared instance every service has opted out of has no consumer and is not provisioned at all. The defaulting webhook materializes spec.infrastructure.database whenever it is omitted, so once every declared database consumer — Keystone, Glance, Placement, Barbican — has taken a dedicated instance, provisioning the declared set instead would leave a full Galera cluster nothing talks to — with InfrastructureReady blocked on it coming up. A backing service is managed when its clusterRef is set and brownfield (provisions nothing) when host/servers are set instead.

Every managed child — shared or dedicated — is ensured in a single pass before readiness is gated, so a half-provisioned control plane (DB created but cache missing) never occurs; readiness is then evaluated collectively across the whole set. A service whose dedicated database is still converging therefore holds InfrastructureReady False even when every other instance is Ready, so that service's projection stays gated on the database it actually talks to. The condition message names the pending instance and the spec path it was declared at; the reasons stay per-class (WaitingForDatabase / WaitingForCache), so the reason vocabulary is unchanged by the dedicated opt-in.

ensureMariaDB / ensureMemcached take the declared instance rather than reading spec.infrastructure directly, which is what makes a dedicated instance carry the shared block's lifecycle rather than a parallel one of its own. It is created owned, in one of two forms. In the ControlPlane's own namespace it carries a controller owner reference, so it is garbage-collected with the ControlPlane. In a service namespace or on a target cluster it carries the two ownership labels c5c3.io/controlplane-name and c5c3.io/controlplane-namespace, and the finalizer-driven teardown deletes it (see Owner-ref / GC model). It is sized from itsreplicas / storageSize and re-projected on drift while it is owned in either form (isControlPlaneChild). A CR under that name that carries neither is adopted read-only: never reshaped, never GC-claimed.

ensureRabbitMQ is the twin of ensureMemcached: read-modify-write on an *unstructured.Unstructured carrying rabbitmqClusterGVK. On NotFound it creates the CR with spec.replicas and a controller owner reference (claimChildOwnership); on a CR it already owns it re-projects spec.replicas and nothing else; a CR of that name owned by someone else is adopted read-only. Its ownership test is the controller owner reference alone (IsControlledBy), because the bus is only provisioned in the ControlPlane's own namespace on the management cluster. The re-projection is asymmetric: growing an owned cluster is an in-place Update, while shrinking it is a delete-and-recreate — the RabbitMQ Cluster Operator refuses an in-place scale-down, so a lowered count written onto the CR would sit there ignored while AllReplicasReady (and with it InfrastructureReady) stayed True. The delete reports the bus not ready, so the next pass takes the NotFound branch and creates the cluster at the declared size. It is destructive by construction — the broker and its volumes come back empty, and with them every durable queue and unacked message — so it is gated on an annotation: without c5c3.io/allow-messaging-recreate: "true" on the ControlPlane the shrink is refused, the broker keeps running at its current size, and the divergence surfaces as InfrastructureReady=False / RabbitMQError with a message naming the annotation. The gate exists because replicas carries a schema default of 3: a commit that merely drops the line off a ControlPlane running five pods reads as a no-op in review and arrives at the reconciler as a scale-down nobody typed. With the annotation set, the recreate is the only path the operator offers for a declared count the running cluster exceeds.

On ControlPlane deletion the bus is not left to the owner-reference cascade: the finalizer deletes it with foreground propagation and waits for it to go, for the cluster-operator race described under Owner-ref / GC model.

A child that is mid-teardown is never reported ready. The RabbitMQ Cluster Operator holds a finalizer on its CRs, so a deleted RabbitmqCluster lingers in Terminating with its spec and its status.conditions intact — AllReplicasReady still reads True on a broker whose StatefulSet and Secrets are going away. A non-nil metadata.deletionTimestamp therefore short-circuits the pass to not-ready before the replica comparison, which would otherwise fall through to the condition read whenever the declared count happens to match the dying cluster's.

Image, resources, persistence, and spec.tls stay at the RabbitMQ Cluster Operator's defaults, or at whatever the platform set on the adopted CR, the posture ensureMariaDB takes on TLS and issuer refs. A bypassed replicas: 0 is floored to infraRabbitMQReplicasDefault (3) so the broker never comes up with no pods.

Its readiness reads AllReplicasReady through unstructuredConditionTrue(u, "AllReplicasReady"). The RabbitMQ Cluster Operator sets no Ready condition at all, so the Ready specialisation would never see a healthy broker. A cluster that does not serve the rabbitmq.com CRD answers the Get with a meta.NoKindMatchError, which apierrors.IsNotFound does not match, so a ControlPlane that declares managed messaging there fails closed with InfrastructureReady=False / RabbitMQError instead of quietly provisioning nothing.

spec.infrastructure is optional: an External-mode Keystone ControlPlane omits it (the validating webhook forbids it in External mode and requires it otherwise). In External mode the sub-reconciler provisions nothing and reports InfrastructureReady=True / ExternallyManaged immediately. A non-External ControlPlane that nevertheless reaches this point with the block unset is a webhook-bypass shape (direct etcd write, admission misconfigured); it fails closed with InfrastructureNotConfigured rather than dereferencing the nil pointer.

Path	Status	Reason	Notes
External keystone mode	True	ExternallyManaged	no MariaDB/Memcached is provisioned; the message names external.authURL
MariaDB create/update fails	False	MariaDBError	returns the error (controller-runtime backoff)
Memcached create/update fails	False	MemcachedError	returns the error
MariaDB not yet Ready	False	WaitingForDatabase	requeue 15s
Memcached not yet Ready	False	WaitingForCache	requeue 15s
RabbitmqCluster create/update fails	False	RabbitMQError	returns the error; a cluster that does not serve rabbitmqclusters.rabbitmq.com lands here with the RESTMapper's no matches for kind
Declared messaging.replicas below the owned cluster's, c5c3.io/allow-messaging-recreate unset	False	RabbitMQError	returns the error; the destructive delete-and-recreate is refused and the broker keeps running at its current size
RabbitmqCluster not yet AllReplicasReady	False	WaitingForMessaging	requeue 15s
spec.infrastructure unset, not External	False	InfrastructureNotConfigured	requeue 15s; unreachable on the admission path — fails closed for a webhook-bypassed CR
A service placed an instance on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 15s; the resolver's own message, cluster not found for a name that was never registered. Nothing is provisioned, on either cluster
All managed children Ready (or pure brownfield)	True	InfrastructureReady	—

The managed MariaDB child is provisioned with a minimal-but-valid spec — replicas: 3, galera.enabled: true, storage.size: 100Gi (infraMariaDBReplicas / infraMariaDBStorageSize) — mirroring the production baseline; the mariadb-operator webhook rejects a CR without a storage size. Both values come from the declared instance, so a dedicated database can be sized (and, with replicas: 1, taken off Galera) independently of the shared cluster. The Memcached child's spec.replicas is likewise taken from the declared instance's cache.replicas (widened to int64 for unstructured nested-field storage). MariaDB readiness is read via conditions.IsReady(mariadb.Status.Conditions); Memcached readiness is read from the unstructured status.conditions[type=Ready].status == "True" (unstructuredReady), where a missing/malformed list is treated as not-ready rather than an error. unstructuredReady is the Ready specialisation of unstructuredConditionTrue, which takes the condition type as an argument; the RabbitmqCluster child is read with AllReplicasReady through the same helper, and its spec.replicas comes from messaging.replicas.

reconcileESOTenantStore

Aspect	Value
File	reconcile_esotenant.go
Condition	ESOTenantStoreReady
Gate	none — runs after Infrastructure and before every store-consuming sub-reconciler so the per-tenant store exists before they gate on it
Projects / Owns	when spec.secretStoreRef is omitted: a ServiceAccount (eso-tenant-auth), a cert-manager mTLS Certificate (eso-tenant-client-tls), and a namespaced SecretStore (openbao-tenant-store), one trio per namespace the ControlPlane occupies, each on the cluster that namespace lives on; when spec.secretStoreRef is set the sub-reconciler provisions nothing
Requeue	esoTenantStoreRequeueAfter = 10s while the SecretStore is not yet Ready

reconcileESOTenantStore provisions the in-cluster half of the ControlPlane's per-tenant OpenBao identity and makes it the enforced default: every ControlPlane that omits spec.secretStoreRef routes its (and its children's) secret traffic through the per-tenant openbao-tenant-store, so OpenBao's templated eso-tenant policy — not a naming convention — isolates one control plane's key material from another. The OpenBao server and Kubernetes-auth mount are read from the shared cluster store (the per-tenant store cannot describe its own bootstrap). An explicit spec.secretStoreRef is an override: the sub-reconciler provisions nothing and reports ESOTenantStoreReady=True with reason StoreRefOverridden, and the store-consuming sub-reconcilers gate on the selected store's own readiness.

An ESO store and the Secrets it materialises are namespace-local and cluster-local, so each namespace the ControlPlane occupies gets its own trio, written to the cluster that namespace lives on and read back from there for the readiness gate. The condition is True only once every store is Ready, and it names the namespace of the first one that is not.

Scenario	Status	Reason
spec.secretStoreRef set (override)	True	StoreRefOverridden
every per-tenant SecretStore Ready	True	ESOTenantStoreReady
a per-tenant SecretStore not yet Ready	False	SecretStoreNotReady
a namespace's target cluster does not resolve	False	TargetClusterUnavailable
provisioning the objects failed	False	ProvisioningError

reconcileDBCredentials

Aspect	Value
File	reconcile_dbcredentials.go
Condition	DBCredentialsReady
Gate	none — runs unconditionally, positioned after Infrastructure and before Keystone so the Keystone CR is never projected before the DB-credential Secret exists
Projects / Owns	Managed-mode (spec.infrastructure.database.clusterRef != nil), effective Dynamic unless credentialsMode: Static: a VaultDynamicSecret generator, ServiceAccount (keystone-db-creds), mTLS client Certificate, and an ExternalSecret named {controlplane.Name}-keystone-db-credentials (dbCredentialSecretName) drawing from the generator, all in cp.KeystoneNamespace() and on the cluster that namespace lives on; brownfield projects nothing
Requeue	dbCredentialsRequeueAfter = 10s while the ExternalSecret is not yet Ready

reconcileDBCredentials projects the per-ControlPlane service database credential so the projected Keystone CR consumes a DB credential scoped to its own ControlPlane. It mirrors reconcileAdminCredential's wait/condition handling.

Every decision below is made on the effective database (effectiveKeystoneDatabase) — the Keystone service's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise — so the credential follows the instance the service actually connects to. A dedicated managed database always takes the Static branch: the OpenBao database engine carries one connection and one role per namespace, bootstrapped against the shared cluster, so no engine role can issue credentials for a dedicated instance. The validating webhook rejects an explicit credentialsMode: Dynamic there, and keying the reconciler's decision on the dedicated declaration (not only on the stored mode) makes a webhook-bypassed CR fail closed onto Static rather than project a generator that could never sync.

The database is managed when the effective clusterRef is set and brownfield when the user supplies a host-based connection:

• Brownfield is a pure no-op. When clusterRef == nil the user owns the DB credential Secret out-of-band, so the operator projects no ExternalSecret and never references OpenBao or the selected secret store; DBCredentialsReady is reported True immediately so the chain proceeds to Keystone.

• Managed defaults to Dynamic (engine-issued). After gating (via secrets.IsStoreRefReady) on the ControlPlane's store, the operator projects (all owner-referenced): a keystone-db-creds ServiceAccount, an mTLS client Certificate from the cluster-scoped openbao-ca-issuer, a generators.external-secrets.io/v1alpha1 VaultDynamicSecret reading database/mariadb/creds/keystone-{cp.Namespace} (dbDynamicCredsPathFor, keyed on the namespace alone), and an ExternalSecret (RefreshInterval 24h, Target.CreationPolicy: Owner) drawing from that generator via dataFrom.sourceRef.generatorRef — no static Data refs and noSecretStoreRef. The generator's OpenBao server URL and Kubernetes-auth mount are copied from the selected store's Vault provider by openBaoConnectionFor (falling back to the documented defaults when unreadable), so the generator cannot drift from the store the rest of the stack uses. All Secret references are same-namespace (the generator is Namespaced), satisfying the OpenBao listener's require-and-verify-client-cert gate. The materialised Secret carries an engine-issued username and password with a finite lease, so no long-lived static DB password remains at rest.

• Managed Static is the opt-out — and the only mode a dedicated managed database has. The operator projects the stage-(a) KV-backed ExternalSecret (SecretStoreRef the ControlPlane's store, built via secrets.ESOSecretStoreRef, with username/password Data reading openstack/keystone/{cp.Namespace}/{cp.Name}/db) and tears down any leftover dynamic-mode objects.

  That KV path is seeded by neither the operator nor the bootstrap. The per-ControlPlane static seed was retired when managed mode moved to engine-issued credentials, so a Static ControlPlane — the explicit opt-out on the shared database, and every dedicated managed database — reaches Ready only once the path has been seeded (username, password) out-of-band; see Migrate the Keystone DB to dynamic credentials. Until then DBCredentialsReady stays False with reason WaitingForDBCredentialSecret, and the message names the exact path to seed.

reconcileKeystone projects the effective mode onto the Keystone CR's spec.database.credentialsMode, so the Keystone operator consumes the matching credential shape.

The credential material follows the Keystone service onto its target cluster, because the ESO that runs the generator and the cert-manager that issues its client certificate are the ones on that cluster, and the Secret they produce has to be mountable by the Keystone pods. The gates in front of it (the store's readiness and the ExternalSecret's) are read from the same cluster. ensureServiceDBCredential, the shared helper behind the Glance, Placement, and Barbican credentials, does the same for their namespaces and reports an unresolvable cluster on the calling service's own condition rather than on DBCredentialsReady.

In External keystone mode the ControlPlane manages no database at all — neither a managed one to issue credentials for, nor a brownfield connection to reference — so neither OpenBao nor any secret store is consulted.

Path	Status	Reason	Notes
External keystone mode	True	ExternallyManaged	no database is managed; nothing is projected and no secret store is read
Brownfield (clusterRef == nil)	True	BrownfieldUserSuppliedCredential	no ExternalSecret projected; user supplies the DB credential Secret out-of-band
Keystone placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s; resolved before the store gate, so nothing is projected and no store is read. The resolver's own message
Selected secret store not Ready	False	SecretStoreNotReady	requeue 10s; managed mode only, checked before projection so an OpenBao/ESO outage surfaces promptly. The message names the store's kind and name
Dynamic generator/SA/Certificate/ExternalSecret create/update fails	False	GeneratorError	returns the error
Static ExternalSecret create/update fails	False	ExternalSecretError	returns the error
ExternalSecret create/update or read fails	False	ExternalSecretError	returns the error
ExternalSecret not yet synced	False	WaitingForDBCredentialSecret	requeue 10s
ExternalSecret Ready	True	DBCredentialsReady	—

reconcileAdminPassword

Aspect	Value
File	reconcile_adminpassword.go
Condition	AdminPasswordReady
Gate	none — runs unconditionally, positioned after DBCredentials and before Keystone so the Keystone CR is never projected before the admin-password Secret exists
Projects / Owns	Managed-mode (spec.infrastructure.database.clusterRef != nil) one external-secrets.io/v1 ExternalSecret named {controlplane.Name}-keystone-admin-credentials (adminPasswordSecretName) in cp.KeystoneNamespace(), on the cluster that namespace lives on; brownfield projects nothing
Requeue	adminPasswordRequeueAfter = 10s while the ExternalSecret is not yet Ready

reconcileAdminPassword projects the per-ControlPlane Keystone admin password as an OpenBao-backed ExternalSecret, so the projected Keystone CR's bootstrap admin-password ref consumes a credential scoped to its own ControlPlane. It mirrors reconcileDBCredentials's wait/condition handling. The database is managed when spec.infrastructure.database.clusterRef is set and brownfield when the user supplies a host-based connection:

• Brownfield is a pure no-op. When clusterRef == nil the user owns the admin-password Secret out-of-band, so the operator projects no ExternalSecret and never references OpenBao or the selected secret store; AdminPasswordReady is reported True immediately so the chain proceeds to Keystone.
• Managed projects the ExternalSecret. The owned ExternalSecret has RefreshInterval 1h, its SecretStoreRef built from the ControlPlane's spec.secretStoreRef via secrets.ESOSecretStoreRef (default: the ControlPlane's store), and Target.CreationPolicy: Owner (so ESO owns the materialised Secret of the same name). Its single password Data key reads from the per-CP remote key bootstrap/{cp.Namespace}/{cp.Name}-keystone/admin (adminPasswordRemoteKeyFor) with Property: password. Unlike the DB-credential path this key is Keystone-name-scoped ({cp.Name}-keystone, not {cp.Name}) so it matches the bootstrap seeder and the keystone-operator's Model-B rotation PushSecret at bootstrap/{keystone.Namespace}/{keystone.Name}/admin; the {namespace}/{keystone-name} scoping still keeps two ControlPlanes from colliding on the cluster-global OpenBao backend. The builder adminPasswordExternalSecret(cp) sets no owner reference; the reconciler applies the ExternalSecret via Server-Side Apply under the shared field manager (cobaltcore-operator), which stamps the ControlPlane controller reference for GC. In a dedicated Keystone namespace, and on a target cluster, no owner reference is possible: the ExternalSecret carries the ControlPlane's ownership labels instead and the finalizer-driven teardown deletes it. A placed Keystone takes the ExternalSecret with it, since only the ESO on that cluster can sync it into a Secret the Keystone pods can mount.

The managed-mode effective admin-password ref (effectiveAdminPasswordSecretRef) points the projected Keystone child's spec.bootstrap.adminPasswordSecretRef at this materialised Secret's password key ({controlplane.Name}-keystone-admin-credentials); in brownfield mode it stays the user-declared spec.korc.adminCredential.passwordSecretRef verbatim. The cp-level spec default for passwordSecretRef remains keystone-admin.

effectiveAdminPasswordSecretRef keys its External branch on spec.services.keystone.mode, not on the database shape, and returns spec.korc.adminCredential.passwordSecretRef verbatim. That Secret is the admin password source: the operator only ever reads it, because the external Keystone's admin password is owned out-of-band. Its SHA-256 feeds adminPasswordHashAnnotation, so updating that Secret is what drives a hash-driven re-mint of the admin application credential — the only supported rotation path in External mode. The field indexer (controlPlaneSecretNameExtractor) follows the same ref, so an edit to the user's Secret wakes the ControlPlane immediately.

Path	Status	Reason	Notes
External keystone mode	True	ExternallyManaged	the admin password is read from the user-supplied passwordSecretRef Secret; no ExternalSecret is projected and no OpenBao bootstrap path is seeded
Brownfield (clusterRef == nil)	True	BrownfieldUserSuppliedCredential	no ExternalSecret projected; user supplies the admin-password Secret out-of-band
Keystone placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s; resolved before the store gate, so no ExternalSecret is projected and no store is read. The resolver's own message
Selected secret store not Ready	False	SecretStoreNotReady	requeue 10s; managed mode only, checked before the ExternalSecret is projected. The message names the store's kind and name
ExternalSecret create/update or read fails	False	ExternalSecretError	returns the error
ExternalSecret not yet synced	False	WaitingForAdminPasswordSecret	requeue 10s
ExternalSecret Ready	True	AdminPasswordReady	—

Identity-backend watch

KeystoneIdentityBackend CRs are authored by the operator, not projected by the ControlPlane, so they carry no ControlPlane owner reference an Owns() could match. SetupWithManager therefore registers a Watches(&KeystoneIdentityBackend{}, ...) bound to identityBackendToControlPlaneMapper, which enqueues the ControlPlane whose keystoneName(cp) equals the backend's keystoneRef.name. A backend attached to a hand-rolled Keystone beside a ControlPlane therefore never wakes it.

listIdentityBackends resolves a ControlPlane's backends with a cache-backed List in childNamespace(cp) and the same keystoneRef.name filter applied in memory — the set holds one backend per identity provider plus one per LDAP domain. Backends carrying a deletionTimestamp are dropped: a backend's own reconcileDelete never demotes Ready while it waits for de-projection, so a Terminating backend would otherwise keep offering an SSO choice whose Keystone-side federation objects are being torn down — and would collide with the same-named replacement its webhook admits during teardown.

RBAC is read-only (get;list;watch) in both the kubebuilder marker and the shared Helm rules helper: the reconciler never writes a backend.

Attaching, detaching, or a backend reaching Ready re-projects the Horizon websso choices immediately, without waiting for a periodic resync. The Keystone trusted_dashboard does not depend on the backends.

reconcileKeystone

Aspect	Value
File	reconcile_keystone.go
Condition	KeystoneReady
Gate	InfrastructureReady == True
Projects / Owns	one Keystone child named {controlplane.Name}-keystone (keystoneNameSuffix) in childNamespace(cp)
Requeue	keystoneInfraGateRequeueAfter = 5s while gated; infraRequeueAfter = 15s while the child is not Ready

reconcileKeystone projects spec.services.keystone into an owned Keystone CR. The projection is deliberately thin — it reuses the ControlPlane's own infrastructure specs verbatim so Keystone points at the same backing services the ControlPlane provisioned:

• Image: repository defaults to ghcr.io/c5c3/keystone with the tag derived from spec.openStackRelease; spec.services.keystone.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database / Cache: keystone.Spec.Database and keystone.Spec.Cache are DeepCopies of the effective instances — the service's dedicated database/cache when it opted into one (effectiveKeystoneDatabase / effectiveKeystoneCache), the shared spec.infrastructure instance otherwise (the default). Projecting the effective spec is what carries the opt-in through the rest of the chain with no per-class special-casing: the keystone-operator derives its logical database, its MariaDB User/Grant CRs, and its NetworkPolicy database/cache egress rules from spec.database / spec.cache, so all of them follow the instance the service actually talks to.
• Bootstrap: the admin-password Secret ref is the effective ref (effectiveAdminPasswordSecretRef) — in managed mode the operator-projected per-CP Secret {controlplane.Name}-keystone-admin-credentials (see reconcileAdminPassword), in brownfield mode the user-declared cp.Spec.KORC.AdminCredential.PasswordSecretRef verbatim (so Keystone and K-ORC agree on the admin-password source) — and the region is cp.Spec.Region.
• Sizing: spec.sizing.keystone (resolved) projects onto spec.deployment (replicas, falling back to commonv1.DefaultReplicas; resources; node selector, tolerations and priority class with the top-level fallback; spread completed with keystonev1alpha1.APIPodSelector), spec.uwsgi, spec.autoscaling, spec.jobs, and spec.federation.proxyResources. Every field is assigned on every pass.
• Federation: spec.federation.proxyImage is the spec.services.keystone.federationProxyImage override when set, else ghcr.io/c5c3/keystone-federation-proxy:latest, either with spec.imagePullPolicy when it names no pullPolicy of its own; spec.federation.trustedDashboards is the ControlPlane's own dashboard origin (cp.Spec.Services.Horizon.DerivedPublicEndpoint() + "/auth/websso/"), or nil when no dashboard is externally reachable. Both are assigned unconditionally, so clearing the override or the horizon block reverts the child. The origin is derived top-down from cp.Spec, never from the Horizon child's status, so this projection carries no ordering dependency on reconcileHorizon (which is gated on KeystoneReady and therefore runs strictly after it). Both fields are inert until a federation backend attaches.
• Policy: policy.MergePolicies(cp.Spec.GlobalPolicyOverrides, cp.Spec.Services.Keystone.PolicyOverrides) (the shared internal/common/policy helper) merges the global base with per-service overrides (per-service wins on conflict).
• Rotation: when spec.services.keystone.rotationInterval is set, intervalToCron converts it to a cron schedule applied to bothFernet.RotationSchedule and CredentialKeys.RotationSchedule. Only 168h (weekly, 0 0 * * 0) and positive whole-day multiples (daily, 0 0 * * *) are supported.
• Target cluster: spec.targetClusterRef is a DeepCopy of services.keystone.targetClusterRef, and a nil source projects no field at all. The Keystone CR itself is applied through the local client whichever cluster it names, so that one field is the whole hand-over: the keystone-operator reads the CR on the management cluster and owns everything it creates on the target. The same holds for the Horizon, Glance, Placement, and Barbican children. The value is frozen on both sides — the ControlPlane webhook rejects an edit, and each child CRD carries its own CEL transition rule — so re-projecting it never trips the child's freeze.

In External keystone mode no child is projected — and none is deleted. A Managed -> External flip is rejected at admission (adopting an existing installation must be a fresh External-mode ControlPlane), so no child can exist; were one to appear anyway, the fail-safe that preserves a child unless c5c3.io/allow-keystone-deletion: "true" opts in is the only sanctioned teardown path, because the child's credential/fernet keys are irreplaceable.

Path	Status	Reason	Notes
spec.services.keystone unset	True	KeystoneNotManaged	no identity plane at all; a previously-projected child is preserved by default
External keystone mode	True	ExternallyManaged	identity is managed against external.authURL; no child is projected and none is deleted
InfrastructureReady not True	False	WaitingForInfrastructure	requeue 5s; no Keystone CR is created while infra is unready
Invalid rotationInterval	False	InvalidRotationInterval	returns the error so the reconcile chain stops at Keystone and the manager requeues with backoff (the validating webhook already rejects unrepresentable intervals at admission, so this is defense-in-depth)
Keystone create/update fails	False	KeystoneError	returns the error
Keystone child not yet Ready	False	WaitingForKeystone	requeue 15s
Keystone child Ready	True	KeystoneReady	—

reconcileHorizon

Aspect	Value
File	reconcile_horizon.go
Condition	HorizonReady
Gate	KeystoneReady == True
Projects / Owns	one Horizon child named {controlplane.Name}-horizon (horizonNameSuffix) in childNamespace(cp) — only when spec.services.horizon is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated; infraRequeueAfter = 15s while the child is not Ready

reconcileHorizon is optional: spec.services.horizon unset means this ControlPlane manages no dashboard, and the sub-reconciler reports HorizonReady=True / HorizonNotManaged so the aggregate is not blocked (staged adoption). A previously-projected child is preserved unless the ControlPlane opts in with c5c3.io/allow-horizon-deletion: "true" (then the orphan is deleted) — the same annotation UX as Keystone, though the dashboard is stateless.

When managed, the projection mirrors the Keystone one's thin discipline, reusing the ControlPlane's own specs so the dashboard points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/horizon with the tag derived from spec.openStackRelease; spec.services.horizon.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Cache: a DeepCopy of the effective cache (effectiveHorizonCache) — the dashboard's dedicated cache when it opted into one, the shared spec.infrastructure.cache otherwise — with the same clusterRef / servers / replicas, except that Backend is overridden to the Horizon Django default django.core.cache.backends.memcached.PyMemcacheCache. The shared CacheSpec.Backend carries the oslo.cache dogpile path Keystone consumes (dogpile.cache.pymemcache), which Django renders verbatim as a CACHES backend and rejects with InvalidCacheBackendError, so the dashboard would never go Ready — only the endpoint-bearing fields are reused unchanged.
• Keystone endpoint: derived top-down via horizonKeystoneEndpoint(cp) from the Keystone child's naming convention, not read from the Keystone child's status (no machine consumer reads status endpoints, per the settled convention). The cluster-local Service URL (the same URL K-ORC authenticates against) as long as the dashboard and Keystone resolve to the same cluster — never the external publicEndpoint or gateway hostname, which the dashboard pods may not be able to reach: the dashboard's Django backend connects to this URL server-side, the browser never does. A dashboard on another cluster than Keystone cannot resolve that Service DNS name at all and gets keystonePublicEndpoint instead (see Reaching a placed service).
• SecretKeyRef: defaults to the kind shim Secret horizon-secret-key (key secret-key), which is pinned to the default ControlPlane identity; spec.services.horizon.secretKeyRef overrides it, and a second ControlPlane must set its own so each dashboard reads distinct SECRET_KEY material.
• Gateway: a DeepCopy of spec.services.horizon.gateway; a nil source clears the projected gateway so removing the block tears the HTTPRoute down.
• Sizing: spec.sizing.horizon.api projects onto spec.deployment (replicas falling back to commonv1.DefaultReplicas, resources, placement, spread) and spec.autoscaling, inside the CreateOrUpdate closure. Every field is assigned unconditionally, so clearing a value reverts the fetched child instead of pinning a lost update.
• WebSSO: projected from the Ready OIDC KeystoneIdentityBackend CRs attached to the Keystone child (see Identity-backend watch). One choice per Ready backend, keyed {identityProvider}_{protocol} (truncated to a digest-suffixed 64 characters when the two names together exceed the Horizon CRD's bound on choices[].id), with the local-credentials fallback leading the list and preselected; keystoneURL is keystonePublicEndpoint(cp) — the browser-facing endpoint, because the browser follows the SSO redirect. At most 16 federated choices are projected (maxProjectedFederationChoices); the excess is dropped and logged rather than rejected by the API server as a choices/idpMapping overflow that would wedge every later Horizon change. nil when no OIDC backend is attached, so a choice never appears for a backend whose federation objects are not provisioned yet — and nil too when the hand-off could not complete anyway: no trusted dashboard origin (trustedDashboards(cp)) means Keystone bounces the browser after the user has entered their corporate credentials, and no keystonePublicEndpoint(cp) means the redirect targets a cluster-local DNS name the browser cannot resolve. Both are logged.
• MultiDomain: enabled with defaultDomain: Default once any LDAP backend is Ready, so the login form gains a domain field. domainChoices / domainDropdown are deliberately not projected: upstream openstack_auth turns the domain field into a select bounded by OPENSTACK_KEYSTONE_DOMAIN_CHOICES, and the operator only ever sees the LDAP-backed domains — a dropdown built from them would lock out every user of a domain it cannot enumerate (a SQL-backed domain populated out-of-band, or the domain an OIDC backend targets). nil when no LDAP backend is attached.
• Detached vs. unhealthy. Detaching the last backend of a type clears its block, so the login page reverts to local credentials. A backend that is attached but not Ready retains the previously-projected block instead (projectWebSSO / projectMultiDomain): a backend's aggregate Ready can drop on a failed observation while the Keystone-side federation objects it provisioned are untouched, so the SSO button keeps working. Rebuilding the block from that view would re-render local_settings.py, roll the dashboard Deployment, and roll it back on recovery — twice, for a login page that was never broken. The retention is logged.

A failure to list the backends surfaces as HorizonReady=False with reason IdentityBackendsUnavailable and returns the error — joined into the tail group's error, with controller-runtime's backoff driving the retry — never an empty websso block, which would silently remove a working SSO button.

Path	Status	Reason	Notes
spec.services.horizon unset	True	HorizonNotManaged	staged adoption — does not block the aggregate; a previously-projected child is preserved unless c5c3.io/allow-horizon-deletion: "true" is set
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Horizon CR is projected while Keystone is unready
Horizon create/update fails	False	HorizonError	returns the error
Projected spec rejected (HTTP 422 Invalid)	False	HorizonProjectionRejected	returns the error; the projection violates a Horizon CRD/webhook rule — reconcile the ControlPlane spec to a valid projection to recover
Horizon child not yet Ready	False	WaitingForHorizon	requeue 15s
Horizon child Ready	True	HorizonReady	—

Built-in service registrations

Glance, Placement, Barbican, Neutron, Cinder and Nova each need a Keystone catalog row and a Keystone service user. None of the six registers either itself: every one of them projects a KeystoneService child and lets that CR's controller do the work. builtin_registrations.go is the leg all six share, so it is described once here rather than six times below, and a seventh built-in service adds the values it registers with rather than another copy of the leg.

Aspect	Value
File	builtin_registrations.go
Conditions	GlanceReady, PlacementReady, BarbicanReady, NeutronReady, CinderReady, NovaReady — the leg writes the caller's condition, never one of its own
Projects / Owns	one KeystoneService named {controlplane.Name}-{service} in the namespace that service is placed in, applied through the local client whatever cluster the service runs on
Requeue	korcRequeueAfter = 10s while the registration is not usable yet

What the child asserts. spec.controlPlaneRef names the ControlPlane's namespace explicitly rather than relying on the default, so the child resolves the same plane wherever it is placed. spec.catalog carries the service's type and name with two endpoints, internal and public, from the start, so no later catalog migration is needed. spec.account carries the service's user with a project of its own and create: true, plus the roles the caller passes: every service needs service, the role the OpenStack policy files reserve for service-to-service calls, and Cinder and Nova name admin on top. Each service creates its own project (service-placement, service-barbican, service-neutron, service-cinder, service-nova), because two registrations creating one project would each adopt the other's Keystone row.

Two registrations reach past service. Cinder deletes the Barbican secret of an encrypted volume as a fallback when the volume's owner cannot (decision D9 of #979), and under Barbican's secure-RBAC defaults the secret:get and secret:delete rules a bare admin holds are what carry that reach across projects. Nova calls the block-storage API through its [cinder] service user in admin contexts, where it carries no user token of its own, and cinder refuses a caller holding service alone on a volume that belongs to a user (decision D10 of #1019). builtinRegistration therefore takes the role list as a parameter instead of pinning service as a constant.

The URLs each row advertises. The internal endpoint advertises the in-cluster API Service URL — http://{controlplane.Name}-glance.<glance-namespace>.svc:9292 (glanceEndpointURL), …-placement.<placement-namespace>.svc:8778 (placementEndpointURL), …-barbican.<barbican-namespace>.svc:9311 (barbicanEndpointURL), …-neutron.<neutron-namespace>.svc:9696 (neutronEndpointURL), …-cinder.<cinder-namespace>.svc:8776 (cinderEndpointURL), …-nova.<nova-namespace>.svc:8774 (novaEndpointURL). The first four carry no path suffix; unlike identity, those APIs are served at the root. Block storage and compute do carry one. The two block-storage rows take /v3, the microversioned prefix the API is served under, project-less because cinder resolves the project from the token rather than from the URL (decision D8 of #979); the two compute rows take /v2.1, the compute API's own version prefix. cinderCatalogURL and novaCatalogURL append theirs exactly once whichever origin wins, since the validating webhook admits a publicEndpoint only as a bare origin. The public endpoint resolves through one preference order (glanceCatalogURL and its siblings): an explicit services.<svc>.publicEndpoint, advertised verbatim, then the externally routable gateway hostname https://{gateway.hostname}, then the same in-cluster URL when the service is not exposed through a Gateway.

A service carrying a targetClusterRef advertises its public URL on the internal interface too (internalCatalogURL): the in-cluster Service URL resolves only on the cluster that service runs on, so K-ORC — and every other consumer reading the catalog from elsewhere — would otherwise get an address it cannot connect to. Admission requires a placed catalog service to carry a publicEndpoint or a gateway, so that URL is never empty. The K-ORC children the registration projects stay on the management cluster in the ControlPlane's namespace whatever the rows advertise, under the naming convention the KeystoneService Reconciler documents.

The gate is two-staged. The service child is not projected until the registration reports AccountReady (builtinRegistrationAccountReady), because a service pointed at a Keystone user that does not exist, with a password its consumer Secret does not carry, cannot start. A previously projected child keeps running on the credentials it already has. The service's own condition then stays False until the registration reports its aggregate Ready (foldBuiltinRegistrationReady), which covers the catalog row: a running service whose catalog entry never landed is reachable by nothing that discovers it through the catalog.

The relayed reason is the child's first failing sub-condition, walked in keystoneServiceSubConditionTypes order, not the aggregate's. NotAllReady says only that something is wrong, while ServiceCollision on the registration's CatalogReady says what and admits a fix — so a catalog-side reason can and does surface on GlanceReady. The message names which child it came from.

Foreign spec fields are reclaimed, not tolerated. A field manager other than this operator can write assertions the projecting apply cannot take back: spec.catalog.adopt, spec.account.adopt, spec.account.rotation, and endpoint interfaces the ControlPlane never declared. Leaving them standing is not an option, because an undeclared endpoint row is published in the Keystone catalog by the registration's own controller, and every admin-scoped client that resolves it sends its token wherever the row points. An ordinary Update reclaims them, the pass halts, and the next one picks the reclaimed child up. Both a Warning event and the ServiceRegistrationFieldsReclaimed condition record it: an event ages out of etcd on the cluster's TTL, and without the condition a tampering remediated at 02:00 would leave the ControlPlane reporting Ready=True with no durable trace.

Two registrations carry no catalog entry at all. Both spec blocks of a KeystoneService are optional, so a registration naming only the account provisions the Keystone user and touches the catalog not at all. desiredNeutronNovaNotifierRegistration builds one of them: the {controlplane.Name}-neutron-nova child carrying the user the network service posts its port-status notifications to the compute service as. A user another service authenticates as answers no requests itself, so it has no endpoint to advertise, and the KeystoneService controller reports CatalogReady=True with reason CatalogNotDeclared for it. Its project is referenced rather than created: desiredNeutronRegistration owns service-neutron, and a second registration creating the same project would have the first teardown delete it under the second. The account holds admin beside service, which the network service's own registration does not: nova resolves the instance behind a notified port with the caller's own context, unelevated, so an account holding service alone has every notification answered 404 and leaves the port in BUILD. Every other rule builtinRegistration documents holds here too, the explicit ControlPlane namespace and the unset adopt flags among them. The other is {controlplane.Name}-nova-hypervisor-operator, which desiredNovaHypervisorOperatorRegistration builds while services.nova.hypervisorOperator is set.

Credentials follow a placed service. For a service on a target cluster the leg mirrors the registration's consumer credentials there (ensureBuiltinRegistrationMirror): it resolves that cluster, gates on the store being ready on that cluster, and writes an ExternalSecret drawing the same OpenBao path. A co-located service is a no-op. For the credentials' shape and the aggregate condition over every projected registration, see reconcileServiceAccounts.

Path	Status	Reason	Notes
projecting, reading, reclaiming or mirroring the child fails	False	ServiceRegistrationError	returns the error. The refusal to adopt a same-named CR this ControlPlane did not create is relayed verbatim, since it spells out what it refuses and why
the child carries spec fields this ControlPlane never projected	False	ServiceRegistrationFieldsReclaimed	the fields are reset first, so neither the event nor the condition claims a reset that did not land; requeue 10s
the target cluster of a placed service does not resolve	False	TargetClusterUnavailable	the resolver's own message; a wait, not a failed reconcile; requeue 10s
the store on that target cluster is not ready	False	SecretStoreNotReady	an ExternalSecret written against it would never sync, and the service would wait on a Secret whose absence nothing explains; requeue 10s
the registration has not provisioned the account yet	False	WaitingForServiceRegistration	no service child is written at all; requeue 10s
the registration reports a False sub-condition	False	(the child's own reason)	relayed with the child's message and which child it came from; requeue 10s
the registration has not reported Ready at all yet	False	WaitingForServiceRegistration	a child created moments ago, before its controller reconciled it; requeue 10s

reconcileGlance

Aspect	Value
File	reconcile_glance.go
Condition	GlanceReady
Gate	KeystoneReady == True (Glance validates every token against the Keystone child) and the AccountReady of the KeystoneService registration it projects (see Built-in service registrations)
Projects / Owns	one Glance child named {controlplane.Name}-glance (glanceNameSuffix) in cp.GlanceNamespace(); one GlanceBackend child per services.glance.backends entry, named {controlplane.Name}-glance-{entry}; and — managed database only — the per-ControlPlane DB-credential objects in the Glance service namespace: in Dynamic mode (the managed-shared default) a ServiceAccount glance-db-creds, an mTLS client Certificate {controlplane.Name}-glance-db-openbao-client, a VaultDynamicSecret generator reading database/mariadb/creds/glance-{glance-namespace} (auth role glance-db), and a generator-backed ExternalSecret {controlplane.Name}-glance-db-credentials; in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/glance/{glance-namespace}/{controlplane.Name}/db (properties username, password). Only when spec.services.glance is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on Keystone; korcRequeueAfter = 10s while the glance service account is not yet Ready; dbCredentialsRequeueAfter = 10s while the Dynamic DB credential has not landed; infraRequeueAfter = 15s while the child is not Ready

reconcileGlance is the fifth member of the tail group and runs before reconcileServiceAccounts. It gates on the AccountReady of the registration it projects itself, and reconcileServiceAccounts folds that registration later in the same pass. It is optional: spec.services.glance unset means this ControlPlane manages no image service, and the sub-reconciler reports GlanceReady=True / GlanceNotManaged so the aggregate is not blocked (staged adoption). A previously-projected child — and its GlanceBackend children, DB-credential ExternalSecret, and (from a prior Dynamic deployment) the VaultDynamicSecret generator, its client Certificate, and the glance-db-creds ServiceAccount — is preserved unless the ControlPlane opts in with c5c3.io/allow-glance-deletion: "true" (then the orphans, plus the image catalog K-ORC CRs, are deleted). Cross-namespace children are ownership-checked, so a hand-created GlanceBackend sharing the namespace is never touched.

When managed, the projection mirrors the Keystone/Horizon thin discipline, reusing the ControlPlane's own specs so Glance points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/glance with the tag derived from spec.openStackRelease; spec.services.glance.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database: a DeepCopy of the effective database (effectiveGlanceDatabase — Glance's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise) with its logical database name forced to glance so Glance's schema stays isolated from Keystone's on a shared cluster. In managed mode (clusterRef set) the secretRef is repointed at the operator-owned {controlplane.Name}-glance-db-credentials Secret, and the projected credentialsMode is the effective mode: Dynamic (engine-issued) by default on the managed shared database — Glance now has its own engine role glance-{glance-ns}, provisioned by setup-database-tenant.sh — flipped to Static by the shared-block opt-out or the per-service services.glance.databaseCredentialsMode override, and always Static for a dedicated glance database (fail-closed even when the stored mode says otherwise). In Dynamic mode the generator objects above (the glance-db-creds ServiceAccount, the client Certificate, the VaultDynamicSecret, and the generator-backed ExternalSecret) are ensured before the child; in Static mode they are torn down and the KV-backed ExternalSecret is projected, reading a path seeded out-of-band (see Migrate the Glance DB to dynamic credentials). A brownfield database keeps the user-supplied secretRef and credentialsMode.
• Cache: a DeepCopy of the effective cache (effectiveGlanceCache).
• Keystone endpoint: keystoneEndpoint is derived top-down via glanceKeystoneEndpoint(cp) — the cluster-local {controlplane.Name}-keystone Service URL (the same URL K-ORC authenticates against) while Glance and Keystone resolve to the same cluster, never the external publicEndpoint or gateway hostname, because Glance validates tokens server-side and the pods must reach it in-cluster. Placed apart, Glance gets the public URL, the only one that leaves Keystone's cluster (see Reaching a placed service). keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint (empty when Keystone is not externally exposed, in which case the child falls back to the internal URL).
• Service user: derived from the glance account the projected KeystoneService registration provisions — its username, project, and user/project domains — with the password read from the consumer Secret that registration delivers, {controlplane.Name}-glance-credentials.
• Backends: each services.glance.backends entry projects one GlanceBackend child (the CP-side S3 endpoint maps to the child's spec.s3.host; an unset bucketURLFormat serializes away so the child's own path default applies), and previously-projected children whose entry was removed are pruned by pruneProjectedChildren on glanceBackendChildren with the declared names in Keep, matched by the c5c3 ownership and the {controlplane.Name}-glance- name prefix (see Projected satellites and the shared bus). A prune de-registers a store whose id existing image location rows still reference: the backend's name is its store id, so dropping the entry removes the [<name>] section from backends.conf and the <name>:s3 entry from enabled_backends, and Glance no longer resolves that store. Images stored there stay listed in the catalog but GET /v2/images/{id}/file fails for each of them, so the breakage is invisible until download. Recovery is re-creating the entry under the identical name. Unlike unsetting spec.services.glance, which is gated behind c5c3.io/allow-glance-deletion, removing one backends[] line takes effect immediately and is not gated.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.glance.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.glance projects onto spec.deployment (replicas falling back to commonv1.DefaultReplicas), spec.autoscaling and spec.jobs, and its process and thread counts onto spec.apiServer.uwsgi; the resolved store selection and spec.region are projected through.

A child placed outside the ControlPlane's namespace (services.glance.namespace) carries no owner reference — it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.glance unset	True	GlanceNotManaged	staged adoption; a previously-projected child is preserved unless c5c3.io/allow-glance-deletion: "true" is set — but its dynamic DB-credential generator, ServiceAccount, and client Certificate are torn down either way, so no credential minter outlives the service
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Glance CR is projected while Keystone is unready
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Glance child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
Glance placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s; the DB credential resolves its namespace's cluster before it writes anything (ensureServiceDBCredential). The resolver's own message
DB-credential ensure fails (Dynamic generator objects or Static ExternalSecret)	False	GlanceDBCredentialError	returns the error (managed database only)
Dynamic DB credential not yet materialised	False	WaitingForGlanceDBCredential	requeue 10s; no Glance CR is projected and an existing child keeps its current mode until the generator-backed ExternalSecret reports Ready and the Secret it targets carries an engine-issued username. The message names the database/mariadb/creds/glance-<namespace> path, which only exists after setup-database-tenant.sh has onboarded the tenant, or — on a Static→Dynamic migration, where the in-place ExternalSecret update leaves the previous Static sync's Ready in place over a stale Secret — the non-engine-issued username it found
projected GlanceBackend rejected (HTTP 422 Invalid)	False	GlanceBackendProjectionRejected	returns the error; reconcile the services.glance.backends entries to a valid projection to recover
GlanceBackend project/prune fails	False	GlanceBackendError	returns the error
Glance child not yet Ready	False	WaitingForGlance	requeue 15s
projected Glance spec rejected (HTTP 422 Invalid)	False	GlanceProjectionRejected	returns the error; the projection violates a Glance CRD/webhook rule — reconcile the ControlPlane spec to a valid projection to recover
Glance create/update fails	False	GlanceError	returns the error
Glance child Ready	True	GlanceReady	—

reconcilePlacement

Aspect	Value
File	reconcile_placement.go, reconcile_placement_dbcredentials.go
Condition	PlacementReady
Gate	KeystoneReady == True (Placement validates every token against the Keystone child) and the AccountReady of the KeystoneService registration it projects (see Built-in service registrations)
Projects / Owns	one Placement child named {controlplane.Name}-placement (placementNameSuffix) in cp.PlacementNamespace(); and, on a managed database only, the per-ControlPlane DB-credential objects in the Placement service namespace: in Dynamic mode (the managed-shared default) a ServiceAccount placement-db-creds, an mTLS client Certificate {controlplane.Name}-placement-db-openbao-client, a VaultDynamicSecret generator reading database/mariadb/creds/placement-{placement-namespace} (auth role placement-db), and a generator-backed ExternalSecret {controlplane.Name}-placement-db-credentials; in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/placement/{placement-namespace}/{controlplane.Name}/db (properties username, password). Only when spec.services.placement is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on Keystone; korcRequeueAfter = 10s while the placement service account is not yet Ready; dbCredentialsRequeueAfter = 10s while the Dynamic DB credential has not landed; infraRequeueAfter = 15s while the child is not Ready

reconcilePlacement is the sixth member of the tail group, after reconcileGlance and before reconcileServiceAccounts. It gates on the AccountReady of the registration it projects itself. It is optional: spec.services.placement unset means this ControlPlane manages no placement service, and the sub-reconciler reports PlacementReady=True / PlacementNotManaged so the aggregate is not blocked (staged adoption). A previously-projected child is preserved unless the ControlPlane opts in with c5c3.io/allow-placement-deletion: "true"; with the annotation set, the child, its DB-credential ExternalSecret, the Dynamic-mode generator objects, and the placement catalog K-ORC CRs are deleted. Each object is only removed when this ControlPlane still owns it, so a foreign object colliding on a name is left alone.

The credential minter comes down either way. On the preserve branch the VaultDynamicSecret, its client Certificate, and the placement-db-creds ServiceAccount are torn down before the condition is written: a live generator keeps issuing a fresh MySQL user with all privileges on the placement schema at every refresh interval, for a service this ControlPlane has been told it no longer manages, behind a PlacementReady=True condition that surfaces none of it.

When managed, the projection follows the same thin discipline as its Keystone, Horizon, and Glance siblings, reusing the ControlPlane's own specs so Placement points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/placement with the tag derived from spec.openStackRelease; spec.services.placement.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database: a DeepCopy of the effective database (effectivePlacementDatabase: Placement's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise) with its logical database name forced to placement, so its schema stays isolated from Keystone's on a shared cluster. In managed mode (clusterRef set) the secretRef is repointed at the operator-owned {controlplane.Name}-placement-db-credentials Secret (key password), and the projected credentialsMode is the effective mode: Dynamic (engine-issued) by default on the managed shared database, drawn from the engine role placement-{placement-ns} that setup-database-tenant.sh provisions, flipped to Static by the shared-block opt-out or the per-service services.placement.databaseCredentialsMode override, and always Static for a dedicated placement database (fail-closed even when the stored mode says otherwise). A brownfield database keeps the user-supplied secretRef and credentialsMode.
• Cache: a DeepCopy of the effective cache (effectivePlacementCache).
• Keystone endpoint: keystoneEndpoint is derived top-down via placementKeystoneEndpoint(cp), the cluster-local {controlplane.Name}-keystone Service URL (the same URL K-ORC authenticates against) while Placement and Keystone resolve to the same cluster, and never the external publicEndpoint or gateway hostname, because Placement validates tokens server-side and the pods have to reach it in-cluster. Placed apart, Placement gets the public URL (see Reaching a placed service). keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint (empty when Keystone is not externally exposed, in which case the child falls back to the internal URL).
• Service user: derived from the placement account the projected KeystoneService registration provisions, its username, project, and user/project domains, with the password read from the consumer Secret that registration delivers, {controlplane.Name}-placement-credentials.
• ExtraConfig: spec.globalExtraConfig merged with services.placement.extraConfig (the per-service value winning key by key), assigned unconditionally so clearing the ControlPlane block reverts the child instead of pinning the last projected value.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.placement.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.placement projects onto spec.deployment (replicas falling back to commonv1.DefaultReplicas), spec.autoscaling, spec.jobs, and spec.apiServer.uwsgi (only when a process or thread count is set, so the child-side uWSGI defaults apply otherwise); the resolved store selection and spec.region are projected through.

The DB-credential objects are ensured before the child, so the Secret it references exists when the placement operator resolves it. They are ensured for a managed database only: a brownfield one carries a user-supplied credential out-of-band, leaving nothing to project. In Dynamic mode the projection is held (WaitingForPlacementDBCredential) until the generator-backed ExternalSecret reports Ready and the Secret it targets carries an engine-issued username, so an existing child keeps its current mode rather than being handed a DSN for a login the engine never issued.

The placement catalog entry, a placement-type K-ORC Service with an internal and a public Endpoint, is registered by reconcileCatalog rather than here; unsetting spec.services.placement with the deletion annotation set sweeps those CRs along with the child.

A child placed outside the ControlPlane's namespace (services.placement.namespace) carries no owner reference: it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.placement unset	True	PlacementNotManaged	staged adoption; a previously-projected child is preserved unless c5c3.io/allow-placement-deletion: "true" is set, but its dynamic DB-credential generator, ServiceAccount, and client Certificate are torn down either way, so no credential minter outlives the service
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Placement CR is projected while Keystone is unready
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Placement child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
Placement placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s; the DB credential resolves its namespace's cluster before it writes anything (ensureServiceDBCredential). The resolver's own message
DB-credential ensure fails (Dynamic generator objects or Static ExternalSecret)	False	PlacementDBCredentialError	returns the error (managed database only)
Dynamic DB credential not yet materialised	False	WaitingForPlacementDBCredential	requeue 10s; the message names the database/mariadb/creds/placement-<namespace> path, which only exists once setup-database-tenant.sh has onboarded the tenant, or the non-engine-issued username it found in the target Secret
Placement child not yet Ready	False	WaitingForPlacement	requeue 15s
projected Placement spec rejected (HTTP 422 Invalid)	False	PlacementProjectionRejected	returns the error; the projection violates a Placement CRD/webhook rule, so reconcile the ControlPlane spec to a valid projection to recover
Placement create/update fails	False	PlacementError	returns the error
Placement child Ready	True	PlacementReady	—

reconcileBarbican

Aspect	Value
File	reconcile_barbican.go, reconcile_barbican_openbao.go, reconcile_barbican_dbcredentials.go
Condition	BarbicanReady
Gate	KeystoneReady == True (Barbican validates every token against the Keystone child) and the AccountReady of the KeystoneService registration it projects (see Built-in service registrations)
Projects / Owns	a trio in cp.BarbicanNamespace(): one Barbican child {controlplane.Name}-barbican (barbicanNameSuffix), one BarbicanSecretStore {controlplane.Name}-barbican-store (barbicanSecretStoreNameSuffix), and — on a dedicated secret store only — one OpenBaoCluster {controlplane.Name}-barbican-bao (barbicanOpenBaoNameSuffix) with the ensemble below. Plus, on a managed database only, the per-ControlPlane DB-credential objects in the same namespace: in Dynamic mode (the managed-shared default) a ServiceAccount barbican-db-creds, an mTLS client Certificate {controlplane.Name}-barbican-db-openbao-client, a VaultDynamicSecret generator reading database/mariadb/creds/barbican-{barbican-namespace} (auth role barbican-db), and a generator-backed ExternalSecret {controlplane.Name}-barbican-db-credentials; in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/barbican/{barbican-namespace}/{controlplane.Name}/db (properties username, password). Only when spec.services.barbican is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on Keystone; korcRequeueAfter = 10s while the barbican service account is not yet Ready; dbCredentialsRequeueAfter = 10s while the Dynamic DB credential has not landed; infraRequeueAfter = 15s while the dedicated ensemble's target cluster does not resolve, while the OpenBao instance is not Available, while the secret store is being recreated, and while the child is not Ready

reconcileBarbican runs after reconcileGlance and reconcilePlacement in the tail group, with reconcileOVN and reconcileNeutron behind it. It gates on the AccountReady of the KeystoneService registration it projects for itself, which the same pass writes. It is optional: spec.services.barbican unset means this ControlPlane manages no key manager, and the sub-reconciler reports BarbicanReady=True / BarbicanNotManaged so the aggregate is not blocked (staged adoption).

Unsetting the block deletes nothing on its own. c5c3.io/allow-barbican-deletion: "true" opts in to releasing the child, its secret store, its DB-credential ExternalSecret, and the key-manager catalog K-ORC CRs; destroying a dedicated OpenBao instance takes a second annotation on top of that one. Each object is only removed while this ControlPlane still owns it, so a foreign object colliding on a name is left alone. See Barbican secret-store teardown for the order and for what each annotation authorises.

The credential minter comes down either way. On the preserve branch the VaultDynamicSecret, its client Certificate, and the barbican-db-creds ServiceAccount are torn down before the condition is written: a live generator keeps issuing a fresh MySQL user with all privileges on the barbican schema at every refresh interval, for a service this ControlPlane no longer manages, behind a BarbicanReady=True condition that surfaces none of it.

When managed, the projection follows the same thin discipline as its Keystone, Horizon, Glance, and Placement siblings:

• Image: repository defaults to ghcr.io/c5c3/barbican with the tag derived from spec.openStackRelease; spec.services.barbican.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database: a DeepCopy of the effective database (effectiveBarbicanDatabase: Barbican's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise) with its logical database name forced to barbican. The name is fixed: the pre-wired OpenBao bootstrap grants the dynamic credential access to that schema alone. In managed mode (clusterRef set) the secretRef is repointed at the operator-owned {controlplane.Name}-barbican-db-credentials Secret (key password), and the projected credentialsMode is the effective mode: Dynamic (engine-issued) by default on the managed shared database, drawn from the engine role barbican-{barbican-ns} that setup-database-tenant.sh provisions, flipped to Static by the shared-block opt-out or the per-service services.barbican.databaseCredentialsMode override, and always Static for a dedicated barbican database (fail-closed even when the stored mode says otherwise). A brownfield database keeps the user-supplied secretRef and credentialsMode.
• Cache: a DeepCopy of the effective cache (effectiveBarbicanCache).
• Keystone endpoint: keystoneEndpoint is derived top-down via barbicanKeystoneEndpoint(cp), the cluster-local {controlplane.Name}-keystone Service URL (the same URL K-ORC authenticates against) while Barbican and Keystone resolve to the same cluster, and never the external publicEndpoint or gateway hostname, because Barbican validates tokens server-side and the pods have to reach it in-cluster. Placed apart, Barbican gets the public URL (see Reaching a placed service). keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint, the URL Barbican advertises on a 401 (empty when Keystone is not externally exposed, in which case the child falls back to the internal endpoint).
• Service user: derived from the barbican account the projected KeystoneService registration provisions, its username, project, and user/project domains, with the password read from the consumer Secret that registration delivers, {controlplane.Name}-barbican-credentials.
• ExtraConfig: spec.globalExtraConfig merged with services.barbican.extraConfig (the per-service value winning key by key), assigned unconditionally so clearing the ControlPlane block reverts the child instead of pinning the last projected value.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.barbican.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.barbican projects onto spec.deployment (replicas falling back to commonv1.DefaultReplicas), spec.autoscaling, spec.jobs, and spec.apiServer.uwsgi (only when a count is set); the resolved store selection and spec.region are projected through. spec.dbClean is not set, so the clean-up schedule keeps tracking the barbican operator's defaults.

The DB-credential objects are ensured before the child, so the Secret it references exists when the barbican operator resolves it. In Dynamic mode the projection is held (WaitingForBarbicanDBCredential) until the generator-backed ExternalSecret reports Ready and the Secret it targets carries an engine-issued username.

The dedicated OpenBao ensemble

A secretStore.dedicated Barbican gets an OpenBao instance of its own in the Barbican service namespace, projected by ensureBarbicanOpenBao in dependency order before the store that points at it. Every object is named after the instance {controlplane.Name}-barbican-bao, because the openbao-operator derives several of its own object names from that name:

Object	Name	Purpose
OpenBaoTenant	{instance}-tenant	Admits the service namespace to the openbao-operator
Secret	{instance}-unseal-key	The static-seal key, under data key key. Written before the CR exists, or the operator blind-creates an immutable Secret of its own and the generated key is lost
Certificate (cert-manager)	{instance}-tls-server, {instance}-tls-ca	The two fixed-name Secrets the instance's External TLS mode consumes
ServiceAccount	{instance}-provisioner	The identity the Kubernetes-auth role provisioner binds to, which the barbican operator logs in with
Role + RoleBinding	{instance}-provisioner-token	Let the barbican operator mint a bound token for that account
ClusterRoleBinding	{instance}-{hash}-auth-delegator	Grants the TokenReview every Kubernetes-auth login is validated with. Cluster-scoped, so the name folds in eight bytes of sha256("{namespace}/{instance}"): two ControlPlanes of the same name in different namespaces derive the same instance name and would otherwise collide on one binding

The instance runs the Development profile at a single replica with a static seal (see BarbicanDedicatedSecretStoreSpec for what that posture costs) and deletionPolicy: DeletePVCs. The deletion policy is not optional: the instance name is derived from the ControlPlane name, so a re-created ControlPlane of the same name would meet the previous instance's data-{instance}-0 PVC, raft storage initialised under a seal key that no longer exists, and never unseal. Its self-init requests (the KV mount, the AppRole identity, the Kubernetes auth method, and the two policies scoping them) run once against freshly initialised storage, so changing them requires recreating the instance and its PVC.

The whole ensemble is written to the cluster the Barbican service was placed on (its targetClusterRef), because everything that acts on it runs there: the openbao-operator that reconciles the instance, the cert-manager that issues its transport certificates, the TokenReview its Kubernetes-auth logins are validated with, and the Barbican pods that read their key material from it. The cluster-scoped ClusterRoleBinding is created on the target for the same reason — the account it names is a target-cluster account. On the target every object is claimed by the ownership labels alone. The one owner reference in the ensemble is target-local and stays: the unseal Secret takes the instance's, and the two live in the same namespace on the same cluster. The cluster is resolved before the first object is written, so an unresolvable name parks BarbicanReady=False/TargetClusterUnavailable and leaves the ensemble unwritten on both clusters.

The instance's spec.network carries two allowlists. trustedIngressPeers names the barbican operator's pods and the Barbican API pods, the only sources admitted to the API port. apiServerEndpointIPs is the egress half, and resolveAPIServerAccess resolves it per pass from the EndpointSlice kubernetes in default on the cluster the instance runs on, deduplicated and sorted. The policy is enforced by the CNI there, over pods that reach their own API server, so for a placed Barbican the management cluster's addresses would allow the instance nothing it needs. Without it the operator-rendered NetworkPolicy allows the API server only at the in-cluster service VIP on port 443, which a CNI enforcing egress against the post-DNAT destination never matches, and the instance loses the API server: raft auto-join times out, self-init cannot complete, and the partial raft state wedges every later initialization attempt. Sorting keeps the desired-versus-live comparison from reading endpoint reordering as drift, and the read goes through that cluster's uncached reader so no cluster-wide EndpointSlice informer starts for one well-known object.

The openbao-operator renders its rule for apiServerEndpointIPs on port 6443 and no other port. That is where kube-apiserver listens on kind, but behind Gardener's apiserver-proxy the slice publishes port 443. The resolution therefore also reads the slice's port (kube-apiserver publishes one, https) and projects egressRules: one rule per address, a /32 block (/128 for IPv6) on TCP at that port, in the same sorted order. On kind the rules duplicate the operator's own. A slice that carries addresses but no port is refused like an empty one.

The same pass reads the Service kubernetes in default on that cluster and writes https://<ClusterIP>:<port> as the kubernetes_host of the instance's Kubernetes auth method, the address OpenBao sends its TokenReviews to. The ClusterIP rather than the name kubernetes.default.svc, because the name needs a lookup and the operator-rendered NetworkPolicy admits DNS on port 53 alone. Where the resolver answers on another port behind its Service (Gardener's CoreDNS listens on 8053, and a CNI enforcing post-DNAT sees that port) the lookup is dropped, every login through the auth method waits out its deadline, and the store reports ProvisioningReady=False/OpenBaoUnreachable. The ClusterIP needs no lookup and is a subject alternative name of the API server's certificate on every cluster, which the endpoint addresses are not (Gardener's advertise address is absent from it). A Service without a ClusterIP or port is refused like an empty slice.

Once the instance exists, spec.selfInit is carried over from it like spec.storage: self-init ran once against its storage, and the openbao-operator ignores the field on an initialised instance, so a re-projected host would change nothing inside OpenBao. An instance created before the operator wrote the ClusterIP therefore keeps https://kubernetes.default.svc, in its spec and in auth/kubernetes/config. Where that name resolves for the instance pods nothing changes. Where it does not, as on a Gardener shoot, the store stays at ProvisioningReady=False/OpenBaoUnreachable. Deleting the OpenBaoCluster has the next pass recreate it with the ClusterIP, but deletionPolicy: DeletePVCs takes its PVC and every secret Barbican stored in it, so do that only while the store holds nothing Barbican still needs. For a placed Barbican, grant the Service read on the target before upgrading the operator (see Target Clusters).

That resolution fails closed. An instance created without the egress rules is recoverable only by deleting it together with its PVC, and self-init is one-shot, so a pass that cannot resolve the addresses or the Service writes no OpenBaoCluster at all and reports BarbicanReady=False/BarbicanOpenBaoError.

The store, and with it the child, waits until the instance is Available (BarbicanReady=False/WaitingForOpenBaoInstance): a store attached to an instance that is still initialising reports ProvisioningDenied and would have to be re-driven from a failure state. That gate is read from the cluster the instance was provisioned on. The BarbicanSecretStore and the Barbican child themselves do not move: they are the CRs the barbican operator reconciles from the management cluster, and it is that operator which then projects its own children onto the target.

Projecting the secret store

reconcileBarbicanSecretStore projects the one BarbicanSecretStore the service attaches to. It references its Barbican by name (inverted attachment), so it can be applied before the child exists.

A store whose frozen fields moved cannot be updated: the CRD's transition rules reject the write, and the loop would re-attempt it on every requeue with no self-heal. barbicanSecretStoreImmutableDrift compares the four fields the CRD freezes (the barbicanRef, the type, the store mode instanceRef vs server, and the KV mountpoint); when one moved, the live store is deleted and recreated on the next pass, with BarbicanReady=False/RecreatingBarbicanSecretStore reported in between. The AppRole credentials the barbican operator minted for it carry its owner reference, so they are collected with it rather than left pointing at the retired mount.

Every write routes through ensureUnownedOrOwned, and the prune runs pruneProjectedChildren on barbicanSecretStoreChildren with the projected store in Keep, so it deletes only c5c3-owned stores carrying the Barbican child's name prefix and a hand-created store attached to the same Barbican is never pruned or overwritten (see Projected satellites and the shared bus).

The key-manager catalog entry, a key-manager-type K-ORC Service with an internal and a public Endpoint, is registered by reconcileCatalog rather than here. A child placed outside the ControlPlane's namespace (services.barbican.namespace) carries no owner reference: it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.barbican unset	True	BarbicanNotManaged	staged adoption; the child, its store, and the catalog CRs are preserved unless c5c3.io/allow-barbican-deletion: "true" is set, and the dedicated OpenBao instance until c5c3.io/allow-barbican-secret-store-data-deletion: "true" is set on top of it. The dynamic DB-credential generator, ServiceAccount, and client Certificate are torn down either way. The message names whichever annotation is still missing
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Barbican CR is projected while Keystone is unready
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Barbican child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
Barbican placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s from the DB credential, which resolves its namespace's cluster before it writes anything (ensureServiceDBCredential); requeue 15s from the dedicated OpenBao ensemble, which is where a brownfield database reaches the resolver first. Nothing is written, on either cluster. The resolver's own message
DB-credential ensure or read fails (Dynamic generator objects or Static ExternalSecret)	False	BarbicanDBCredentialError	returns the error (managed database only)
Dynamic DB credential not yet materialised	False	WaitingForBarbicanDBCredential	requeue 10s; the message names the database/mariadb/creds/barbican-<namespace> path, which only exists once setup-database-tenant.sh has onboarded the tenant, or the non-engine-issued username it found in the target Secret
provisioning the dedicated OpenBao ensemble fails	False	BarbicanOpenBaoError	returns the error
dedicated OpenBao instance not yet Available	False	WaitingForOpenBaoInstance	requeue 15s; the store and the child are projected once it serves requests
BarbicanSecretStore project or prune fails	False	BarbicanSecretStoreError	returns the error
projected BarbicanSecretStore rejected (HTTP 422 Invalid)	False	BarbicanSecretStoreProjectionRejected	returns the error; reconcile services.barbican.secretStore to a valid projection to recover
live store's frozen fields moved, so it was deleted	False	RecreatingBarbicanSecretStore	requeue 15s; recreated on the next pass
Barbican child not yet Ready	False	WaitingForBarbican	requeue 15s
projected Barbican spec rejected (HTTP 422 Invalid)	False	BarbicanProjectionRejected	returns the error; the projection violates a Barbican CRD/webhook rule, so reconcile the ControlPlane spec to a valid projection to recover
Barbican create/update fails	False	BarbicanError	returns the error
Barbican child Ready	True	BarbicanReady	—

reconcileOVN

Aspect	Value
File	reconcile_ovn.go
Condition	OVNReady
Gate	none. The OVNCentral is deployed outside the plane, so nothing this chain produces can make it ready
Projects / Owns	nothing. The sub-reconciler only READS the OVNCentral named by spec.services.neutron.ovn.centralRef, in cp.NeutronOVNCentralNamespace() (the ref's namespace, or the ControlPlane's own when it is empty)
Requeue	infraRequeueAfter = 15s on every not-yet arm

reconcileOVN is the one sub-reconciler that writes nothing. The OVN control plane is referenced the way the infrastructure clusters in spec.infrastructure are: the ControlPlane never projects the central, never updates it, never deletes it. What it takes from the central is what the Neutron projection consumes, the two database addresses the ML2/OVN mechanism driver dials and the client Secret it presents there. The pass reads the central, decides whether those are usable, and records the verdict in OVNReady.

The read goes through the local client whatever cluster the network service is placed on: the OVNCentral CR lives on the management cluster, where the ovn-operator reconciles it, and only the children it projects land on the target cluster its own spec.targetClusterRef selects.

Which address form is mirrored depends on placement. A network service sharing a cluster with the central reads status.northbound.internalDbAddress / status.southbound.internalDbAddress; one on another cluster leaves its own cluster network and reads status.northbound.dbAddress / status.southbound.dbAddress, which exist only while the central publishes both databases on the node network. A cross-cluster placement therefore also requires spec.northbound.externallyReachable and spec.southbound.externallyReachable on the central.

Before the read, the pass re-runs the reach rule the validating webhook enforces on centralRef.namespace (ValidateNeutronOVNCentralNamespace, exported for this caller). It is the controller-side backstop keystoneServiceNamespaceAllowed is for the sibling cross-namespace trust decision: a ControlPlane can reach etcd without ever passing through admission — an unregistered webhook during install, a GitOps or etcd restore replaying stored objects — and naming a foreign plane's central is not a read-only act. The arms below would relay that central's database addresses and status, and the Neutron projection gated on OVNReady would hand the child a pointer whose operator mirrors the central's client Secret, a full mTLS identity for its Northbound and Southbound databases, into this plane's namespace. So the read does not happen at all, and the condition carries the webhook's own message.

No arm returns an error except the read failure: nothing the ControlPlane does can converge a central it does not own, so every other arm requeues after 15s and waits for the ovn-operator.

Path	Status	Reason	Notes
spec.services.neutron unset	True	OVNNotManaged	staged adoption; no OVN control plane is consumed, so the aggregate is not blocked
the resolved centralRef namespace is outside the plane's reach	False	OVNCentralNamespaceForbidden	requeue 15s; the central is NOT read, so none of its addresses or status reaches this plane. The message is the webhook's own, naming the field and the direction of the refusal
the referenced OVNCentral does not exist	False	OVNCentralNotFound	requeue 15s; the message carries the full <namespace>/<name> the ref resolved to
the OVNCentral CRD is not served	False	OVNCentralReadError	requeue 15s; the message says to install the ovn-operator. A no-match is a deployment gap, not a fault manager backoff can retry out of
reading the OVNCentral fails otherwise	False	OVNCentralReadError	returns the error wrapped as reading OVNCentral <namespace>/<name>
the network service runs on another cluster and the central does not publish both databases	False	OVNCentralNotExternallyReachable	requeue 15s; the message names spec.northbound.externallyReachable and spec.southbound.externallyReachable
the central has not reported Ready, or reports it not True	False	WaitingForOVNCentral	requeue 15s; the central's own reason and message are relayed verbatim, because "not ready" alone says nothing an operator can act on
the central reports Ready but has not published both addresses and status.clientSecretName	False	OVNEndpointsPending	requeue 15s; the message names each status field that is still empty, in the address form the placement selected
the central is ready and complete	True	OVNCentralReady	the message carries the Northbound and Southbound addresses that were selected

reconcileNeutron

Aspect	Value
File	reconcile_neutron.go, reconcile_service_messaging.go, reconcile_neutron_dbcredentials.go
Condition	NeutronReady
Gate	KeystoneReady == True (Neutron validates every token against the Keystone child), OVNReady == True (the ML2/OVN mechanism driver writes every network into the referenced central's Northbound database) and the AccountReady of the KeystoneService registration it projects (of both registrations while spec.services.nova is set; see Built-in service registrations)
Projects / Owns	one Neutron child named {controlplane.Name}-neutron (neutronNameSuffix) in cp.NeutronNamespace(); the bus delivery beside it, a {controlplane.Name}-neutron-messaging Secret (key transport_url) and, only while the shared bus declares tls, a {controlplane.Name}-neutron-messaging-ca Secret (key ca.crt), both written on the client that namespace resolves to and claimed by a controller owner reference at home or by the ownership labels in a service namespace or on a target cluster; and, on a managed database only, the per-ControlPlane DB-credential objects in the same namespace: in Dynamic mode (the managed-shared default) a ServiceAccount neutron-db-creds, an mTLS client Certificate {controlplane.Name}-neutron-db-openbao-client, a VaultDynamicSecret generator reading database/mariadb/creds/neutron-{neutron-namespace} (auth role neutron-db), and a generator-backed ExternalSecret {controlplane.Name}-neutron-db-credentials; in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/neutron/{neutron-namespace}/{controlplane.Name}/db (properties username, password). Only when spec.services.neutron is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on KeystoneReady or OVNReady; infraRequeueAfter = 15s while the bus material has not landed, while the namespace's cluster does not resolve, and while the child is not Ready; korcRequeueAfter = 10s while the neutron service account is not yet Ready; dbCredentialsRequeueAfter = 10s while the Dynamic DB credential has not landed

reconcileNeutron gates in order: KeystoneReady, OVNReady, the shared message bus delivered into the network service's namespace, the KeystoneService registration whose account Neutron authenticates as, and the DB credential the child references. The bus runs ahead of the registration because it reads nothing the registration produces and writes into the namespace on the cluster the child is applied to, so an unresolvable target cluster surfaces before the plane projects the registration that mints the Keystone user. The transport URL's digest is not projected onto the child: the neutron operator rolls its pods off the Secret it derives itself, so a second digest on the child would only add a redundant rollout trigger.

While spec.services.nova is set the pass takes a second registration, the account-only {controlplane.Name}-neutron-nova, and projects spec.nova on the child from it: the region, the four identity names, and the password read from that registration's consumer Secret {controlplane.Name}-neutron-nova-credentials under the key password. The presence of the block is the switch on the child, so a Neutron that carries it turns both notify_nova_on_port_* flags on and renders the [nova] credentials. It is projected only while a compute service is declared, because without one the account is a Keystone user nothing ever authenticates as, and its account gate parks NeutronReady on WaitingForServiceRegistration exactly as the first registration's does. Clearing services.nova assigns spec.nova = nil and, once the child reports the generation that apply produced, deletes the registration (pruneNeutronNovaNotifierRegistration), which is what removes the Keystone user behind it. The wait matters because the deletion cascades to the credentials Secret, and every neutron-server process sources OS_NOVA__PASSWORD from it through a non-optional secretKeyRef until the neutron-operator has re-rendered the workloads without spec.nova; a prune ahead of that, or ahead of a gate that halts the pass before the apply, leaves every neutron-server that restarts in the window unable to start. The preserve-by-default reserve a dropped services.<svc> block gets does not apply: the account is a wire between two services of this ControlPlane rather than a service of its own, and an account outliving the wire only leaves a credential in Keystone nothing uses.

It is optional: spec.services.neutron unset means this ControlPlane manages no network service, and the sub-reconciler reports NeutronReady=True / NeutronNotManaged so the aggregate is not blocked (staged adoption).

When managed, the projection follows the same thin discipline as its Placement and Barbican siblings, reusing the ControlPlane's own specs so Neutron points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/neutron with the tag derived from spec.openStackRelease; spec.services.neutron.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database: a DeepCopy of the effective database (effectiveNeutronDatabase: Neutron's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise) with its logical database name forced to neutron. In managed mode (clusterRef set) the secretRef is repointed at the operator-owned {controlplane.Name}-neutron-db-credentials Secret (key password), and the projected credentialsMode is the effective mode: Dynamic (engine-issued) by default on the managed shared database, drawn from the engine role neutron-{neutron-ns} that setup-database-tenant.sh provisions, flipped to Static by the shared-block opt-out or the per-service services.neutron.databaseCredentialsMode override, and always Static for a dedicated neutron database. A brownfield database keeps the user-supplied secretRef and credentialsMode.
• Cache: a DeepCopy of the effective cache (effectiveNeutronCache).
• Messaging: what serviceMessagingSpec renders for the Neutron target (see Projected satellites and the shared bus): a brownfield secretRef naming the {controlplane.Name}-neutron-messaging Secret the pass wrote beside the child, under the key transport_url, plus a tls.caBundleSecretRef naming the {controlplane.Name}-neutron-messaging-ca mirror under ca.crt, set only while spec.infrastructure.messaging.tls is declared. Both resolve in the Neutron's own namespace on the Neutron's own cluster, which is where the neutron operator looks for them. Dropping the tls block reverts both halves in order: the projection removes the child's pointer first, and the mirror goes once the pass clears the child's readiness return and messagingCAMirrorReleasable opens the gate (the bus declares no tls, the applied child names no TLS block, and status.observedGeneration has caught up with the generation the apply produced). pruneServiceMessagingCA performs the reap. Both steps of the order matter, because the pointer and the volume are removed on different passes: the reap does not sit on the messaging leg, which runs ahead of every gate that can halt the pass (a service-account rotation, a DB credential mid-rotation, a transient API error), and the neutron operator re-renders the Deployment that mounts the mirror only after the pointer is gone from the CR. Reaping ahead of either step leaves a live pod template naming a volume source that no longer exists, wedging every restarting pod on CreateContainerConfigError.
• OVN: spec.ovn.centralRef carries the referenced central's name and its namespace resolved here rather than passed through: an empty ref namespace means the ControlPlane's own namespace, which is not the namespace the child would default it to once it is placed elsewhere.
• Keystone endpoint: keystoneEndpoint is derived top-down via neutronKeystoneEndpoint(cp), the cluster-local {controlplane.Name}-keystone Service URL while Neutron and Keystone resolve to the same cluster, and the public URL when they are placed apart (see Reaching a placed service). keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint.
• Service user: derived from the neutron account the projected KeystoneService registration provisions, its username, the service-neutron project, and both domains from the ControlPlane's effective admin domain, with the password read from the consumer Secret that registration delivers.
• ExtraConfig: spec.globalExtraConfig merged with services.neutron.extraConfig (the per-service value winning key by key), assigned unconditionally so clearing the ControlPlane block reverts the child instead of pinning the last projected value.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.neutron.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.neutron projects onto spec.deployment, spec.workers.deployment (replicas, resources and placement, no spread; both replica counts fall back to commonv1.DefaultReplicas), spec.apiServer.uwsgi (only when a count is set), spec.autoscaling and spec.jobs; the resolved store selection and spec.region are projected through. spec.ovnDBSync, spec.networkPolicy and spec.logging are not set, so the child-side defaults stay authoritative.

Unsetting spec.services.neutron deletes nothing on its own. With c5c3.io/allow-neutron-deletion: "true", deleteOrphanedNeutron releases, in order, the Neutron child, the DB-credential ExternalSecret, the Dynamic-mode VaultDynamicSecret, its client Certificate and the neutron-db-creds ServiceAccount, the two messaging Secrets, and finally both KeystoneService registrations, the network service's own and the compute-notifier account beside it. The first one's finalizer is what tears the network catalog rows, the service user and its project down; the second removes the notifier user alone. Each object is only removed while this ControlPlane still owns it, so a foreign object colliding on a name is left alone. The referenced OVNCentral is never deleted: it is deployed outside the plane and only read.

The credential minter comes down either way. On the preserve branch the VaultDynamicSecret, its client Certificate, and the neutron-db-creds ServiceAccount are torn down before the condition is written: a live generator keeps issuing a fresh MySQL user with all privileges on the neutron schema at every refresh interval, for a service this ControlPlane has been told it no longer manages, behind a NeutronReady=True condition that surfaces none of it.

A child placed outside the ControlPlane's namespace (services.neutron.namespace) carries no owner reference: it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.neutron unset	True	NeutronNotManaged	staged adoption; a previously-projected child is preserved unless c5c3.io/allow-neutron-deletion: "true" is set, but its dynamic DB-credential generator, ServiceAccount, and client Certificate are torn down either way
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Neutron CR is projected while Keystone is unready
OVNReady not True	False	WaitingForOVN	requeue 5s; a Neutron pointed at a central whose databases do not serve cannot program a single network
the shared bus has not delivered its transport URL yet	False	WaitingForMessagingCredentials	requeue 15s; nothing is written, so the child never sees a partial URL
the messaging CA bundle Secret is absent or carries no data under its key	False	WaitingForMessagingCABundle	requeue 15s; an empty key is the ordinary transient of a two-step create-then-populate flow, so it waits rather than mirroring an empty trust anchor
resolving the URL, writing either messaging Secret, or reaping the stale CA mirror after the child stopped naming it fails	False	NeutronMessagingError	returns the error
the cluster the Neutron namespace resolves to is unavailable	False	TargetClusterUnavailable	the resolver's own message; a wait, not a failed reconcile. Requeue 15s from the bus delivery, 10s from the DB credential
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Neutron child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
DB-credential ensure fails (Dynamic generator objects or Static ExternalSecret)	False	NeutronDBCredentialError	returns the error (managed database only)
Dynamic DB credential not yet materialised	False	WaitingForNeutronDBCredential	requeue 10s; the message names the database/mariadb/creds/neutron-<namespace> path, which only exists once setup-database-tenant.sh has onboarded the tenant, or the non-engine-issued username it found in the target Secret
Neutron child not yet Ready	False	WaitingForNeutron	requeue 15s
projected Neutron spec rejected (HTTP 422 Invalid)	False	NeutronProjectionRejected	returns the error; the projection violates a Neutron CRD/webhook rule, so reconcile the ControlPlane spec to a valid projection to recover
Neutron create/update fails	False	NeutronError	returns the error
Neutron child Ready and its registration Ready	True	NeutronReady	—

reconcileCinder

Aspect	Value
File	reconcile_cinder.go, reconcile_service_messaging.go, reconcile_cinder_dbcredentials.go
Condition	CinderReady
Gate	KeystoneReady == True (Cinder validates every token against the Keystone child) and the AccountReady of the KeystoneService registration it projects (see Built-in service registrations). There is no OVN gate, and neither Glance nor Barbican gates the pass
Projects / Owns	one Cinder child named {controlplane.Name}-cinder (cinderNameSuffix) in cp.CinderNamespace(); one CinderBackend satellite per services.cinder.backends entry, named after the entry itself, and one CinderBackupBackend named after services.cinder.backupBackend; the bus delivery beside the child, a {controlplane.Name}-cinder-messaging Secret (key transport_url) and, only while the shared bus declares tls, a {controlplane.Name}-cinder-messaging-ca Secret (key ca.crt); the {controlplane.Name}-cinder KeystoneService registration; and, on a managed database only, the per-ControlPlane DB-credential objects in the same namespace: in Dynamic mode (the managed-shared default) a ServiceAccount cinder-db-creds, an mTLS client Certificate {controlplane.Name}-cinder-db-openbao-client, a VaultDynamicSecret generator reading database/mariadb/creds/cinder-{cinder-namespace} (auth role cinder-db), and a generator-backed ExternalSecret {controlplane.Name}-cinder-db-credentials; in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/cinder/{cinder-namespace}/{controlplane.Name}/db (properties username, password). Only when spec.services.cinder is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on KeystoneReady; infraRequeueAfter = 15s while the bus material or the handed remote bus URL has not landed, while the backing services or the bus do not resolve, and while the child is not Ready; korcRequeueAfter = 10s while the cinder service account is not yet Ready; dbCredentialsRequeueAfter = 10s while the Dynamic DB credential has not landed

reconcileCinder runs in a fixed order: the not-managed branch first, then a nil-safety fail-safe on the resolved database, cache and shared bus (a webhook-bypassed CR requeues after 15s rather than dereferencing a nil), then the KeystoneReady gate, the bus delivery into the block-storage service's namespace, the KeystoneService registration whose account Cinder authenticates as, the DB credential the child references, the two satellite kinds, and finally the child itself through ProjectChild. Past the apply it reaps a released messaging CA mirror and folds the registration's readiness into CinderReady.

The satellites are written before the child. A satellite references its Cinder by name (inverted attachment), so the order is immaterial to the cinder operator, but a satellite that could not be written has to halt the pass: a child applied behind it would run with a backend set the ControlPlane never projected.

It is optional: spec.services.cinder unset means this ControlPlane manages no block-storage service, and the sub-reconciler reports CinderReady=True / CinderNotManaged so the aggregate is not blocked (staged adoption).

When managed, the projection follows the same thin discipline as its Neutron sibling, reusing the ControlPlane's own specs so Cinder points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/cinder with the tag derived from spec.openStackRelease; spec.services.cinder.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Database: a DeepCopy of the effective database (effectiveCinderDatabase: Cinder's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise) with its logical database name forced to cinder. In managed mode (clusterRef set) the secretRef is repointed at the operator-owned {controlplane.Name}-cinder-db-credentials Secret (key password), and the projected credentialsMode is the effective mode: Dynamic (engine-issued) by default on the managed shared database, drawn from the engine role cinder-{cinder-ns} that setup-database-tenant.sh provisions, flipped to Static by the shared-block opt-out or the per-service services.cinder.databaseCredentialsMode override, and always Static for a dedicated cinder database. A brownfield database keeps the user-supplied secretRef and credentialsMode.
• Cache: a DeepCopy of the effective cache (effectiveCinderCache).
• Messaging: what serviceMessagingSpec renders for the Cinder target (see Projected satellites and the shared bus): a brownfield secretRef naming the {controlplane.Name}-cinder-messaging Secret the pass wrote beside the child, under the key transport_url, plus a tls.caBundleSecretRef naming the {controlplane.Name}-cinder-messaging-ca mirror under ca.crt, set only while spec.infrastructure.messaging.tls is declared. Dropping the tls block reverts both halves in the order the Neutron leg documents, and pruneServiceMessagingCA runs on the far side of the child's readiness return for the same reason. The transport URL's digest is not projected: the cinder operator rolls its pods off the Secret it derives itself.
• Keystone endpoint: keystoneEndpoint is derived top-down via cinderKeystoneEndpoint(cp), the cluster-local {controlplane.Name}-keystone Service URL while Cinder and Keystone resolve to the same cluster, and the public URL when they are placed apart (see Reaching a placed service). keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint.
• Service user: derived from the cinder account the projected KeystoneService registration provisions, its username, the service-cinder project, and both domains from the ControlPlane's effective admin domain, with the password read from the consumer Secret that registration delivers.
• Backends: each services.cinder.backends entry projects one CinderBackend satellite of the entry's type carrying the entry's bare name, its nfs or rbd block, and its image-volume cache bounds; services.cinder.backupBackend projects the one CinderBackupBackend. Unset mountOptions, clusterName, secretUUID, fileSize and compression serialize away, so the satellite CRD's own defaults (for secretUUID, the driver's) apply at one layer. Both passes prune what the spec no longer declares (pruneProjectedChildren on cinderBackendChildren / cinderBackupBackendChildren, ownership only, no name prefix). The volume pass projects and then prunes. The backup pass prunes first, because a Cinder takes one backup backend and the cinder webhook refuses a second one beside a live one: a renamed or retyped backup target leaves the plane without one for a pass, and until the entry is fixed if the new satellite is refused. A new backup name that a CinderBackupBackend this ControlPlane did not create already holds is refused before the prune, so the current target stays. Dropping a backends[] entry detaches a live volume backend: the pruned CinderBackend keeps the cinder operator's service-remove finalizer until the backend's volume Deployment is gone and a <cinder>-<backend>-service-remove Job has removed the service row, and the volumes stay on the share or in the pool while nothing serves them any more. Recovery is re-creating the entry under the identical name and type: no admission layer refuses a re-add under the other type, whose driver would take over the volumes it did not create. Unlike unsetting spec.services.cinder, which is gated behind c5c3.io/allow-cinder-deletion, removing one backends[] line takes effect immediately and is not gated.
• Glance, Barbican and the internal tenant: glanceEndpoint is glanceEndpointURL(cp) while services.glance is declared and empty otherwise; keyManager names the Barbican child by naming convention while services.barbican is declared and is nil otherwise; internalTenant carries the projectID and userID the registration publishes on its status.account, and stays nil until both are there. Neither sibling gates the pass: a Cinder with an empty glanceEndpoint and no keyManager runs, and the pass that follows a sibling block being declared re-renders the child with it.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.cinder.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.cinder projects onto spec.api.deployment (replicas falling back to commonv1.DefaultReplicas), spec.api.uwsgi, spec.autoscaling, spec.scheduler.deployment (falling back to one replica), the resources and placement of spec.volume.deployment and spec.backup.deployment, and spec.jobs; the resolved store selection and spec.region are projected through. The volume and backup deployment.replicas are pinned to 1, and the scheduler's is written explicitly: all three blocks are struct values, so the API server materializes the shared default of three onto the wire before the cinder operator's own defaulting webhook runs, and the Cinder CRD's CEL rules admit only 1 on the volume and backup blocks. spec.dbPurge, spec.networkPolicy, spec.logging and spec.policyOverrides are not set, so the child-side defaults stay authoritative.

Unsetting spec.services.cinder deletes nothing on its own. With c5c3.io/allow-cinder-deletion: "true", deleteOrphanedCinder releases, in order, the Cinder child, the DB-credential ExternalSecret, the Dynamic-mode VaultDynamicSecret, its client Certificate and the cinder-db-creds ServiceAccount, the two messaging Secrets, every owned satellite of both kinds, and finally the KeystoneService registration, whose finalizer is what tears the block-storage catalog rows, the service user and its project down. Each object is only removed while this ControlPlane still owns it, so a foreign object colliding on a name is left alone, and a pruned CinderBackend outlives the call while its detach Job runs.

The credential minter comes down either way. On the preserve branch the VaultDynamicSecret, its client Certificate, and the cinder-db-creds ServiceAccount are torn down before the condition is written: a live generator keeps issuing a fresh MySQL user with all privileges on the cinder schema at every refresh interval, for a service this ControlPlane has been told it no longer manages, behind a CinderReady=True condition that surfaces none of it.

A child placed outside the ControlPlane's namespace (services.cinder.namespace) carries no owner reference: it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.cinder unset	True	CinderNotManaged	staged adoption; a previously-projected child and its satellites are preserved unless c5c3.io/allow-cinder-deletion: "true" is set, but the dynamic DB-credential generator, ServiceAccount, and client Certificate are torn down either way
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Cinder CR is projected while Keystone is unready
the shared bus has not delivered its transport URL yet	False	WaitingForMessagingCredentials	requeue 15s; nothing is written, so the child never sees a partial URL
the messaging CA bundle Secret is absent or carries no data under its key	False	WaitingForMessagingCABundle	requeue 15s; an empty key is the ordinary transient of a two-step create-then-populate flow, so it waits rather than mirroring an empty trust anchor
resolving the URL, writing either messaging Secret, or reaping the stale CA mirror after the child stopped naming it fails	False	CinderMessagingError	returns the error
the cluster the Cinder namespace resolves to is unavailable	False	TargetClusterUnavailable	the resolver's own message; a wait, not a failed reconcile. Requeue 15s from the bus delivery, 10s from the registration mirror and the DB credential
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Cinder child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
the store on a placed service's target cluster is not ready	False	SecretStoreNotReady	the registration's credentials cannot be materialised there; requeue 10s
DB-credential ensure fails (Dynamic generator objects or Static ExternalSecret)	False	CinderDBCredentialError	returns the error (managed database only)
Dynamic DB credential not yet materialised	False	WaitingForCinderDBCredential	requeue 10s; the message names the database/mariadb/creds/cinder-<namespace> path, which only exists once setup-database-tenant.sh has onboarded the tenant, or the non-engine-issued username it found in the target Secret
projecting or pruning a satellite fails	False	CinderBackendError	returns the error; the pass halts before the child, which would otherwise run with a backend set the ControlPlane never projected
a projected satellite is rejected (HTTP 422 Invalid)	False	CinderBackendProjectionRejected	returns the error; reconcile the services.cinder.backends / services.cinder.backupBackend entries to a valid projection to recover
Cinder child not yet Ready	False	WaitingForCinder	requeue 15s
projected Cinder spec rejected (HTTP 422 Invalid)	False	CinderProjectionRejected	returns the error; the projection violates a Cinder CRD/webhook rule, so reconcile the ControlPlane spec to a valid projection to recover
Cinder create/update fails	False	CinderError	returns the error
Cinder child Ready and its registration Ready	True	CinderReady	—

reconcileNova

Aspect	Value
File	reconcile_nova.go, reconcile_service_messaging.go, reconcile_nova_dbcredentials.go, reconcile_nova_metadata_secret.go, reconcile_nova_remote_compute.go, reconcile_nova_hypervisor_operator.go, reconcile_nova_metadata_agent.go
Condition	NovaReady
Gate	KeystoneReady == True (Nova validates every token against the Keystone child), PlacementReady == True (the conductor claims every instance's resources in Placement before it boots) and the AccountReady of the KeystoneService registration it projects (see Built-in service registrations). Neutron and Glance are required at admission but gate nothing here, and Cinder and Barbican are optional siblings
Projects / Owns	one Nova child named {controlplane.Name}-nova (novaNameSuffix) in cp.NovaNamespace(); the bus delivery beside it, a {controlplane.Name}-nova-messaging Secret (key transport_url) and, only while the shared bus declares tls, a {controlplane.Name}-nova-messaging-ca Secret (key ca.crt); only while services.nova.remoteCompute is set, a {controlplane.Name}-nova-remote-messaging Secret (key transport_url) carrying the handed external bus URL; the {controlplane.Name}-nova KeystoneService registration; the generated metadata shared secret, an ESO Password generator and an ExternalSecret both named {controlplane.Name}-nova-metadata-secret, unless services.nova.metadataSharedSecretRef names one instead; and, on a managed database only, two per-ControlPlane DB-credential chains in the same namespace. In Dynamic mode (the managed-shared default) each chain carries a ServiceAccount (nova-api-db-creds / nova-cell-db-creds), an mTLS client Certificate ({controlplane.Name}-nova-api-db-openbao-client / {controlplane.Name}-nova-db-openbao-client), a VaultDynamicSecret generator reading database/mariadb/creds/nova-api-{nova-namespace} (auth role nova-api-db) or database/mariadb/creds/nova-cell-{nova-namespace} (auth role nova-cell-db), and a generator-backed ExternalSecret ({controlplane.Name}-nova-api-db-credentials / {controlplane.Name}-nova-db-credentials); in the Static opt-out a KV-backed ExternalSecret of the same name reading openstack/nova/{nova-namespace}/{controlplane.Name}/api-db or …/db (properties username, password); while services.nova.hypervisorOperator is set, the account-only {controlplane.Name}-nova-hypervisor-operator KeystoneService registration and the {controlplane.Name}-nova-hypervisor-operator-auth Secret beside the child, plus a copy of that Secret on every compute-config mirror target; and a {controlplane.Name}-nova-metadata-agent-secret Secret (key shared_secret) in the OVN central's namespace on every cluster a NeutronMetadataAgent of the plane names it on, never deleted by the plane. Only when spec.services.nova is set
Requeue	keystoneInfraGateRequeueAfter = 5s while gated on KeystoneReady or PlacementReady; infraRequeueAfter = 15s while the bus material has not landed, while the backing services or the bus do not resolve, while a compute-config mirror cannot be served, while the child is not Ready, while the hypervisor operator's password has not landed or a target does not resolve, and while the compute contract carries no metadata shared secret for an agent's copy or an agent's cluster does not resolve; korcRequeueAfter = 10s while the nova or the hypervisor operator's account is not yet Ready; dbCredentialsRequeueAfter = 10s while either Dynamic DB credential has not landed

reconcileNova runs in a fixed order: the not-managed branch first, then a nil-safety fail-safe on the resolved database, cache and shared bus (a webhook-bypassed CR requeues after 15s rather than dereferencing a nil), then the KeystoneReady gate, the PlacementReady gate, the bus delivery into the compute service's namespace, the KeystoneService registration whose account Nova authenticates as, the two DB credentials, the metadata shared secret, and finally the child itself through ProjectChild. Past the apply, and only once the child reports Ready for the generation the apply produced, it reaps a released messaging CA mirror and a generated metadata shared secret the child no longer names, delivers the compute contract to every mirror target, runs the hypervisor-operator leg against the same targets, copies the metadata shared secret to the metadata agents on target clusters, and folds the registration's readiness into NovaReady.

The API chain is ensured before the cell chain. Its schema is the one the cell schema's mappings are registered in, so an operator watching a stalled onboarding reads the two waits in the order the db-sync will need them. The mode is resolved once (novaDBCredentialsDynamicEnabled) and drives both projections, both readiness gates, and the value stamped on both of the child's database blocks: the Nova CRD rejects a child whose two blocks carry different credentialsModes, so one verdict has to cover both.

The metadata secret does not wait for its value to materialise. The Nova child gates on that itself (WaitingForMetadataSharedSecret), which is the same wait one requeue later and keeps the two legs from both parking the plane on one Secret.

The generated pair is written while the child reads it: metadataSharedSecretRef unset, or naming {controlplane.Name}-nova-metadata-secret itself. The ExternalSecret renames the generator's password key to shared_secret, the key both the Nova child and a NeutronMetadataAgent default their reference to. Once metadataSharedSecretRef names another Secret the pass writes nothing more, and reapGeneratedNovaMetadataSecret deletes the pair behind the child's convergence rather than ahead of the projection: the Secret goes with the ExternalSecret, and the live metadata Deployment sources its env from it until the nova operator has re-rendered it, which that operator does only once the new Secret exists. The Password is read through the uncached teardown reader, because nothing watches the kind and a cached read would start an informer over every Password on the cluster, one that never syncs on a target whose credentials cannot list the kind. A forbidden read counts as nothing to reap: the generator is only ever written through those same credentials, so a target whose access release predates the passwords grant holds none.

It is optional: spec.services.nova unset means this ControlPlane manages no compute service, and the sub-reconciler reports NovaReady=True / NovaNotManaged so the aggregate is not blocked (staged adoption).

When managed, the projection follows the same thin discipline as its Cinder sibling, reusing the ControlPlane's own specs so Nova points at the same backing services:

• Image: repository defaults to ghcr.io/c5c3/nova with the tag derived from spec.openStackRelease; spec.services.nova.image overrides the whole image reference when set. An image without a pullPolicy of its own takes spec.imagePullPolicy (withControlPlanePullPolicy).
• Two databases: both blocks are a DeepCopy of the effective database (effectiveNovaDatabase: Nova's dedicated database when it opted into one, the shared spec.infrastructure.database otherwise), each taking a copy of its own so the two never alias. apiDatabase.database is forced to nova_api and database.database to nova, the two schemas the pre-wired OpenBao engine roles grant on; the nova operator derives nova_cell0 from the second. In managed mode (clusterRef set) each secretRef is repointed at its own operator-owned Secret (key password) and both blocks carry the one effective credentialsMode: Dynamic by default on the managed shared database, flipped to Static by the shared-block opt-out or the per-service services.nova.databaseCredentialsMode override, and always Static for a dedicated nova database. A brownfield database keeps the user-supplied secretRef and credentialsMode on both.
• Cache: a DeepCopy of the effective cache (effectiveNovaCache).
• Messaging: what serviceMessagingSpec renders for the Nova target (see Projected satellites and the shared bus): a brownfield secretRef naming the {controlplane.Name}-nova-messaging Secret the pass wrote beside the child, plus a tls.caBundleSecretRef naming the {controlplane.Name}-nova-messaging-ca mirror, set only while spec.infrastructure.messaging.tls is declared. Dropping the tls block reverts both halves in the order the Neutron leg documents, and pruneServiceMessagingCA runs on the far side of the child's readiness return for the same reason. The transport URL's digest is not projected: the nova operator rolls its pods off the Secret it derives itself.
• Remote compute: spec.remoteCompute is assigned unconditionally from novaRemoteComputeSpec(cp): nil without services.nova.remoteCompute, and otherwise the public Keystone URL (keystonePublicEndpoint) paired with the {controlplane.Name}-nova-remote-messaging Secret reconcileNovaRemoteMessaging wrote beside the child right after the shared bus. Clearing the ControlPlane block reverts the child, and reapNovaRemoteMessaging deletes the delivered Secret beside the CA-mirror reap, once the child reports observedGeneration >= generation, because the nova operator reads it on every pass until then. A webhook-bypassed ControlPlane with no Keystone publication projects an empty keystoneEndpoint, which the Nova CRD refuses under NovaProjectionRejected.
• Keystone endpoint: keystoneEndpoint is derived top-down via novaKeystoneEndpoint(cp), the cluster-local {controlplane.Name}-keystone Service URL while Nova and Keystone resolve to the same cluster, and the public URL when they are placed apart (see Reaching a placed service). Nova validates every token against it and calls Placement, Neutron, Glance, Cinder and Barbican through it. keystonePublicEndpoint is a pass-through of the Keystone service's own public endpoint.
• Service user: derived from the nova account the projected KeystoneService registration provisions, its username, the service-nova project, and both domains from the ControlPlane's effective admin domain, with the password read from the consumer Secret that registration delivers.
• Metadata: metadata.deployment is projected from spec.sizing.nova.metadata with the replica count written explicitly (falling back to 1), metadata.uwsgi from its process and thread counts, metadata.gateway is a DeepCopy of services.nova.metadataGateway, and metadata.sharedSecretRef is resolved rather than materialised: the Secret services.nova.metadataSharedSecretRef names when the ControlPlane supplies one, and otherwise the generated {controlplane.Name}-nova-metadata-secret under the key shared_secret. Removing a user-supplied reference therefore reverts the child to the generated value instead of pinning the last one.
• Console proxy: three shapes. An absent services.nova.consoleProxy leaves the switch absent on the wire, so the nova defaulting webhook enables the proxy, and carries the deployment sized from spec.sizing.nova.consoleProxy (one replica under Standard, the value that webhook would materialize). A disabled proxy projects the switch and nothing else, because the Nova CRD rejects a spec.consoleProxy.deployment written on a disabled proxy. An enabled one carries the switch, the sized deployment and the listener it was given.
• Endpoints: endpoints.cinder.enabled follows services.cinder and endpoints.barbican.enabled follows services.barbican, so a plane that runs block storage lets Nova attach volumes and one that runs a key manager lets it read an encrypted volume's key. Every endpoint override stays empty on all five sections, Placement, Neutron and Glance included (decision D10 of #1019): the catalog's internal rows already carry the managed in-cluster URLs, so resolving through the catalog reaches the addresses an override would have pinned, and a placed service's rows follow it without a second projection having to agree.
• DB archive: services.nova.dbArchive is DeepCopied and converted onto the child's spec.dbArchive (the two types share one field layout), so the child never aliases cp.Spec. A nil block projects nil, which the nova operator resolves exactly like an empty one.
• Gateway / Replicas / SecretStoreRef / Region: gateway is a DeepCopy of spec.services.nova.gateway (a nil source clears it, tearing the HTTPRoute down); spec.sizing.nova projects onto spec.api.deployment (replicas falling back to commonv1.DefaultReplicas), spec.api.uwsgi, spec.autoscaling and spec.jobs, while the scheduler and conductor Deployments are written explicitly (falling back to 1, for the reason the metadata count is) together with their workers; the resolved store selection and spec.region are projected through. spec.networkPolicy and spec.logging are not set, so the child-side defaults stay authoritative.

The compute-config mirror. The nova operator publishes the compute contract, the nova.conf fragment and bus credentials a nova-compute needs to join this plane, as {controlplane.Name}-nova-compute-config in the Nova namespace. A NovaCompute node pool mounts it in its own namespace on the cluster its pods run on, so the Secret has to exist there a second time. novaComputeConfigMirrorTargets lists the NovaComputes in the Nova namespace and returns one target per cluster a pool of this plane's Nova runs on, sorted, in the Nova namespace. Pools sharing a cluster share a target, the Nova's own placement is dropped because the published Secret lives there already, and a pool being deleted is left out. An unserved NovaCompute kind yields no target; any other list error fails NovaReady with NovaComputeConfigError. mirrorNovaComputeConfig copies the Secret into each target, stamped with this ControlPlane's ownership labels so the teardown reaps it with the rest, and with nova.openstack.c5c3.io/compute-config-mirror: "true". While services.nova.remoteCompute is set the source is the remote contract, {controlplane.Name}-nova-remote-compute-config, whose addresses a compute cluster reaches; the mirror keeps the in-cluster contract's name, the one a NovaCompute reads from its Nova's status.computeConfigSecretRef. A target that cannot be served parks NovaReady rather than failing the pass: the control plane is up, but a compute cluster that never receives the contract registers no hypervisor.

The figure shows the mirror beside the other Secrets a compute cluster receives, and the addresses its components dial.

The plane records nothing in its status about what it mirrored, because reconcileNova runs in the parallel group, which keeps only conditions and metadata. A mirror left behind by the last pool of a cluster is reaped by that pool's own teardown, which deletes the Secret only while it carries the mirror label. A NovaCompute watch wakes the plane whose Nova a pool names, narrowed to pools being created, deleted or starting to be deleted, so a pool's status polls do not reconcile the plane.

The hypervisor-operator leg. While services.nova.hypervisorOperator is set, reconcileNovaHypervisorOperator runs after the mirror loop and takes its targets. It applies the account-only registration {controlplane.Name}-nova-hypervisor-operator through reconcileBuiltinRegistration (user hypervisor-operator, project service-hypervisor-operator created by the registration, role admin), whose halts relay their own reasons onto NovaReady. It then reads the password from the registration's consumer Secret {controlplane.Name}-nova-hypervisor-operator-credentials on the Nova's cluster, writes {controlplane.Name}-nova-hypervisor-operator-auth beside the Nova with the seven keys the hypervisor operator's chart values take (auth_url, username, user_domain_name, project_name, project_domain_name, region_name, password), and copies it into every target with the mirror label, which the last pool on a cluster reaps together with the contract mirror. The source carries no mirror label, so that reap never deletes it. Running after the child is Ready means a stuck account holds NovaReady and never the Nova projection. The Secret-name index lists the consumer Secret while the block is set, so a rotation of the account rewrites the auth Secret and every copy on the next pass.

While the block is unset, pruneNovaHypervisorOperator runs at the same place instead: it deletes the registration, whose finalizer removes the user and the project from Keystone, the credentials ExternalSecret on a placed Nova's cluster, the source Secret, and each target's copy that still carries the mirror label, each only while this ControlPlane owns it. That costs 2 + len(targets) reads per Nova pass, one more for a placed Nova. A read on another cluster goes through that cluster's uncached reader, so the prune starts no Secret informer on a compute cluster. See ServiceNovaHypervisorOperatorSpec for what the account and the Secret are for.

The metadata-agent leg. reconcileNovaMetadataAgentSecrets runs after the hypervisor-operator leg and before the registration fold, so it runs only once the Nova child is Ready. novaMetadataAgentTargets lists the NeutronMetadataAgents in cp.NeutronOVNCentralNamespace(), which is where every agent answering for this plane's ports lives: an agent's chassisRef is namespace-local, a chassis's centralRef is too, and one OVNCentral serves one ControlPlane. An agent qualifies while it is not being deleted, has a targetClusterRef, and names {controlplane.Name}-nova-metadata-agent-secret in spec.novaMetadata.sharedSecretRef; the result is one target per distinct cluster, sorted, in the central's namespace. The Nova's own cluster is not dropped, because the copy's name differs from every Secret the plane writes there. Without services.neutron nothing is listed, and a cluster that does not serve the kind yields no target. With no target the leg reads and writes nothing. Otherwise it reads {controlplane.Name}-nova-compute-config in the Nova namespace through the client childrenClientFor returns, takes its metadata_proxy_shared_secret, and writes the copy onto each target through ensureUnownedOrOwned: type Opaque, the single key shared_secret, the ControlPlane's ownership labels and neutron.openstack.c5c3.io/metadata-shared-secret-mirror: "true". The contract's bus URL and service password never reach the agent's privileged namespace, and a rotated value is rewritten on the next pass. A target that fails holds back none sorted after it: each of those still receives its copy, and NovaReady lists every failed target: NovaMetadataAgentSecretError when a write failed, TargetClusterUnavailable when clusters only did not resolve. A NeutronMetadataAgent watch wakes the plane, narrowed to agents arriving, leaving or re-pointing their shared secret.

The plane never deletes a copy: deleteOrphanedNova leaves it alone, and each agent's teardown reaps the copy on its own cluster once no other live agent there names it (see On a compute cluster). A plane pass already in flight when an agent starts deleting can rewrite a copy that teardown just reaped. The compute-contract mirror has the same race, and the next agent teardown in that namespace on that cluster reaps the copy.

Unsetting spec.services.nova deletes nothing on its own. With c5c3.io/allow-nova-deletion: "true", deleteOrphanedNova releases the Nova child, both DB-credential chains (each ExternalSecret, its VaultDynamicSecret, its client Certificate and its ServiceAccount), the metadata ExternalSecret and the Password generator behind it, the three messaging Secrets (the shared bus, its CA mirror, and the remote bus URL), the hypervisor operator's registration and auth Secret, and finally the KeystoneService registration, whose finalizer is what tears the compute catalog rows, the service user and its project down. The auth Secret's copies on compute clusters are each NovaCompute's own teardown to reap, and the metadata shared-secret copies each NeutronMetadataAgent's. The materialised metadata Secret carries ESO's own owner reference, so it comes down with its ExternalSecret. Each object is only removed while this ControlPlane still owns it, so a foreign object colliding on a name is left alone.

The credential minters come down either way. On the preserve branch both VaultDynamicSecrets, their client Certificates and the two ServiceAccounts are torn down before the condition is written: a live generator keeps issuing a fresh MySQL user with all privileges on a nova schema at every refresh interval, for a service this ControlPlane has been told it no longer manages, behind a NovaReady=True condition that surfaces none of it.

A child placed outside the ControlPlane's namespace (services.nova.namespace) carries no owner reference: it is stamped with the ownership labels and applied unowned, and the finalizer sweeps it by those labels.

Path	Status	Reason	Notes
spec.services.nova unset	True	NovaNotManaged	staged adoption; a previously-projected child is preserved unless c5c3.io/allow-nova-deletion: "true" is set, but both dynamic DB-credential generators, their ServiceAccounts, and their client Certificates are torn down either way
KeystoneReady not True	False	WaitingForKeystone	requeue 5s; no Nova CR is projected while Keystone is unready
PlacementReady not True	False	WaitingForPlacement	requeue 5s; a compute service ahead of its placement service accepts boot requests it cannot serve. The gate reads the condition, so a plane that manages no placement service passes it on the not-managed reason
the shared bus has not delivered its transport URL yet	False	WaitingForMessagingCredentials	requeue 15s; nothing is written, so the child never sees a partial URL
the messaging CA bundle Secret is absent or carries no data under its key	False	WaitingForMessagingCABundle	requeue 15s
resolving the URL, writing either messaging Secret, or reaping the stale CA mirror after the child stopped naming it fails	False	NovaMessagingError	returns the error
services.nova.remoteCompute is set but the handed Secret or its key is missing or empty	False	WaitingForRemoteMessaging	requeue 15s; the message is prefixed services.nova.remoteCompute.transportURLSecretRef: and no Nova child is written this pass
the handed URL is not a rabbit:// URL, or writing or reaping {controlplane.Name}-nova-remote-messaging fails	False	NovaRemoteMessagingError	returns the error, which names the scheme and never the URL
the cluster the Nova namespace resolves to is unavailable	False	TargetClusterUnavailable	the resolver's own message; a wait, not a failed reconcile. Requeue 15s from the bus delivery, the metadata secret and the compute-config mirror, 10s from the registration mirror and the DB credentials
the projected registration has not provisioned the account yet	False	WaitingForServiceRegistration	requeue 10s; no Nova child is written until the Keystone user and its password exist. See Built-in service registrations
projecting, reading or mirroring the registration child fails	False	ServiceRegistrationError	returns the error
the registration child carries foreign spec fields	False	ServiceRegistrationFieldsReclaimed	they are reset and the pass halts; requeue 10s
the store on a placed service's target cluster is not ready	False	SecretStoreNotReady	the registration's credentials cannot be materialised there; requeue 10s
API-schema DB-credential ensure fails	False	NovaAPIDBCredentialError	returns the error (managed database only)
the API-schema Dynamic credential has not materialised	False	WaitingForNovaAPIDBCredential	requeue 10s; the message names database/mariadb/creds/nova-api-<namespace>, or the non-engine-issued username it found in the target Secret
cell-schema DB-credential ensure fails	False	NovaCellDBCredentialError	returns the error (managed database only)
the cell-schema Dynamic credential has not materialised	False	WaitingForNovaCellDBCredential	requeue 10s; the twin of the row above over database/mariadb/creds/nova-cell-<namespace>
ensuring the generated metadata shared secret, or reaping it once the converged child names another Secret, fails	False	NovaMetadataSecretError	returns the error
Nova child not yet Ready	False	WaitingForNova	requeue 15s
projected Nova spec rejected (HTTP 422 Invalid)	False	NovaProjectionRejected	returns the error; the projection violates a Nova CRD/webhook rule, so reconcile the ControlPlane spec to a valid projection to recover
Nova create/update fails	False	NovaError	returns the error
the compute contract has not been published yet	False	WaitingForComputeConfig	requeue 15s; only reachable once a NovaCompute runs on a cluster other than the Nova's own. The message names the source waited for, {controlplane.Name}-nova-remote-compute-config while services.nova.remoteCompute is set
listing the NovaComputes, reading the compute contract, or writing a mirror fails	False	NovaComputeConfigError	returns the error
hypervisorOperator set, the account provisioned, but its consumer Secret absent or without a password	False	WaitingForHypervisorOperatorCredentials	requeue 15s; no auth Secret is written yet
reading the consumer Secret, writing the auth Secret or a copy, or deleting any of them after the block was cleared fails	False	HypervisorOperatorError	returns the error; a failed copy names the namespace and the cluster
an agent asks for the metadata shared-secret copy, but {controlplane.Name}-nova-compute-config is absent or carries no metadata_proxy_shared_secret	False	WaitingForComputeConfig	requeue 15s; the absent-contract message names the central's namespace whose agents sign with the value
the Nova's cluster or an agent's cluster does not resolve for the copy	False	TargetClusterUnavailable	requeue 15s; the resolver's own message, for an agent's cluster prefixed with the namespace and the cluster, one per cluster that did not resolve
listing the agents, reading the contract, or writing a copy fails, a foreign same-named Secret included	False	NovaMetadataAgentSecretError	returns the error; every failed copy names its namespace and cluster, and the message also lists the agent clusters that did not resolve
Nova child Ready, every mirror target served, the hypervisor operator's Secret delivered while its block is set, every requested metadata shared-secret copy written, and its registration Ready	True	NovaReady	—

reconcileKORC

Aspect	Value
File	reconcile_korc.go
Condition	KORCReady
Gate	none (but defers until the admin-password Secret is readable)
Projects / Owns	one K-ORC ApplicationCredential named {controlplane.Name}-admin-app-credential and the password-based clouds.yaml Secret {controlplane.Name}-admin-password-cloud, both in childNamespace(cp)
Requeue	korcRequeueAfter = 10s while deferring, while a re-mint is in progress, or while the AC is not yet Available

reconcileKORC create-or-updates an owned K-ORC ApplicationCredential CR that instructs K-ORC to mint the admin application credential, and drives re-mint. Key behaviours:

• Restricted → Unrestricted inversion (CRITICAL). Our ApplicationCredentialSpec.restricted is the inverse of K-ORC's spec.resource.unrestricted: restricted=true ⇒ Unrestricted=false (ptr.To(!restricted)). restricted defaults to true (least-privilege) when unset, matching the defaulting webhook.
• Password-cloud (breaks the self-referential deadlock). The AC authenticates via an operator-owned, password-based clouds.yaml Secret {controlplane.Name}-admin-password-cloud (ensureAdminPasswordCloud), notk-orc-clouds-yaml. That matters because k-orc-clouds-yaml is the minted application credential itself, so deleting the AC to re-mint would invalidate the very clouds.yaml needed to re-authenticate; a restricted application credential also cannot mint a new application credential. The password-cloud is re-rendered from the live admin password on every pass (so a rotation flows through to it) and is not churned when the password is unchanged. The Domain/User imports and the catalog Service/Endpoint keep using the spec's CloudCredentialsRef (k-orc-clouds-yaml) and tolerate the brief auth gap during a re-mint by requeueing.
• Mode-aware clouds.yaml (korc_cloudsyaml.go). Both builders render auth_url, endpoint_type and region_name through three resolvers. korcAuthURL takes the cluster the document is consumed on and returns the in-cluster Keystone Service DNS in managed mode — keystonePublicEndpoint whenever that cluster is not Keystone's (see Reaching a placed service) — and spec.services.keystone.external.authURL in External mode; korcEndpointType returns internal in managed mode (K-ORC runs in-cluster, so public would resolve to an unreachable Gateway host) and the configured endpointType (default public) in External mode; korcRegion returns spec.region (default RegionOne). Managed-mode output is byte-identical to before, pinned by golden tests, so no upgraded ControlPlane churns its Secrets. The key must be endpoint_type, never interface — K-ORC drops gophercloud's Interface field (the authoritative note lives on buildAppCredCloudsYAML).
• Admin identities. buildPasswordCloudsYAML renders username, project_name and both domain keys from spec.korc.adminCredential.userName/.projectName/.domainName, and ensureKORCAdminImports uses the same userName/domainName as the User and Domain import filters. Same-user constraint: Keystone's default policy mints an application credential only for the token's own user, so the clouds.yaml username and the imported User (the AC's UserRef) must be the same user — both derive from adminUserName.
• UserRef. The required K-ORC UserRef points at the deterministic, cp.Name-scoped User CR {controlplane.Name}-user-admin, imported as unmanaged by ensureKORCAdminImports. The CR name is a stable handle; the OpenStack user it resolves to comes from the import filter above.
• CA bundle (cacert). When spec.services.keystone.external.caBundleSecretRef is set, the referenced bundle is read from the ControlPlane namespace and projected verbatim as the inline cacert key into both operator-owned credentials Secrets (setCACertKey). K-ORC reads that key natively from the same Secret as clouds.yaml, so there is no mount and no upstream change. The password-cloud is what the AC authenticates with directly; the app-credential Secret is the PushSecret's whole-Secret source, so the bundle also reaches OpenBao and is read back by the cacert entry ensureKORCCloudsYAMLExternalSecret adds — gated on the resolved bundle, the same predicate setCACertKey writes the source key under, so the read-back can never point at a property the PushSecret did not push. Clearing the ref deletes the key and drops the read-back entry. A missing Secret/key — or a present-but- empty key, the transient of a two-step "create then populate" flow — defers the mint (WaitingForCABundle); see the CA-cache aliasing caveat.
• Access rules. projectAccessRules maps our {service, method, path} list onto K-ORC's rule shape: service becomes a serviceRef (Kubernetes name ref to an ORC Service CR, e.g. identity), method becomes the typed HTTPMethod enum, and path becomes a string pointer.
• Re-mint trigger (delete + recreate). K-ORC's AC actuator implements only Create + Delete, so a rotated admin password cannot re-mint in place. The SHA-256 of the admin password is stamped onto the AC CR under the cobaltcore.c5c3.io/admin-password-hash annotation (adminPasswordHashAnnotation); on a later pass a mismatch (the hash moved, or the CredentialRotation reconciler zeroed the annotation to nudge) drives reconcileKORC to delete the AC — the finalizer revokes the old Keystone credential, authenticating via the password-cloud — and regenerate the secret value, so the next pass recreates the AC for a fresh mint. The hash is computed by the package-level computeAdminPasswordHash, shared with the CredentialRotation reconciler so both agree on one derivation. The annotation is (re-)stamped only on a fresh mint or when it is absent, never overwriting a present-but-empty value — that empty value is the CredentialRotation reconciler's nudge marker, so preserving it keeps a concurrently-cleared nudge from being silently lost (shouldStampPasswordHash).
• Re-mint on immutable resource-block drift. K-ORC declares the AC's whole spec.resource block immutable via CEL (self == oldSelf), so a legal, webhook- admitted change to restricted or accessRules cannot be reconciled by an in-place update — it would be rejected on every pass. reconcileKORC detects drift on the operator-managed fields (Unrestricted, UserRef, SecretRef, AccessRules — never the whole struct, so a K-ORC/CRD-defaulted sub-field can never read as permanent drift) and routes it through the same delete+recreate re-mint (adminACResourceDrifted).
• Re-mint progress / stall. While the old AC is Terminating the condition is KORCReady=False/ReMinting; if it stays terminating longer than remintStallTimeout (5m) — a finalizer K-ORC cannot clear, e.g. it cannot reach Keystone to revoke — it escalates to KORCReady=False/ReMintStalled.
• Status reflection. updateAdminApplicationCredentialStatus reflects the observed AC into cp.Status.AdminApplicationCredential (ID, the inverted Restricted, and a LastRotation re-stamped whenever the credential ID changes — i.e. advanced by a completed re-mint).
• Missing-CRD safety. If the K-ORC CRD is absent the apiserver/RESTMapper returns a no-match error, detected via meta.IsNoMatchError and surfaced as a clean condition without crash-looping the operator.
• The admin password is read where it lives. In managed mode the operator materialises it beside the Keystone child (reconcileAdminPassword), so a placed Keystone takes it to its target cluster and the read goes through that cluster's client. The client is resolved for effectiveAdminPasswordSecretNamespace(cp), which is the ControlPlane's own namespace in External and brownfield mode — the Secret is the user's there, and that namespace never leaves the management cluster. It is the only cross-cluster read in this sub-reconciler: everything it writes stays local, because K-ORC runs on the management cluster.

Path	Status	Reason	Notes
Keystone placed on a target cluster that does not resolve	False	TargetClusterUnavailable	requeue 10s; the resolver's own message. Nothing is minted, seeded or written, on either cluster
Admin password Secret/key missing	False	WaitingForAdminPassword	requeue 10s (via secrets.IsMissingSecretOrKey)
Admin password read fails otherwise	False	AdminPasswordError	returns the error
Keystone CA bundle Secret/key missing or empty	False	WaitingForCABundle	requeue 10s; no credentials Secret is written before the endpoint can be verified. The message names the field the bundle was read off — external.caBundleSecretRef in External mode, caBundleSecretRef on a placed Keystone
External CA bundle read fails otherwise	False	CABundleError	returns the error
Password-cloud ensure fails	False	PasswordCloudError	returns the error
Hash mismatch → AC deleted for re-mint	False	ReMinting	requeue 10s; AC deleted + value regenerated, recreated next pass
spec.resource drift (restricted/accessRules) → AC deleted for re-mint	False	ReMinting	requeue 10s; the block is CEL-immutable, so the change forces a delete+recreate
Re-mint stuck Terminating past remintStallTimeout (5m)	False	ReMintStalled	requeue 10s; finalizer cannot revoke the old credential
AC create/update/delete/read fails otherwise	False	ApplicationCredentialError	returns the error
value regeneration fails	False	SecretError	returns the error
clearing a latched transport error from an admin import or the AC fails	False	TransportErrorRetryFailed	returns the error; for example a Forbidden status patch when the chart's RBAC predates the patch grant on the K-ORC status subresources. What is latched and what is cleared: Latched transport errors
AC reports a terminal K-ORC error	False	ApplicationCredentialFailed	requeue 10s; gated on orcv1alpha1.GetTerminalError(ac) (an unrecoverable/invalid-config Progressing reason, e.g. K-ORC cannot authenticate with the clouds.yaml) so a credential that will never converge is not reported as an eternal wait. A terminal InvalidConfiguration whose message classifies as a transport failure (no such host, connection refused, dial tcp, i/o timeout), on the AC or on an admin Domain/User import, is cleared from the child's status before this check — at most once every 30s per child, so K-ORC retries and the pass reports WaitingForApplicationCredential. Every other terminal error lands here
AC not yet Available	False	WaitingForApplicationCredential	requeue 10s; gated on orcv1alpha1.IsAvailable(ac) (K-ORC uses Available, not Ready)
AC minted and Available	True	ApplicationCredentialMinted	—

Both the ApplicationCredentialFailed and WaitingForApplicationCredential messages fold in the admin Domain/User import status (ensureKORCAdminImports returns the admin Domain/User imports, and korcAdminImports.statusFragment() names the first that is terminally failed or not yet Available), so the documented endpoint/clouds.yaml failure class — where K-ORC swallows a list error and an import hangs on "created externally" — names the stuck dependency instead of surfacing as an opaque wait.

External-mode failure classification

K-ORC collapses every hard failure against a pre-existing Keystone — a wrong admin password (401), an unresolvable authURL, an untrusted private CA, a region/endpointType absent from the catalog — into the same non-terminalProgressing condition with reason=TransientError. Nothing in the observed inventory is terminal, so neither GetTerminalError nor the reason discriminates: the failure class survives only in the free-text message.

In External mode the ControlPlane therefore classifies on message substrings and relays K-ORC's message verbatim alongside the reason. Classification walks the admin Domain, then the admin User, then the ApplicationCredential, so the root stuck dependency is reported rather than the resource that merely blocked on it. Precedence, most specific first: catalog mismatch, credential drift, TLS, authentication, reachability.

It is gated on External mode — a managed ControlPlane's KORCReady reasons are byte-identical to before — and never runs against an Available credential: K-ORC leaves the message of the last transient attempt on the Progressing condition, and re-classifying it would flip a converged ControlPlane back to AuthenticationFailed on a failure it has already recovered from. A latched transport error on the ApplicationCredential is cleared ahead of the classification, so a latch that is being cleared is never reported as EndpointUnreachable.

Path (External mode only)	Status	Reason	Notes
message contains 401 / Unauthorized	False	AuthenticationFailed	the external Keystone rejected the admin credential — typically the password was rotated out-of-band and passwordSecretRef is stale
message contains no such host / connection refused / dial tcp / i/o timeout	False	EndpointUnreachable	external.authURL could not be dialled
message contains x509	False	TLSVerificationFailed	supply the private CA via external.caBundleSecretRef
message contains No suitable endpoint could be found in the service catalog	False	CatalogEndpointMismatch	a wrong external.endpointType or spec.region; fails loud rather than silently importing nothing. gophercloud never names the interface or region it looked for, so the message appends the effective endpointType and spec.region as the values the external catalog must publish
message names identity:create_application_credential (403)	False	CredentialDrift	an application-credential create against a stale resolve-once import id; see drift
admin import stuck on "Waiting for OpenStack resource to be created externally" beyond externalImportStallGrace	False	ImportStalled	the silent-empty detector

externalImportStallGrace is 2 minutes. In External mode every import target pre-exists by definition, so an import that keeps waiting to be "created externally" never resolves on its own — K-ORC is looking in the wrong place. The message names the stuck import and points at external.endpointType and spec.region. The window is deliberately shorter than remintStallTimeout / orcTeardownStallTimeout (both 5m): those wait on work that is genuinely in flight, whereas a resolvable import has nothing to wait for.

Drift is surfaced, never fought

The external installation can change under the CR: the admin password is rotated without updating the referenced Secret, or the admin user is deleted and recreated. K-ORC imports are resolve-once — a resolved status.id is never re-resolved — so after a recreate the import stays Available=True with a stale id while an application-credential create against that id yields a Keystone 403 (identity:create_application_credential; Keystone's default policy allows creating an application credential only for the token's own user).

The operator never remediates the external installation. It signals drift on the existing sub-conditions with the documented CredentialDrift reason, and reconcileKORC emits a Warning CredentialDrift event on the transition into the drifted state (not on every 10s requeue). The remedy is to update the Secret spec.korc.adminCredential.passwordSecretRef names, which changes its digest and drives a hash-driven re-mint.

Hard CRD dependency. K-ORC (like Memcached, ESO, MariaDB, and Keystone) is a hard dependency: SetupWithManager Owns/Watches its kinds, so the manager fails fast at startup if any CRD is absent. A missing K-ORC CRD never reaches the reconcile path, so there is no dedicated CRD-not-installed condition; a no-match error that could only occur if a CRD were deleted after startup propagates as a hard error (ApplicationCredentialError / ServiceError) and the manager requeues with backoff.

reconcileAdminCredential

Aspect	Value
File	reconcile_admincredential.go
Condition	AdminCredentialReady
Gate	KORCReady == True, the ControlPlane's store is Ready, the K-ORC clouds.yaml ExternalSecret ({childNamespace(cp)}/{CloudCredentialsRef.SecretName}, co-located with the K-ORC CRs per C1) is Ready, the admin app-credential PushSecret has actually synced to OpenBao (its Ready condition is True), and the materialised clouds.yaml Secret semantically matches (parsed application-credential id+secret) the freshly assembled credential
Owns	the operator-owned Secret {controlplane.Name}-admin-app-credential and the PushSecret {controlplane.Name}-admin-app-credential-backup, both in childNamespace(cp)
Requeue	korcRequeueAfter = 10s while any gate is unmet (including a stale/absent materialised clouds.yaml)

reconcileAdminCredential commits the minted credential and mirrors it to OpenBao:

• Operator-owned Secret. The operator generates key value of {name}-admin-app-credential once and keeps it across reconciles, because a new value would force a re-mint. K-ORC only reads value and passes it to Keystone. reconcileAdminCredential writes key clouds.yaml and skips the write when the content is unchanged.
• clouds.yaml gate. Readiness is checked via secrets.WaitForExternalSecret(childNamespace(cp)/CloudCredentialsRef.SecretName) so the credential is never published before K-ORC can actually authenticate. The Secret is co-located with the K-ORC CRs (C1) because K-ORC resolves CloudCredentialsRef in the resource's own namespace; on a fresh cluster reconcileKORC itself seeds a password-based bootstrap clouds.yaml into the {controlplane.Name}-admin-app-credential Secret (seedBootstrapCloudsYAML, write-if-empty) and the PushSecret mirrors it to the per-ControlPlane OpenBao path, so the operator-created per-CR ExternalSecret can materialise before any credential is minted — once the AC is minted the PushSecret carries the minted credential-based clouds.yaml instead.
• PushSecret to OpenBao. secrets.EnsurePushSecret (applied via server-side apply under a fixed field manager that owns only the fields the operator sets, so repeated applies of an unchanged desired spec are no-ops at the API server) builds the PushSecret to the ControlPlane's store (its store ref comes from spec.secretStoreRef via secrets.PushSecretStoreRefs, and switching the ref moves the push in place — unchanged name and remote key) at the per-ControlPlane remote key openstack/keystone/{cp.Namespace}/{cp.Name}/admin/app-credential (adminAppCredentialRemoteKeyFor) with DeletionPolicy: Delete; see Security invariant for why the credential leaves OpenBao with the ControlPlane.
• Forced re-push on credential change. ESO's PushSecret controller does not watch its source Secret: its refresh gate reacts only to the PushSecret object's own label/annotation hash, so a source-Secret update — e.g. the fresh-create handoff from the password-based bootstrap clouds.yaml to the minted credential — would otherwise not reach OpenBao until the hourly refreshInterval, leaving AdminCredentialReady stuck False for up to an hour. forceRepushAdminAppCredential therefore stamps the assembled clouds.yaml content hash onto the PushSecret (c5c3.io/push-content-hash, read-modify-write under RetryOnConflict), changing its metadata hash and forcing an immediate re-push. The annotation is keyed by the content hash, so it fires exactly once per credential change and a steady-state pass is a no-op.
• PushSecret sync gate. AdminCredentialReady is gated on the PushSecret's Ready condition — not merely on the CR existing — so a backend permission failure (e.g. the ESO role missing the push policy) surfaces as WaitingForPushSecret instead of a false-positive Ready while OpenBao still serves the password-based bootstrap clouds.yaml.
• Live clouds.yaml gate (stale-credential window). A re-mint revokes the old credential immediately, but the k-orc-clouds-yaml Secret only refreshes from OpenBao at the ExternalSecret's hourly refreshInterval, so the PushSecret-Ready check above can pass while the materialised Secret K-ORC actually authenticates with still holds the revoked credential. After assembling the clouds.yaml, reconcileAdminCredential stamps the external-secrets.io/force-sync annotation to nudge ESO to re-materialise immediately. The trigger value is contentHash + "/" + PushSecret.status.syncedResourceVersion, not the content hash alone: both nudges (the PushSecret re-push above and this force-sync) are stamped in the same reconcile pass, but ESO processes the two objects independently — keyed on the content hash alone, the ExternalSecret refresh can read OpenBao before the re-push has written it, re-materialise the stale bootstrap document, and (with the annotation already at its final value) never be nudged again, wedging AdminCredentialReady at WaitingForCloudsYamlSync until the hourly refresh. ESO bumps syncedResourceVersion only after a completed push, so folding it in re-nudges the ExternalSecret exactly once more as soon as the re-push lands; both inputs are stable once converged, so a steady-state pass still leaves the ExternalSecret untouched. It then compares the materialised Secret semantically — by the parsed application-credential id and secret, not byte-for-byte, so a benign ESO/OpenBao re-serialisation (a stripped trailing newline, reordered keys, requoting) cannot wedge the gate permanently — and only reports AdminCredentialReady=True when they match. The semantic compare — not the best-effort force-sync — is the correctness guarantee: the condition never reads True against a stale credential. A sync that never converges is bounded: once the materialised Secret has failed to match for longer than cloudsYamlSyncStuckTimeout (measured from the credential's LastRotation), the reason escalates from the transient WaitingForCloudsYamlSync to the alertable CloudsYamlSyncStuck, so a permanently broken sync is distinguishable from a 2-second transient miss.
• Drift escalation (External mode). When the KORCReady gate is closed because the external Keystone reports drift, the gate reports CredentialDrift rather than the opaque WaitingForKORC. Note that the live drift signal is KORCReady itself: reconcileKORC writes the drifted KORCReady earlier in the pass, and because both run inside the non-short-circuiting tail group, reconcileAdminCredential runs later in the same pass and observes it directly — so this CredentialDrift escalation is the live same-pass path that keeps AdminCredentialReady honest, not merely a defense-in-depth fallback. An unreachable endpoint, a TLS failure, a catalog mismatch or a stalled import are not drift and keep WaitingForKORC — drift is a statement about the credential.

Path	Status	Reason	Notes
KORCReady not True	False	WaitingForKORC	requeue 10s
KORCReady reports drift (AuthenticationFailed / CredentialDrift), External mode	False	CredentialDrift	requeue 10s; names the Secret the operator reads and states that the external installation is never remediated
Selected secret store not Ready	False	SecretStoreNotReady	requeue 10s; checked after the KORCReady gate so an OpenBao/ESO outage surfaces before the clouds.yaml wait. The message names the store's kind and name
Selected secret store check errors	False	SecretStoreError	returns the error; stamped rather than returned bare so the peers gating on AdminCredentialReady later in the tail group cannot read the previous pass's True
clouds.yaml ES check errors	False	CloudsYamlError	returns the error (also covers a force-sync/materialised-Secret read error)
clouds.yaml ES not Ready	False	WaitingForCloudsYaml	requeue 10s
operator Secret ensure fails	False	SecretError	returns the error
PushSecret ensure, re-push stamp, or readback fails	False	PushSecretError	returns the error
PushSecret not yet synced to OpenBao	False	WaitingForPushSecret	requeue 10s; the PushSecret's Ready condition is not True — the assembled credential has not landed in OpenBao yet
materialised clouds.yaml absent or semantically stale	False	WaitingForCloudsYamlSync	requeue 10s; force-sync annotation stamped, semantic compare (parsed id+secret) against the assembled document not yet satisfied
materialised clouds.yaml stuck stale past cloudsYamlSyncStuckTimeout	False	CloudsYamlSyncStuck	requeue 10s; the sync has not converged since LastRotation — alertable, distinguishable from a transient miss
committed, mirrored, and materialised	True	AdminCredentialReady	—

reconcileCatalog

Aspect	Value
File	reconcile_catalog.go (Managed), reconcile_catalog_external.go (External)
Condition	CatalogReady
Gate	AdminCredentialReady == True, and every catalog child reports Available
Owns	Managed mode: a K-ORC identity Service ({controlplane.Name}-identity-service) and its public Endpoint ({controlplane.Name}-identity-endpoint). That is the whole managed table: a built-in service's catalog row belongs to the KeystoneService registration projected for it, not here. Managed mode also owns the Region ({controlplane.Name}-region, onDelete: detach) that adopts the bootstrap region. External mode: the same identity Service plus one Endpoint per interface ({controlplane.Name}-identity-endpoint-{interface}), all unmanaged imports, and nothing managed. All in childNamespace(cp)
Requeue	korcRequeueAfter = 10s while gated, while a child is not yet Available, or on a terminal K-ORC failure

reconcileCatalog drives CatalogReady. Everything up to and including the AdminCredentialReady gate and the admin CloudCredentialsReference is mode-agnostic; below it the two postures are opposites, so the reconciler forks on cp.IsExternalKeystone(). K-ORC is a hard CRD dependency (see the note above), so a missing Service/Endpoint CRD never reaches this path and there is no CRD-not-installed condition.

Managed mode — the control plane owns the catalog

reconcileCatalog registers the OpenStack service-catalog entries the control plane itself owns as owned K-ORC CRs, driven from a per-service table (managedCatalogRows). That table holds exactly one row, the identity (Keystone) service: an identity-type Service named keystone, plus a public Endpoint whose URL defaults to the conventional in-cluster identity URL http://keystone.<namespace>.svc:5000/v3 and whose serviceRef points at the identity Service. The child is projected idempotently via Server-Side Apply under the shared field manager (cobaltcore-operator).

The rows for the built-in services are not here. Glance's image row, Placement's placement row and Barbican's key-manager row each belong to the KeystoneService registration their service leg projects, which owns the row's K-ORC children and their teardown; see Built-in service registrations for what each one advertises. The identity row stays ControlPlane-owned in both modes because nothing registers it on the plane's behalf: it is the entry K-ORC authenticates through.

The identity row registers a public interface only, so it needs none of the target-cluster switching a placed service's row does. Its Service/Endpoint CRs stay in childNamespace(cp) on the management cluster: they are K-ORC's to reconcile, and K-ORC runs there. See Reaching a placed service.

Alongside the rows, the reconciler projects one managed Region ({controlplane.Name}-region) per ControlPlane. It adopts the Keystone region named by spec.region, the row the keystone bootstrap inserted, and adoption runs in two phases. The first apply carries the region name and no description, which is what K-ORC's adoption filter matches on; once the CR reports status.id, the next pass adds spec.resource.description from spec.regionDescription. That order is fixed by the filter: it also matches on the description when the spec carries one, and the bootstrap row's description is empty, so a described first apply matches nothing, falls through to create, and takes Keystone's 409 Conflict as a terminal error. Each pass does one Get and one apply, never two, so the description does not flip the CR's generation on every pass.

spec.managedOptions.onDelete is detach. Keystone answers 403 for a region that still has endpoints, and the bootstrap identity endpoints reference this one, so deleting the CR leaves the Keystone row where it is; that row goes with the Keystone database. An empty spec.regionDescription keeps the Keystone description empty, and K-ORC applies the spec value on every resync, so a description set by hand in Keystone is overwritten either way. There is no third state: a managed Region drives the description, and K-ORC reads an absent spec.resource.description as the empty string. On upgrade that makes the first reconcile destructive for a control plane whose region was described by hand (openstack region set RegionOne --description ...) — the description is cleared, and nothing in the cluster holds the prior value. Copy it into spec.regionDescription before rolling out this version to keep it. Because that upgrade adopts the region without anyone editing the ControlPlane, admission never runs on it: the pass that creates the Region CR with spec.regionDescription empty therefore emits a Warning RegionDescriptionCleared event on the ControlPlane, naming the region and the field that would keep a description.

Only the upgrade raises it. The event is gated on CatalogReady being True while the Region CR is still absent, which is a state a fresh install cannot reach — there the CR is created on the same pass that first registers the catalog, and CatalogReady only turns True once that Region reports Available. So the notice reaches the control planes whose region may carry a hand-set description and stays off the ones whose bootstrap row has none to lose. It is emitted once, on the pass before K-ORC first writes to the region — no later pass repeats it, a failed Region apply stamps CatalogReady False and silences the retry pass, and a ControlPlane that carries a description never raises it.

The identity row names no region of its own: managedCatalogService and managedCatalogEndpoint set the management policy, the credentials ref, and the resource block (type/name/enabled, and interface/URL/serviceRef), and nothing else. The row K-ORC adopts for it is the public endpoint the keystone bootstrap inserted into spec.region, so it already sits in that region. The rows the registration children create are placed there explicitly: each KeystoneService imports spec.region and names the import as every Endpoint's regionRef (see Catalog Projection). A client that sets region_name, as every nova client section and the neutron notifier do, finds no region-less row. spec.region is also the clouds.yamlregion_name the admin credential renders.

Registering the child CRs only instructs K-ORC to create the catalog entries — it does not mean they exist in Keystone — so CatalogReady is gated on every child reporting Available for its current generation (korcAvailableUpToDate, which refuses a stale Available condition whose ObservedGeneration lags the object — the same generation gate GetTerminalError already applies via its Progressing check — so an endpoint/region edit that moves the catalog URL cannot flip CatalogReady True before K-ORC re-reconciles the new value), and a terminal K-ORC failure (GetTerminalError, the documented wrong-endpoint / import-stuck class) is surfaced as the distinct CatalogFailed reason instead of a false-positive Ready.

Path	Status	Reason	Notes
AdminCredentialReady not True	False	WaitingForAdminCredential	requeue 10s
Service create/update fails	False	ServiceError	returns the error
Endpoint create/update fails	False	EndpointError	returns the error
Region read (other than NotFound) or apply fails	False	RegionError	returns the error
clearing a latched transport error from a Service, Endpoint or the Region fails	False	TransportErrorRetryFailed	returns the error
a catalog entry's Service/Endpoint, or the adopted Region, reports a terminal K-ORC error	False	CatalogFailed	requeue 10s (Service before its Endpoints, so the root stuck dependency surfaces; the Region is checked after the rows and its message names Region). A latched transport error (the InvalidConfiguration rule in the KORCReady table) is cleared before this check, so K-ORC retries and the pass reports WaitingForCatalog. Every other terminal error lands here
a catalog entry's Service/Endpoint, or the adopted Region, registered but not yet Available	False	WaitingForCatalog	requeue 10s; the Region also fails this gate on the pass where the description apply bumped its generation
the region adopted, every catalog entry registered and Available	True	CatalogRegistered	the message names the adopted region and counts the registered entries, which is the identity row alone today; the table is what a future ControlPlane-owned row would be added to

External mode — import-first

The catalog belongs to the pre-existing installation. Keystone enforces no uniqueness on service names, so registering an identity Service against a populated catalog would silently duplicate rows. reconcileCatalogExternal therefore imports instead: the identity Service and each of its three endpoint interfaces become K-ORC CRs with managementPolicy: unmanaged, an import filter, and no desired resource. K-ORC resolves them read-only and writes nothing; deleting their CRs later removes only the Kubernetes objects.

The ControlPlane creates zero catalog entries in External mode: this branch of reconcileCatalog is import-only. A genuinely new row is declared with a KeystoneService CR, whose own reconciler creates it, gates its own CatalogReady on it, and deletes it at that CR's teardown — so nothing the ControlPlane writes can duplicate or outlive a row in a catalog it does not own.

status.catalog.imports is rebuilt before any failure return, so an unresolved import is visible as resolved: false rather than omitted.

Import-first inverts the failure modes, and detecting them is the point. A K-ORC import that matches nothing does not error: it waits indefinitely on Available=False, reason=Progressing, "Waiting for OpenStack resource to be created externally" — by conditions indistinguishable from a resource that is about to appear. For a gating import the target pre-exists by definition, so past externalImportStallGrace (2m) that wait is a misconfiguration signal, not a wait. An import that matches several entries is terminal in K-ORC itself, which refuses to guess and stops retrying.

Only two of the four imports gate CatalogReady: the identity Service, and the Endpoint of the interface external.endpointType selects. The control plane already authenticates through that interface, so a catalog that does not publish it is not the catalog K-ORC was pointed at. The other two interfaces are imported for visibility, projected into status.catalog.imports, and may stall forever — or resolve ambiguously — without failing the condition. An external installation is free not to publish an interface, and free to publish it once per region (which K-ORC's region-less EndpointFilter cannot select among, so no spec edit repairs it); both are precisely the brownfield posture External mode adopts.

The precedence below reports the most specific cause first, and the Service before the Endpoints so the root stuck dependency is named rather than an endpoint merely blocked on the service it references:

#	Path	Status	Reason	Notes
—	AdminCredentialReady not True	False	WaitingForAdminCredential	requeue 10s; no import CR is reconciled
—	import create/update fails	False	ImportError	returns the error
—	clearing a latched transport error from an import fails	False	TransportErrorRetryFailed	returns the error
1	an unresolved import carries a classifiable K-ORC message	False	AuthenticationFailed | EndpointUnreachable | TLSVerificationFailed | CatalogEndpointMismatch | CredentialDrift	requeue 10s; K-ORC's message is relayed verbatim. A resolved import is never re-classified — K-ORC leaves the last transient attempt's message on Progressing, and classifying it would flip a converged catalog to a failure it has recovered from
2	an import reports a terminal K-ORC error	False	CatalogFailed	requeue 10s; gating or not — K-ORC has given up on it. A latched transport error (the InvalidConfiguration rule in the KORCReady table) is cleared before this row, so K-ORC retries and the pass falls through to the waits below. On the >1-match message the hint names external.catalog.identityServiceName for the Service import, or the region limitation for an Endpoint import (K-ORC's EndpointFilter carries no region, so no spec field can select among per-region rows). One exception: an InvalidConfiguration on a non-gating import does not fail the condition. A non-gating import has no user-supplied configuration to fix — its filter is entirely operator-derived — so it has no remediation and nothing depends on it; it is tolerated exactly like the 0-match of row 3 and reported as resolved: false. The exception is keyed on K-ORC's machine-readable reason, never on the >1-match message text: keying it on the text would turn a K-ORC rewording into a permanent CatalogReady=False. An UnrecoverableError gates on every import, and so does any terminal error on a gating one
3	a gating import stalled past externalImportStallGrace	False	ImportStalled	requeue 10s; the 0-match case. The message names the stuck import, the authURL, and external.endpointType / spec.region — plus, for an Endpoint import, that the external catalog may publish no such interface
4	a gating import is unresolved	False	WaitingForCatalog	requeue 10s; the bounded, legitimate wait
5	every gating import resolved	True	CatalogImported	the message reports how many of the three endpoint interfaces resolved

A latched transport error is cleared ahead of row 1, so a latch that is being cleared is not classified as EndpointUnreachable.

publicEndpoint is forbidden in External mode, so keystoneCatalogURL — the URL the Managed branch registers — is never consulted here: advertisement visibility is owned by the imports.

Promote-to-managed is reserved, not implemented. Turning an import into a managed row (to edit its endpoint URL declaratively) is a later phase. K-ORC's managementPolicy is CEL-immutable, so it will have to be a delete-and-recreate of the import CR. Nothing in the deterministic CR names or the spec-derived filters chosen here precludes that.

reconcileServiceAccounts

Aspect	Value
File	reconcile_serviceaccounts.go
Condition	ServiceAccountsReady
Gate	none — the member only reads the KeystoneService children the built-in service legs wrote earlier in the same pass, so there is no projection it could defer
Requeue	korcRequeueAfter = 10s while a registration is not yet Ready

reconcileServiceAccounts aggregates: it reads the KeystoneService registration child each enabled built-in service leg projects (services.glance, services.placement, services.barbican, services.neutron, services.cinder, services.nova, in that order) and folds their readiness into the one condition operators alert on. The network service's account-only compute-notifier registration is aggregated beside its own, and only while both services.neutron and services.nova are set: without a compute service the account is a Keystone user nothing authenticates as. The hypervisor operator's account-only registration {controlplane.Name}-nova-hypervisor-operator follows nova's own while services.nova.hypervisorOperator is set. The Nova leg applies it only once the Nova child is Ready, so until then the aggregate reads it as not projected yet (WaitingForServiceRegistration). It projects no OpenStack resource itself; the registration CR owns the Keystone user, its project, its role assignments, the generation-scoped password, and the OpenBao round-trip that delivers the credentials.

The double reporting is intended. A failing registration already fails its own service condition (GlanceReady, PlacementReady, BarbicanReady, NeutronReady, CinderReady, NovaReady); the aggregate names the same cause under the condition type that does not depend on knowing which service broke.

Path	Status	Reason	Notes
no built-in service block is declared	True	NoServiceRegistrationsProjected	every External-mode ControlPlane; set rather than omitted so the condition schema does not depend on the spec
a registration child is absent	False	WaitingForServiceRegistration	its service leg is gated on something upstream of its own apply; requeue 10s
reading a registration child fails	False	ServiceRegistrationError	returns the error, wrapped with which child was being read
a registration reports a False sub-condition	False	(the child's own reason)	relayed verbatim with the child's message, prefixed by which child it came from; requeue 10s
a registration has not reported Ready yet	False	WaitingForServiceRegistration	requeue 10s
every registration reports Ready	True	ServiceAccountsProvisioned	the message counts them

The relayed reason is the child's first failing sub-condition, not the aggregate's. keystoneServiceSubConditionTypes orders CatalogReady before AccountReady, so a registration whose Keystone catalog row failed reports a catalog reason (CatalogFailed, ServiceCollision) under a condition named for service accounts. That is deliberate: NotAllReady says only that something is wrong, while the sub-condition's own reason says what and admits a fix.

Consumption contract. Consumers read the credentials from the Secret the registration materializes in its own namespace — {keystoneservice.Name}-credentials, keys password and a ready-to-use clouds.yaml — named in the registration's status.account.secretName. For which OpenBao paths exist in each Keystone mode, see OpenBao paths per ControlPlane mode.

Deletion. The registrations go first at teardown: reconcileDelete deletes the projected KeystoneService children and waits for them, because their K-ORC CRs belong to the registration and its controller tears them down through the admin credential the next step revokes.

reconcileRegistrationTenantStores

Aspect	Value
File	reconcile_esotenant.go
Condition	RegistrationTenantStoresReady
Gate	none — it consumes no condition this chain produces; the trio it writes depends on cert-manager and OpenBao alone
Projects / Owns	the same trio reconcileESOTenantStore writes (a ServiceAccount eso-tenant-auth, a cert-manager mTLS Certificate eso-tenant-client-tls, and a namespaced SecretStore openbao-tenant-store), one per allowlisted namespace hosting at least one KeystoneService; nothing when spec.secretStoreRef overrides the default
Requeue	esoTenantStoreRequeueAfter = 10s while a store is not yet Ready, and on a provisioning failure

Credential delivery is namespace-local. A registration's consumer Secret is materialized through the tenant store in the CR's own namespace, and reconcileESOTenantStore provisions one only in the namespaces the ControlPlane occupies. Without this member an allowlisted foreign registration would mint its Keystone account and then wait forever on a store nobody creates.

It shares ensureESOTenantStoreTrioIn with its blocking-prefix twin, so the ownership mechanism, the adoption refusal and the error wording are written once. The OpenBao side needs nothing: the eso-tenant role binds the ServiceAccount name in any namespace and its templated policy confines each token to its own paths, so a trio in a foreign namespace authenticates as that namespace's tenant identity and reaches only its own paths.

Why it is a member of its own, at the end of the group. The tenant-store step runs in the chain's blocking prefix, where a non-zero result parks DBCredentials, AdminPassword and Keystone behind it. These namespaces are not the operator's — a foreign object can occupy the store's name, a certificate can fail to issue there — and none of that may reach the plane's core reconciliation. A failure here surfaces in this condition, and through it in the aggregate Ready, and stops. For the same reason a provisioning failure requeues rather than returning an error: the cause is usually a tenant-side one no backoff resolves, and returning an error would put the whole ControlPlane reconcile into exponential backoff for it. Every namespace is attempted even after one fails, so a single broken namespace cannot starve its peers, and the failures are reported together.

Collecting, and the one namespace it will not collect. The namespaces already provisioned are enumerated by ownership label before the allowlist is consulted, so emptying the allowlist still collects the trios whose registrations have left. A namespace that no longer appears in the allowlist but still holds registrations is frozen, not collected: de-listing is an admission gate, and revoking the store under a running service would destroy credentials it depends on. See ServiceRegistrationsSpec.

Path	Status	Reason	Notes
spec.secretStoreRef set (override)	True	StoreRefOverridden	nothing to provision and nothing this member created standing to collect; a registration resolves that store in its own namespace itself
no allowlisted namespace hosts a registration	True	NoRegistrationNamespaces	both arms share one message: nothing admitted at all, and an allowlist no registration sits in. Returns before the registration List, so a ControlPlane that never uses the feature never depends on the KeystoneService field index
provisioning or collecting a trio failed	False	ProvisioningError	requeue 10s, no error returned; the message names the failing namespaces and the joined error. A failed List while enumerating the provisioned namespaces or counting registrations sets the same reason and returns the error
a store is written but not yet Ready	False	SecretStoreNotReady	requeue 10s, naming the namespace; the delivery leg gates on exactly this, so reporting True early would claim a path that does not carry yet. A failed readiness read returns the error with no condition written, leaving the previous value standing
every store Ready	True	RegistrationTenantStoresReady	the message counts the namespaces

reconcileNamespaceAssignments

Aspect	Value
File	reconcile_namespace_assignments.go
Condition	NamespaceAssignmentsReady
Gate	none. It only reads
Projects / Owns	nothing. It writes status.namespaceAssignments, one entry per spec.namespaceAssignments entry, in spec order
Requeue	namespaceAssignmentRequeueAfter = 1m while an entry is not Assigned

Per entry the step resolves the entry's cluster through ResolveChildrenClient and reads the assigned namespace there with one GET. A target cluster is read through its uncached reader, the management cluster through the cache. A pass costs at most 32 GETs and no LIST. The step never creates, labels or deletes an assigned namespace and writes nothing on any cluster, so the remote-children finalizer is not involved.

Each GET times out after namespaceAssignmentReadTimeout = 10s. A cluster that fails a GET with a transport error is not read again in the same pass: its later entries report ClusterUnreachable with the message namespace "<ns>" not read: <cluster> did not answer an earlier read in this pass: <err>. A cluster that accepts no connection therefore costs the pass one timeout in total. An API error such as Forbidden is read per entry, because the answer may differ per namespace.

No watch reaches an assigned namespace. The local Namespace watch filters on the ControlPlane's ownership labels, and a target cluster has no Namespace watch. A namespace created later, or a cluster registered later, shows up through the 1m requeue.

The step never returns an error. Each entry gets one of five reasons:

Entry reason	clusterReachable	namespaceExists	When
Assigned	true	true	the namespace exists
NamespaceNotFound	true	false	the cluster answers NotFound
NamespaceTerminating	true	true	the namespace carries a deletion timestamp
TargetClusterUnavailable	false	false	the cluster name does not resolve. The message is the resolver's text
ClusterUnreachable	false	false	the GET fails with any other error, for example Forbidden. The message is getting namespace "<ns>": <err>

The condition is decided over all entries:

Path	Status	Reason	Notes
no entry	True	NoNamespaceAssignments	clears status.namespaceAssignments
any entry TargetClusterUnavailable	False	TargetClusterUnavailable	the message lists every unresolved and unreachable entry as <namespace> on <cluster> and says that orders from these namespaces are refused until the cluster resolves or the entry is removed
else any entry ClusterUnreachable	False	ClusterUnreachable	the same listing
otherwise	True	NamespaceAssignmentsReady	the message counts the entries and, when one or more namespaces are missing, how many do not exist yet. An assignment may precede its namespace

A False condition turns the aggregate Ready False, also when the cluster only serves tenants. See NamespaceAssignmentsReady.

CredentialRotation reconciler

Aspect	Value
File	reconcile_credentialrotation.go
For()	CredentialRotation
Condition	Ready (conditionTypeRotationReady)
Owns / mints	nothing — it never mints
Requeue	credentialRotationWaitInterval = 10s while waiting for the ControlPlane reconciler or for a dependency to appear

The CredentialRotationReconciler drives one-shot rotations of a control-plane credential by nudging the owning reconciler rather than duplicating any mint logic. It dispatches on spec.target: adminApplicationCredential nudges the admin AC (clearing the AC's password-hash annotation), and serviceAccountPassword nudges the managed account of the referenced KeystoneService (clearing the managed User's cobaltcore.c5c3.io/password-generation annotation, which the KeystoneService controller's ensureManagedAccountUser consumes by bumping the password generation on its next pass). Its model:

• Nudge, never mint or delete. For the admin target it clears (zeroes) the cobaltcore.c5c3.io/admin-password-hash annotation on the owned AC CR via clearPasswordHashAnnotation (a no-op Update when already empty). On its next pass reconcileKORC observes the mismatch and performs the delete+recreate re-mint, re-stamping the fresh hash. The serviceAccountPassword target is the same discipline against the managed User: it resolves the KeystoneService named by spec.keystoneService in the CredentialRotation's own namespace (KeystoneServiceNotFound when it is absent), requires that CR's spec.account block (NoAccountDeclared otherwise), and resolves the ControlPlane through the registration's controlPlaneRef (ControlPlaneNotFound when that dangles). Each of the three is a Ready=False wait with the 10s requeue. Because that reference crosses namespaces and the caller controls it, the path then repeats all four pre-write gates the KeystoneService reconciler applies before it touches the account: the plane's registration consent for the CR's namespace (NamespaceNotAllowed — de-listing a namespace freezes its registrations, so the User is still there to nudge while the reconciler that would act on the nudge no longer runs), the role allowlist of the spec.namespaceAssignments entry for that namespace (RoleNotAllowed, which freezes the account the same way; the message ends with cannot rotate), the plane's AdminCredentialReady condition (WaitingForAdminCredential — K-ORC cannot reach Keystone before the admin credential is minted, which freezes the registration the same way), and the ownership labels on the User itself (ForeignServiceAccount — a cross-namespace child carries no owner reference, so the derived name is the only thing tying it to the registration). Each is refused before the nudge, so the generation stays unlatched and the rotation still fires once the gate opens. A CredentialRotation that predates spec.keystoneService decodes with an empty name and finishes Ready=False reason MissingKeystoneService with no requeue, naming the removed spec.serviceAccount field rather than reporting a dangling reference to a KeystoneService nobody wrote. Because there is no external password source to observe, the target has no auto-detect path, so a rotation fires only on an explicit reMint (latched to the spec generation). Keeping the resource lifecycle owned solely by the reconciler that owns the object (the ControlPlane reconciler for the admin AC, the KeystoneService reconciler for the managed account) avoids two controllers racing on the same object.
• reMint is one-shot per spec generation. An explicit spec.reMint is latched on status.lastTriggeredGeneration: the reconciler nudges only while it differs from metadata.generation, then records the generation. A reMint: true left in the spec therefore fires the nudge once per edit, not on every cache resync (~10 min via the shared SyncPeriod) or operator restart — without the latch it would revoke + re-mint the admin credential indefinitely, re-opening the stale-credential window each cycle. A pass over an already-latched generation reports NoRotationNeeded. The auto-detect (password-hash change) path is not latched: it is self-limiting (it stops once the hash matches) and relies on resync to observe an out-of-band password rotation.
• ControlPlane resolution (one-per-namespace). On the adminApplicationCredential target a CredentialRotation carries no explicit ControlPlane reference, so resolveControlPlane lists ControlPlanes in the CredentialRotation's own namespace and requires exactly one. Zero → Ready=False reason NoControlPlane with a short requeue; multiple → Ready=False reason AmbiguousControlPlane with no requeue (an arbitrary pick could rotate the wrong credential). The one-ControlPlane-per-namespace contract is now enforced at admission by the ControlPlane validating webhook (validateUniqueInNamespace), so the AmbiguousControlPlane branch is defense-in-depth: it is retained as a safety fallback but is unreachable while the webhook is active. The serviceAccountPassword target does not use this lookup. It reaches its ControlPlane through the referenced KeystoneService's controlPlaneRef, so it works from a namespace that holds no ControlPlane.
• Bootstrap is idempotent. With spec.bootstrap, an already-existing AC is a no-op success (BootstrapComplete); a missing AC waits (WaitingForBootstrap) for the ControlPlane reconciler to mint it.
• Scheduled fields are read-but-ignored. intervalDays / preRotationDays / gracePeriodDays are accepted but deferred to a later level; when set, an informational ScheduledRotationDeferred event is emitted but no loop runs and no error is raised.
• Target enum. adminApplicationCredential and serviceAccountPassword are supported; any other target finishes Ready=False reason UnsupportedTarget.

Path	Status	Reason	Notes
target neither adminApplicationCredential nor serviceAccountPassword	False	UnsupportedTarget	no requeue
no ControlPlane in namespace	False	NoControlPlane	requeue 10s
multiple ControlPlanes	False	AmbiguousControlPlane	no requeue; defense-in-depth — unreachable while the one-per-namespace webhook is active
ControlPlane List errors	False	ControlPlaneListError	no requeue
bootstrap, AC exists	True	BootstrapComplete	no-op success
bootstrap, AC absent	False	WaitingForBootstrap	requeue 10s
rotation, AC absent	False	WaitingForApplicationCredential	requeue 10s
admin password not yet readable	False	WaitingForAdminPassword	requeue 10s
hash unchanged, no pending reMint (incl. a reMint already latched for this generation)	True	NoRotationNeeded	nothing to do
nudge performed	True	RotationTriggered	emits RotationNudged event; an explicit reMint latches status.lastTriggeredGeneration

The CredentialRotation reconciler is registered with the manager via a plain For(&CredentialRotation{}) — it owns no children and registers no watches or field indexers.

---

K-ORC admin credential chain

The end-to-end path that delivers the admin application credential to the K-ORC controller spans four sub-reconcilers (reconcileAdminPassword, reconcileKORC, reconcileAdminCredential, reconcileCatalog) and the ESO/OpenBao backend.

The path passes OpenBao twice. The numbers in the figure are the steps below.

1. ESO reads property password of kv-v2/bootstrap/{keystone namespace}/{cp}-keystone/admin into the Secret {cp}-keystone-admin-credentials. reconcileAdminPassword creates the ExternalSecret in managed mode, owner-referenced to the ControlPlane. The Keystone namespace is the ControlPlane namespace unless spec.services.keystone.namespace is set.
2. reconcileKORC reads the password, writes the Secret {cp}-admin-password-cloud with a password-based clouds.yaml on every pass, and keeps the SHA-256 of the password for the cobaltcore.c5c3.io/admin-password-hash annotation. Until the password exists: KORCReady=False, reason WaitingForAdminPassword.
3. reconcileKORC writes the Secret {cp}-admin-app-credential. Its key value is the secret of the future application credential, 32 random bytes that the operator generates. Its key clouds.yaml is seeded with the password-based document while it is empty (seedBootstrapCloudsYAML).
4. ESO pushes the Secret whole through the PushSecret {cp}-admin-app-credential-backup to kv-v2/openstack/keystone/{ns}/{cp}/admin/app-credential. The deletion policy is Delete: the key leaves OpenBao with the ControlPlane.
5. ESO reads property clouds.yaml, and cacert when a CA bundle is set, into the Secret that spec.korc.adminCredential.cloudCredentialsRef.secretName names, by default k-orc-clouds-yaml, in the ControlPlane namespace (ensureKORCCloudsYAMLExternalSecret, refresh 1h, creationPolicy: Owner, owner-referenced to the ControlPlane). orc-system holds no copy: K-ORC resolves the cloudCredentialsRef of a resource in that resource's own namespace.
6. K-ORC resolves the unmanaged imports Domain {cp}-domain-default and User {cp}-user-admin with k-orc-clouds-yaml, which still carries the password.
7. K-ORC creates the ApplicationCredential {cp}-admin-app-credential in Keystone. It authenticates with {cp}-admin-password-cloud and passes value as the secret. restricted: true, the default, becomes unrestricted: false on the K-ORC resource. KORCReady turns True with reason ApplicationCredentialMinted.
8. reconcileAdminCredential takes the credential id from the ControlPlane status, overwrites key clouds.yaml of {cp}-admin-app-credential with the application-credential document, and stamps c5c3.io/push-content-hash on the PushSecret, because ESO does not watch the source Secret. Steps 4 and 5 run a second time. The operator force-syncs the ExternalSecret and sets AdminCredentialReady=True once the id and the secret in k-orc-clouds-yaml match.
9. K-ORC authenticates with the application credential and reconciles the catalog Service and Endpoint resources (reconcileCatalog).

Re-mint trigger. A rotation is signalled by comparing SHA-256(admin password) against the cobaltcore.c5c3.io/admin-password-hash annotation last stamped on the AC CR. reconcileKORC re-stamps and re-mints when they differ; the CredentialRotation reconciler forces the same path by clearing the annotation (which guarantees a mismatch). The admin-password Secret watch (see Secret Field Indexer) wakes the ControlPlane the moment the password rotates so the chain converges without waiting for the next periodic requeue. A change of restricted or accessRules re-mints as well, because K-ORC's spec.resource is immutable (adminACResourceDrifted in reconcileKORC).

---

Multi-instance

The ControlPlane reconciler scopes every admin / K-ORC credential it owns to the individual ControlPlane CR, so multiple control planes can coexist in a single cluster without sharing OpenBao state.

• One ControlPlane per namespace (admission-enforced). The ControlPlane validating webhook's validateUniqueInNamespace check runs in ValidateCreate only (not ValidateUpdate): it lists ControlPlanes in the object's namespace through the uncached API reader and returns field.Forbidden naming the incumbent if one already exists. The cluster therefore admits exactly one ControlPlane per namespace. This is what makes the CredentialRotation reconciler's AmbiguousControlPlane branch defense-in-depth (see CredentialRotation reconciler).
• Duplicate guard (reconciler-enforced). As defense-in-depth for CRs that predate the webhook guard, raced through the API server, or were written with the webhook bypassed, Reconcile runs duplicateControlPlaneIncumbent before the sub-reconciler chain: it lists ControlPlanes in the CR's namespace and parks every CR except the oldest (by creationTimestamp, lexically smallest name breaking ties). A parked duplicate gets Ready=False with reason DuplicateControlPlane naming the incumbent, runs no sub-reconcilers, and requeues every 30s — so it takes over automatically once the incumbent is fully deleted (no watch event fires on the duplicate's behalf when that happens).
• Per-CR OpenBao path for the admin AC. The admin application credential is pushed to the per-ControlPlane key openstack/keystone/{cp.Namespace}/{cp.Name}/admin/app-credential (adminAppCredentialRemoteKeyFor), so two ControlPlanes never write to the same OpenBao object. The K-ORC admin User CR is named {cp.Name}-user-admin (its Kubernetes metadata.name), while the OpenStack username it imports stays admin (set via the import Filter.Name) — the K8s-name and the OpenStack-name are deliberately split so the per-CR Kubernetes object is unique while the OpenStack identity is unchanged.
• Per-CR OpenBao path for the service DB credential. The managed-mode service database credential is read from the per-ControlPlane key openstack/keystone/{cp.Namespace}/{cp.Name}/db (dbCredentialRemoteKeyFor), scoped by both namespace and name (mirroring adminAppCredentialRemoteKeyFor) so two ControlPlanes never collide on the cluster-global OpenBao backend. reconcileDBCredentials projects an owned ExternalSecret reading username/password from this key.
• Running multiple control planes. Because the webhook caps a namespace at one ControlPlane and the OpenBao paths are keyed by {namespace}/{name}, operating several control planes means deploying each into its own namespace. Each gets a disjoint OpenBao prefix (openstack/keystone/{namespace}/{name}/admin/app-credential), disjoint child CRs in its own namespace, and an independent rotation lifecycle — no two control planes can clobber one another's credentials.

See Migration: legacy flat paths → per-ControlPlane paths for moving an existing single-instance cluster onto the per-CR layout.

---

Owner-ref / GC model

A child CR created in the ControlPlane's own namespace carries an owner reference to the ControlPlane CR via controllerutil.SetControllerReference(). This enables both automatic garbage collection (deleting the ControlPlane cascades to its children) and watch-based reconciliation (a child change re-reconciles the owner).

A child created in a service namespace — one a namespace assignment places elsewhere — can carry no owner reference: Kubernetes garbage collection only cascades within one namespace, so the API server rejects a cross-namespace controller reference. Such a child is stamped with two ownership labels instead — c5c3.io/controlplane-name and c5c3.io/controlplane-namespace, which together name the owning ControlPlane. They carry the two jobs an owner reference would have done: isControlPlaneChild recognizes the child (so a colliding object owned by nobody is never reshaped or deleted), and a label-mapped Watches() leg resolves an event on it back to a reconcile request. Because nothing garbage-collects such a child, the finalizer tears it down explicitly (see below).

A child written to a target cluster carries no owner reference either, for a stronger reason: that cluster's garbage collector cannot resolve an owner living on the management cluster at all. Such a child takes the two labels above plus the three shared ownership labels (openstack.c5c3.io/owner-kind, -owner-name, -owner-namespace), which name the owning CR across operators, so a Keystone and a ControlPlane projecting into one target namespace each select only their own. Nothing on that cluster collects them, and no cascade crosses the boundary, so a second finalizer holds the ControlPlane in etcd until they are swept.

The figure shows the three cases side by side.

Deletion ordering — the c5c3.io/orc-teardown finalizer

Owner-reference GC alone is unordered: deleting the ControlPlane would garbage-collect every child at once. That is unsafe for the K-ORC CRs the operator owns (ApplicationCredential, Service, Endpoint, Region, User, Domain). Those CRs carry K-ORC finalizers that call the Keystone API to revoke/delete the credentials and catalog entries they minted; if Keystone (and in managed mode its MariaDB) were torn down concurrently, the K-ORC finalizers could never complete and the ControlPlane — and its namespace — would hang indefinitely on Terminating ORC CRs.

The ControlPlane reconciler therefore installs a single finalizer, c5c3.io/orc-teardown, added on the first reconcile before any K-ORC CR is projected. On deletion it:

1. Deletes the projected KeystoneService registrations first and waits for them to go. Their K-ORC CRs belong to the registration, not to the ControlPlane, so no enumeration in the sweep below names them, and their controller tears them down through the very admin credential step 2 revokes. While one is still Terminating the reconciler reports KORCReady=False with reason FinalizingServiceRegistrations and requeues; past registrationTeardownStallTimeout (2 minutes) it continues anyway and a Warning ServiceRegistrationTeardownStalled names what was left behind. That window is a budget of its own rather than a share of the K-ORC one: this wait blocks the sweep below entirely, so a shared deadline would let an unreachable Keystone spend it all here and leave the admin credential's revocation with no time at all. A registration still standing when the ControlPlane is about to be released, on either path, has the children it owns — matched by both their ownership labels and their name prefix — deleted first, and the openstack.k-orc.cloud/* finalizers of the K-ORC ones stripped afterwards, so the owning controller cannot re-add what it just lost. Its PushSecrets carry DeletionPolicy=Delete, so their deletion is left with ESO to purge from OpenBao until orcTeardownDeadline — reported as KORCReady=False with reason FinalizingORC while the wait runs — and only past it are their finalizers stripped too. That wait is entered only while the tenant SecretStore the purge authenticates through is still standing: for a registration in a dedicated namespace the namespace sweep above already took it (Managed: the whole namespace), so the finalizers are stripped at once rather than holding the namespace Terminating for a purge ESO can no longer run. A WarningServiceRegistrationResourcesOrphaned then names the OpenStack resources and the OpenBao paths left behind (unmanaged imports are released without being listed, the same line step 9 draws); a release that abandoned neither — every child was still live, or held nothing yet — reports a NormalServiceRegistrationChildrenReleased instead of a Warning naming empty sets. With its children gone the registration's own controller releases the KeystoneService CR.

2. Deletes the owned K-ORC CRs and holds the ControlPlane CR in etcd. Holding the CR defers the owner-reference GC cascade, so Keystone stays reachable while K-ORC revokes. While ORC CRs are still Terminating the reconciler reports KORCReady=False with reason FinalizingORC and requeues at the K-ORC cadence.

3. Deletes the owned PushSecrets alongside them. Their deletionPolicy: Delete cleanup — ESO deleting the mirrored OpenBao data — needs the per-tenant SecretStore and its eso-tenant-auth ServiceAccount, both of which the post-release GC cascade reaps unsequenced. Deleting the PushSecrets while the finalizer still holds that infrastructure is what keeps the revoked credential from outliving the ControlPlane in OpenBao. A PushSecret still stuck past the stall window has its finalizers force-removed, and a Warning OpenBaoCleanupStalled names the OpenBao paths that may retain data. A placed namespace is swept on its target cluster as well as at home, because both the PushSecrets and the store their purge authenticates through live there; the force-remove goes to the cluster the PushSecret was found on.

4. Tears down the cross-namespace children before releasing. A service placed in a namespace of its own — and the backing services, tenant store, and credential material that follow it — carries no owner reference, so no GC cascade reaches it; releasing the finalizer first would strand every one of them. teardownDedicatedNamespaces deletes them by hand, in order: the service children (Keystone/Horizon, and for a Barbican placed apart its Barbican, BarbicanSecretStore and OpenBaoCluster together) first, waiting for them (their operators run a sequenced ESO cleanup through the tenant store in the same namespace), then the namespace per its lifecycle. The OpenBao instance belongs in that wait set because its own finalizer runs under the tenant RBAC living in the same namespace: deleting the namespace first reaps that RBAC out from under it and leaves the instance unfinalizable, with the namespace stuck Terminating. The service CRs live on the management cluster wherever their service is placed, so waiting for them here is also what waits out the service operators' own remote sweeps. A Managed namespace is deleted, which cascades everything left in it — but only when it carries the ownership marks, read through that cluster's uncached reader because the verdict authorises a whole-namespace cascade; one that does not is left standing with a Warning NamespaceNotOwned, because the operator never destroys a namespace it did not create. On a target cluster the c5c3.io/controlplane-uid annotation is read with them, so a namespace whose mark names another management cluster's ControlPlane is refused rather than cascaded away — while one whose mark was stripped is still reaped, unlike at adoption, since nothing else ever comes back for it. A placed one is deleted on both clusters, since reconcileNamespaces created it on both. An External namespace survives, so its residue (backing services, credential material, tenant-store trio last) is swept by name, each object ownership-checked so a same-named object belonging to somebody else in that shared namespace is left alone. On a placed namespace both run against that cluster's client. The label-selected sweep runs after the External residue sweep and before a Managed namespace is deleted. While children remain the condition reports NamespacesReady=False/FinalizingNamespaces; past the orcTeardownDeadline the sweep stops waiting, emits a WarningNamespaceTeardownStalled naming what is stuck, and releases anyway — a wedged child must not make a namespace undeletable forever.

5. Removes the auth-delegator ClusterRoleBinding by hand. A dedicated Barbican secret store places one cluster-scoped object, {instance}-{hash}-auth-delegator. A namespaced owner cannot own it, so no GC cascade collects it, and the per-namespace sweep in step 3 reaches it only when Barbican was placed in a namespace of its own. deleteBarbicanAuthDelegatorBinding therefore runs on every ControlPlane teardown, Barbican or not, reading through the uncached API reader (a cached read would install a cluster-wide ClusterRoleBinding informer for one object) and deleting only a binding carrying this ControlPlane's ownership labels. It reads and deletes on the cluster Barbican was placed on, where the ensemble wrote the binding. A delete that cluster denies — its access chart grants the write verbs only behind authDelegatorBinding — leaves the binding standing with a WarningAuthDelegatorBindingNotReclaimed naming it, rather than holding the ControlPlane in Terminating for a grant the target withdrew.

6. Deletes the co-located Keystone and the managed message bus by hand, then releases the finalizers once the ORC CRs, PushSecrets, cross-namespace children, the Keystone and the bus are gone, letting GC cascade-delete the infrastructure and the remaining children. Both deletes run in the same pass, so the Keystone wait cannot spend the bus's deadline.

   A Keystone in the ControlPlane's own namespace owns two backup PushSecrets with deletionPolicy: Delete, <keystone>-fernet-keys-backup and <keystone>-credential-keys-backup. Its openbao-finalizer holds it until ESO has purged their OpenBao paths, and that purge authenticates through the per-tenant SecretStore and its eso-tenant-auth ServiceAccount. The cascade deletes all of them at once and in no order. deleteColocatedKeystoneBeforeRelease therefore deletes the Keystone itself, with background propagation, and holds the finalizer until it has left etcd, reported as KeystoneReady=False/FinalizingKeystone with a 15-second requeue. The delete waits for the K-ORC CRs to go first, because K-ORC revokes and deletes through the Keystone API. Only a Keystone this ControlPlane owns is deleted. External mode projects none, and a cluster without the Keystone CRD has none, so neither waits; a Keystone in a dedicated namespace belongs to step 4. Past orcTeardownDeadline (7 minutes from the deletion timestamp, the deadline the steps before it share) a Warning KeystoneTeardownStalled names the Keystone and its paths openstack/keystone/<namespace>/<keystone>/fernet-keys and openstack/keystone/<namespace>/<keystone>/credential-keys, and the release proceeds on the cascade. Those paths may then keep their data and have to be deleted by hand.

   The RabbitmqCluster is the other same-namespace child the cascade is not trusted with: GC deletes with background propagation, so the CR vanishes the instant the RabbitMQ Cluster Operator removes its finalizer, and that operator's deletion path (v2.13.0 and later, rabbitmq/cluster-operator#1864) removes the finalizer through controllerutil.CreateOrUpdate. A second reconcile of the deleting CR, queued by the first one's own pod-label and StatefulSet writes, reads NotFound from its cache and creates a fresh, unowned broker under the same name, which nothing collects. deleteManagedMessagingBeforeRelease therefore deletes the owned bus itself with foreground propagation, which keeps the CR visible until every child the cluster-operator owns is gone, and holds the finalizer until the cluster-operator's own finalizer is off the Terminating CR (or the CR is gone), reported as InfrastructureReady=False/FinalizingMessaging. The CR then lingers under foregroundDeletion only until GC has reaped its children; waiting for NotFound instead would wait on a garbage collector envtest does not run. A bus this ControlPlane does not control (an adopted, externally-provisioned one) is left standing, and a brownfield secretRef names nothing to delete. Past messagingTeardownDeadline (3 minutes from the deletion timestamp) a Warning MessagingTeardownStalled names the broker and the release proceeds on the cascade, so a wedged cluster-operator cannot make the ControlPlane undeletable. The escape paths below release without either wait. The remote-children finalizer goes in the same update, because by then every placed namespace has been swept or its cluster abandoned.

7. Releases a CR-only remainder immediately. K-ORC re-fetches the resource through an authenticated actuator before releasing any finalizer, and those CRs authenticate with the admin application credential whose revocation step 1 already triggered. Once every CR still present is an Unmanaged import or a detach-on-delete resource (the adopted Region), waiting on K-ORC is waiting on a dead-credential retry loop. The reconciler force-removes their openstack.k-orc.cloud/* finalizers right away and emits a Normal ORCImportsReleased event. Neither class owns the OpenStack resource behind it: an import's deletion is CR-only, and a detach resource is released by policy, so the external installation is untouched and nothing is orphaned.

8. Bounds the wait. If managed ORC CRs stay Terminating past orcTeardownDeadline (the 2-minute registration window plus orcTeardownStallTimeout, 5 minutes) — typically because Keystone is already gone and K-ORC cannot revoke — the reconciler force-removes the stuck openstack.k-orc.cloud/* finalizers (preserving any non-K-ORC finalizers), emits a Warning ORCTeardownStalled event, and releases the ControlPlane finalizer so deletion completes rather than wedging forever. A projected registration still present here has its children force-released before that finalizer goes, under the same WarningServiceRegistrationResourcesOrphaned as on the normal release.

9. Names what the escape orphaned. The escape strips the very finalizer that would have revoked the credential or removed the catalog row, so every Managed CR it releases leaves its OpenStack resource behind with no Kubernetes object naming it. A second Warning, ORCResourcesOrphaned, lists exactly those CRs — the admin ApplicationCredential among them — and tells the operator to remove them from Keystone by hand. Unmanaged imports are never listed: their CR delete could not have touched OpenStack. The detach Region is never listed either, because onDelete: detach leaves the Keystone row in place. The classification is by ManagementPolicy, with detach as its one exception, and fails loud (anything else not explicitly Unmanaged is reported), because under-reporting a leak is worse than over-reporting one.

kubectl delete namespace makes the leak deterministic

Children live in the ControlPlane's own namespace, so the namespace controller reaps {name}-admin-password-cloud — the Secret the managed catalog entries authenticate with, chosen precisely so they can still authenticate during teardown — concurrently with the entry CRs themselves. K-ORC then has no credential to delete the rows with, the CRs stall, and after 5 minutes the escape releases them. Delete the ControlPlane CR and let it converge before deleting its namespace.

This mirrors the Keystone reconciler's sequenced-finalizer discipline (MariaDB then OpenBao cleanup); see Keystone reconciler — finalizer. The {name}-admin-app-credential-backup PushSecret carries DeletionPolicy: Delete like the others, so its OpenBao path is purged on teardown.

The placed namespaces — openstack.c5c3.io/remote-children

A ControlPlane that places a service on a target cluster carries a second finalizer, the shared openstack.c5c3.io/remote-children. It goes on once ANY cluster the spec names resolves, because reconcileNamespaces resolves and writes per namespace inside its loop, so that cluster's namespaces are created on the very same pass whatever a sibling ref does. Demanding every cluster would leave the written half unreclaimable when the ORC stall escape releases the other finalizer. While no cluster resolves, nothing has been written to any of them for it to reclaim, and reconcileNamespaces is what reports the unresolvable name. Once installed it stays for the CR's life: a cluster that stops resolving later still holds children.

What it holds the ControlPlane open for is the label-selected sweep in step 4. Per placed namespace, controlPlaneRemoteChildKinds names the fourteen kinds the ControlPlane writes there: MariaDB, Memcached, SecretStore, Certificate, ServiceAccount, Role, RoleBinding, Secret, ExternalSecret, PushSecret, VaultDynamicSecret, Password, OpenBaoTenant, OpenBaoCluster. Every object of them the ControlPlane owns is deleted through that cluster's client, listed through its uncached reader and paged so a shared namespace cannot arrive in one response. The list holds namespaced kinds only. The auth-delegator ClusterRoleBinding and the namespace itself are cluster-scoped and deleted by name (steps 5 and 4), while the service CRs and the K-ORC CRs never leave the management cluster.

The sweep runs through the credentials of the registered cluster's kubeconfig, never through the management ClusterRole. That is why Role and RoleBinding are swept although the markers grant them no list verb at home, and it means the blast radius of a placed ControlPlane is whatever that kubeconfig grants on its cluster.

A cluster that does not resolve while the ControlPlane is terminating is waited for: the pass requeues with both finalizers on and reports NamespacesReady=False/TargetClusterUnavailable. Engagement is asynchronous, so an operator restart is indistinguishable from a deregistration. After AbandonAfter (5 minutes, measured from both the first failed resolve in this process and the deletion timestamp) the cluster is abandoned: a WarningRemoteChildrenAbandoned names the cluster and the namespace whose objects stay behind, and the teardown continues without it — the Managed namespace's copy on the management cluster is still deleted, since abandoning the unreachable half does not license leaking the reachable one. The ORC stall escape (step 8) releases c5c3.io/orc-teardown alone. It never reaches this sweep, so the remote-children finalizer stays on and a later pass runs the sweep and releases it.

ControlPlane-scoped children live in the owner's namespace; service children follow their service. The K-ORC CRs, the clouds.yaml Secret, the PushSecret, and the service-account material belong to the ControlPlane as a whole and are created in childNamespace(cp) = cp.Namespace, owner-referenced. A service and the things that follow it (its MariaDB, Memcached, Keystone/Horizon, tenant store, and credential material) are placed in cp.KeystoneNamespace() / cp.HorizonNamespace() — the ControlPlane's own namespace by default, or the one a namespace assignment gives it. A cross-namespace owner reference is rejected at admission because Kubernetes GC only cascades within a single namespace, so a service child in another namespace carries the ownership labels and is torn down by the finalizer instead of the GC cascade.

Resource	Name	Owner	Notes
MariaDB	{spec.infrastructure.database.clusterRef.name}	ControlPlane CR	managed mode only
Memcached (unstructured)	{spec.infrastructure.cache.clusterRef.name}	ControlPlane CR	managed mode only
ExternalSecret (DB credential)	{name}-keystone-db-credentials	ControlPlane CR	managed mode only; ESO owns the materialised Secret of the same name
ExternalSecret (admin password)	{name}-keystone-admin-credentials	ControlPlane CR	managed mode only; ESO owns the materialised Secret of the same name
Keystone	{name}-keystone	ControlPlane CR	managed mode only
ApplicationCredential	{name}-admin-app-credential	ControlPlane CR	both modes; carries cobaltcore.c5c3.io/admin-password-hash
Secret	{name}-admin-app-credential	ControlPlane CR	both modes; value generated and clouds.yaml assembled by the operator, read by K-ORC
PushSecret	{name}-admin-app-credential-backup	ControlPlane CR	both modes; DeletionPolicy: Delete
User (K-ORC)	{name}-user-admin	ControlPlane CR	both modes; unmanaged import
Domain (K-ORC)	{name}-domain-default	ControlPlane CR	both modes; unmanaged import
Service (K-ORC)	{name}-identity-service	ControlPlane CR	both modes; managed catalog entry in Managed mode, unmanaged import in External mode
Endpoint (K-ORC)	{name}-identity-endpoint	ControlPlane CR	managed mode only; public interface
Endpoint (K-ORC)	{name}-identity-endpoint-{interface}	ControlPlane CR	External mode only; one unmanaged import per interface (public, internal, admin)

Barbican secret-store teardown

Unsetting spec.services.barbican takes a different path from deleting the ControlPlane, and deleteOrphanedBarbican gates it on two annotations rather than one. c5c3.io/allow-barbican-deletion: "true" releases the BarbicanSecretStore (first, so its controller can still reach the server while it finalizes), then the Barbican child, the DB-credential objects, and the key-manager catalog CRs. Reaching the dedicated OpenBao instance takes c5c3.io/allow-barbican-secret-store-data-deletion: "true" on top: the instance carries deletionPolicy: DeletePVCs, so the delete wipes the raft volume, and the teardown removes the static-seal Secret with it, leaving even a recovered volume unreadable. Without the second annotation the instance, its PVC, its seal key and the rest of the ensemble stand, and BarbicanReady says so, so re-declaring the service reattaches to the same instance.

It does not reattach to the same secrets. The first annotation already deletes the Barbican child, whose finalizer deletes the MariaDB Database CR, and database.BuildDatabase sets no cleanupPolicy, so mariadb-operator applies its Delete default and drops the schema. The secret and container rows, secret_store_metadata, the ACLs and the quotas go with it, and db-sync recreates the schema empty on the next projection, leaving the surviving OpenBao payloads unreferenced. A dump of the barbican schema taken before the teardown is the only thing that makes the preserve branch a round trip.

With both set the order is store, child, credentials, catalog rows, then the OpenBaoCluster. The sweep then waits for the instance to leave etcd before deleting the OpenBaoTenant and the rest of the ensemble. The tenant is what admits the namespace to the openbao-operator, and the instance's own finalizer runs under that admission, so removing it first would leave the instance unfinalizable. Each object is ownership-checked, and an object already gone is tolerated.

Both annotations are read from the live CR on every reconcile of that sweep, so the decision is not latched. Removing one while the instance is still finalizing resumes on the preserve branch after the OpenBaoCluster is already deleted, and the ensemble behind it is never swept. The cluster-scoped auth-delegator binding is the object that then has nothing left to collect it.

Deleting the ControlPlane takes the ensemble down either way, with no second opt-in: tearing down a whole control plane is already an explicit destructive act.

Neutron teardown residue

The network service leaves objects in three shapes, and both teardown paths name them. deleteOrphanedNeutron runs when spec.services.neutron is unset with c5c3.io/allow-neutron-deletion: "true"; the ControlPlane teardown reaches the same set through sweepExternalNamespaceResidue for a namespace it does not own, through crossNamespaceServiceChildren for the Neutron child in a dedicated namespace, and through projectedRegistrationKeys for the registration.

• The four DB-credential shapes: the {controlplane.Name}-neutron-db-credentialsExternalSecret and the VaultDynamicSecret generator of the same name, the {controlplane.Name}-neutron-db-openbao-client Certificate, and the neutron-db-creds ServiceAccount.
• The bus delivery: the {controlplane.Name}-neutron-messaging Secret and the {controlplane.Name}-neutron-messaging-ca mirror. Both paths take the two stubs from serviceMessagingSecrets(neutronMessagingTarget(cp)). Nothing else writes them, so an unmanaged service leaves no broker credential behind in the namespace.
• The Neutron child and the {controlplane.Name}-neutron KeystoneService registration, whose finalizer removes the network catalog rows, the service user, and its project.

Each object is ownership-checked against its live state, so a same-named object belonging to somebody else in a shared namespace is left alone. The OVNCentral the child references appears in neither path and is deleted nowhere: it is deployed outside the plane and only read.

Cinder teardown residue

The block-storage service leaves the shapes the network service does plus its two satellite kinds, and both teardown paths name them. deleteOrphanedCinder runs when spec.services.cinder is unset with c5c3.io/allow-cinder-deletion: "true"; the ControlPlane teardown reaches the set through sweepExternalNamespaceResidue for a namespace it does not own, through crossNamespaceServiceChildren for the Cinder child in a dedicated namespace, and through projectedRegistrationKeys for the registration.

• The four DB-credential shapes: the {controlplane.Name}-cinder-db-credentialsExternalSecret and the VaultDynamicSecret generator of the same name, the {controlplane.Name}-cinder-db-openbao-client Certificate, and the cinder-db-creds ServiceAccount.
• The bus delivery: the {controlplane.Name}-cinder-messaging Secret and the {controlplane.Name}-cinder-messaging-ca mirror. Both paths take the two stubs from serviceMessagingSecrets(cinderMessagingTarget(cp)). Nothing else writes them, so an unmanaged service leaves no broker credential behind in the namespace.
• Every owned CinderBackend and CinderBackupBackend in the Cinder namespace, swept kind by kind on ownership alone, since a satellite carries a bare entry name no prefix selects. The teardown waits on them: a deleted CinderBackend holds the cinder operator's service-remove finalizer until its detach Job has run, so the namespace must not be taken out from under that Job.
• The Cinder child and the {controlplane.Name}-cinder KeystoneService registration, whose finalizer removes the block-storage catalog rows, the service user, and its project.

Each object is ownership-checked against its live state, so a hand-created CinderBackend attached to the same Cinder is neither pruned nor swept.

Nova teardown residue

The compute service leaves the shapes its peers do, twice over for the credentials and with the generated metadata secret on top. deleteOrphanedNova runs when spec.services.nova is unset with c5c3.io/allow-nova-deletion: "true"; the ControlPlane teardown reaches the same set through sweepExternalNamespaceResidue for a namespace it does not own, through crossNamespaceServiceChildren for the Nova child in a dedicated namespace, and through projectedRegistrationKeys for its registration keys.

• Two DB-credential chains rather than one, because the nova_api and cell schemas each take a login of their own: the {controlplane.Name}-nova-api-db-credentials and {controlplane.Name}-nova-db-credentials ExternalSecrets, the VaultDynamicSecret generators of the same names, the {controlplane.Name}-nova-api-db-openbao-client and {controlplane.Name}-nova-db-openbao-client Certificates, and the nova-api-db-creds and nova-cell-db-creds ServiceAccounts.
• The generated metadata shared secret: the {controlplane.Name}-nova-metadata-secret ExternalSecret and the Password generator behind it, the latter read uncached. The Secret they materialise carries ESO's own owner reference, so it comes down with the ExternalSecret rather than being deleted by name.
• The bus delivery: the {controlplane.Name}-nova-messaging Secret and the {controlplane.Name}-nova-messaging-ca mirror, both from serviceMessagingSecrets(novaMessagingTarget(cp)), and the {controlplane.Name}-nova-remote-messaging Secret the remote compute contract reads its transport URL from.
• The Nova child and the {controlplane.Name}-nova KeystoneService registration, whose finalizer removes the compute catalog rows, the service user, and its project.
• The hypervisor operator's account: the {controlplane.Name}-nova-hypervisor-operatorKeystoneService registration, which projectedRegistrationKeys always lists so a registration the dropped block left behind still comes down with the plane, and the {controlplane.Name}-nova-hypervisor-operator-auth Secret. Its copies on compute clusters are reaped by each NovaCompute's own teardown, as the compute-contract mirrors are.

The compute-notifier registration {controlplane.Name}-neutron-nova is not in this set: it lives in the Neutron namespace beside the service that authenticates as it, so deleteOrphanedNeutron and the Neutron teardown sweep it (and pruneNeutronNovaNotifierRegistration takes it down once services.nova is cleared while the network service stays and the Neutron child has converged on the spec without spec.nova).

The credential mirror a placed compute service carries is not swept by deleteOrphanedNova either. Like every object that function names it is resolved through NovaNamespace(), which without a services.nova block is the ControlPlane's own namespace, so that sweep reaches co-located objects only; the mirror is reaped by the ControlPlane teardown, which sweeps a placed namespace's label-owned ExternalSecrets on the target cluster.

External-mode deletion resource set

orcChildObjects(cp) derives the swept CR names from the ControlPlane spec, so Managed mode enumerates the six CRs it owns and External mode adds the per-interface identity Endpoint imports. A name that never existed in the current mode is simply NotFound and is tolerated as already-gone.

The enumeration is purely spec-derived, and it can be: every name it produces follows from the ControlPlane's identity and its keystone mode, neither of which a spec edit can drop while leaving a CR behind. Rows a KeystoneService registers are not in this set at all — that CR owns them, and reconcileDelete deletes the projected registrations first and waits for their own teardown.

What a Delete does to the external OpenStack installation is decided by each K-ORC CR's ManagementPolicy, not by the ControlPlane's mode:

• ApplicationCredential — Managed. Its K-ORC finalizer revokes the credential at the Keystone level before the CR delete returns, so authenticating with it immediately afterwards yields 404 Could not find Application Credential (not 401). This is the one identity object the operator minted, so it is the one it destroys.
• User, Domain — Unmanaged imports. Deleting their CRs removes the Kubernetes objects and leaves the OpenStack resources they imported untouched. K-ORC's deletion-guard finalizers also enforce the teardown order: a User cannot go while an ApplicationCredential still references it.
• Service, Endpoint — in Managed mode these are the managed catalog entries, so the sweep deletes them from Keystone's catalog. In External mode the identity Service and its per-interface Endpoints are Unmanaged imports, so deleting them is a CR-only delete and the external catalog is left bit-for-bit intact.
• Region — Managed with onDelete: detach, so a Delete removes the CR and leaves the Keystone row in place; that row goes with the Keystone database. K-ORC still authenticates before releasing the detach finalizer, which is why reconcileDelete releases it together with the unmanaged imports. External mode projects no Region, so the name is NotFound there. That holds for a teardown K-ORC can complete. The stall escape is the deliberate exception: past orcTeardownDeadline it releases every stuck CR by stripping the finalizer that would have done the revoke or the DELETE, so each Managed CR it releases orphans its OpenStack resource. Those are the CRs the ORCResourcesOrphaned Warning names.

The OpenBao-backed Secrets are torn down by owner-reference GC, and the path behind the {name}-admin-app-credential-backup PushSecret is purged with it (DeletionPolicy: Delete). Nothing else is touched — a K-ORC CR the ControlPlane does not own is never swept.

Security invariant

The admin password and the minted application-credential Secret are read only by the c5c3-operator and the K-ORC controller pods — they are never mounted into Keystone or any OpenStack service workload. Keystone receives the admin password solely through its own bootstrap Secret ref for the one-time keystone-manage bootstrap; the long-lived application credential lives exclusively on the c5c3↔K-ORC↔OpenBao path. restricted: true (the default) further bounds the blast radius by scoping the minted credential. These invariants are enforced by the credential_invariant_test.go checks (TestCredentialInvariant_MintedACIsRestricted, TestCredentialInvariant_AppCredentialSecretAbsentFromKeystoneSpec, TestCredentialInvariant_AppCredentialSecretReferencedOnlyByPushSecretAndAC, TestCredentialInvariant_NoWorkloadReferencesAppCredentialSecret).

The PushSecret carries DeletionPolicy: Delete. The credential leaves OpenBao with the ControlPlane. The teardown revokes it in Keystone, so a value kept at the path would be dead, and the next ControlPlane of the same name would authenticate K-ORC with it and fail. A re-created control plane mints a new credential.

---

Metrics Instrumentation

Every sub-reconciler invocation is instrumented for Prometheus via a single package-scope instrumenter, declared in operators/c5c3/internal/controller/instrumentation.go. Its Instrument method comes from the shared internal/common/instrumentation package — the duration/error metric pair and the wrapper logic are identical across all CobaltCore operators and live there; the c5c3 file supplies only the c5c3_operator prefix and the subReconcilerConditionTypes map. Reconcile wraps every sub-reconciler call with it; a direct call that bypasses it is a contract violation.

func (i *Instrumenter) Instrument(
    ctx  context.Context,
    name string,
    fn   func(context.Context) (ctrl.Result, error),
) (ctrl.Result, error)

Behavioural contract:

• Always records one observation in c5c3_operator_reconcile_duration_seconds{sub_reconciler=name} via defer — on the success path, the error path, and even when fn panics (the deferred call runs before the stack unwinds).
• Only increments c5c3_operator_reconcile_errors_total{sub_reconciler=name, condition_type=…} when fn returns a non-nil error.
• Does not recover from panics — they propagate to the caller.
• Carries no per-CR labels (no controlplane / namespace). The two label dimensions (sub_reconciler, and condition_type on the error counter) are bounded by the number of sub-reconcilers, keeping the series count fleet-independent. Per-CR collectors are intentionally out of scope.

Both vectors are registered exactly once on the controller-runtime registry via sync.Once; the histogram buckets are a fixed contract (0.01 … 30s).

Name → condition_type lookup and the drift guard

The condition_type label is resolved from the package-private subReconcilerConditionTypes map in instrumentation.go:

sub_reconciler	condition_type
Sizing	SizingReady
Namespaces	NamespacesReady
Infrastructure	InfrastructureReady
ESOTenantStore	ESOTenantStoreReady
DBCredentials	DBCredentialsReady
Keystone	KeystoneReady
Horizon	HorizonReady
Glance	GlanceReady
Placement	PlacementReady
Barbican	BarbicanReady
OVN	OVNReady
Neutron	NeutronReady
Cinder	CinderReady
Nova	NovaReady
KORC	KORCReady
AdminCredential	AdminCredentialReady
AdminPassword	AdminPasswordReady
Catalog	CatalogReady
ServiceAccounts	ServiceAccountsReady
RegistrationTenantStores	RegistrationTenantStoresReady
NamespaceAssignments	NamespaceAssignmentsReady

The map carries two further entries, KeystoneServiceCatalog and KeystoneServiceAccount, which belong to the KeystoneService controller rather than to this one; see the KeystoneService Reconciler. The RegistrationTenantStores series is kept apart from ESOTenantStore's because the two carry different blast radii and one alert should not read as the other.

If instrumenter.Instrument is ever called with a name absent from the map, the helper emits the sentinel condition_type=UNKNOWN (instrumentation.ConditionTypeUnknown) rather than an empty label, so any drift is visible in dashboards/alerts. One static drift guard keeps the map honest: TestSubReconcilerConditionTypesCoversAllNames asserts that every mapped condition_type is a member of subConditionTypes. Adding a new sub-reconciler therefore requires updating subConditionTypes andsubReconcilerConditionTypes; an unmapped name is not caught by CI and surfaces only as condition_type=UNKNOWN in the metrics.

---

Testing

The reconcilers have comprehensive unit tests using the controller-runtime fake client with gomega (NewGomegaWithT(t)), plus envtest integration tests that drive the full chain in a real manager against a live API server. They cover the full reconcile to Ready=True and two deletion scenarios, TestIntegration_ControlPlaneDeletion_SweepsProjectedRegistrationsFirst and TestIntegration_DedicatedNamespaces.

Running Tests

Scope	Command
All controller unit tests	go test ./operators/c5c3/internal/controller/...
Integration (envtest)	go test -tags integration -run TestIntegration_FullReconcile_ManagedToReady ./operators/c5c3/internal/controller/

Integration test

TestIntegration_FullReconcile_ManagedToReady (integration_test.go, build tag integration) registers the real controller wiring (the inline builder is kept byte-for-byte in step with SetupWithManager) and drives a managed-mode ControlPlane through every sub-reconciler to the aggregate Ready=True. It simulates each external dependency's readiness in dependency order — MariaDB and Memcached Ready → the operator-created admin-password ExternalSecret synced → Keystone child Ready → K-ORC ApplicationCredential Available with a status.id → the {control-plane ns}/k-orc-clouds-yaml ExternalSecret synced — and asserts that every sub-condition and the aggregate Ready (reason AllReady) reach True, that status.observedGeneration and every condition's ObservedGeneration match the CR generation, and that status.adminApplicationCredential mirrors the simulated AC. Beyond the aggregate condition it also asserts the intermediate projected specs so a projection regression is caught: the Keystone image tag derived from openStackRelease, the database/cache clusterRefs wired to the infra CRs, the merged policyOverrides, the restricted→Unrestricted=false inversion on the AC, and the identity Service/Endpoint shape.

A phase between Infrastructure and Keystone exercises the new admin-password projection: it waits for the operator-created per-CP admin-password ExternalSecret, asserts its RemoteRef.Key equals adminPasswordRemoteKeyFor(cp) and that it is controller-owned by the ControlPlane, then simulates the ESO sync — SimulateExternalSecretSync patches only the ExternalSecret's status, so the renamed plain Secret (pre-created under the same name) stays the cleartext source the operator reads — and waits for AdminPasswordReady to reach True before the Keystone child is projected.

Two deletion scenarios run on the same harness with both controllers on one manager. TestIntegration_ControlPlaneDeletion_SweepsProjectedRegistrationsFirst pins the order for a co-located Placement registration: it goes before the admin ApplicationCredential, KORCReady=False/FinalizingServiceRegistrations persists while a K-ORC-shaped finalizer holds the registration's User, and the K-ORC sweep starts only once the registration, its K-ORC children, PushSecret and source Secret are gone. TestIntegration_DedicatedNamespaces asserts that order for a Barbican registration placed in a namespace of its own, ahead of the cross-namespace teardown assertions.

Test Files

File	Coverage
controlplane_controller_test.go	Reconcile orchestration, sequential early-return, Ready aggregation, updateStatus error-join, idempotency
reconcile_sizing_test.go	reconcileSizing for a built-in profile (no read), an existing, a missing and an unreadable SizingProfile; the pipeline stop; sizingProfileToControlPlaneMapper
sizing_projection_test.go	The shared projection helpers: spread completion against the child's selector, the placement fallback, the replica default, uWSGI, Jobs, and the Glance apiServer block
reconcile_infrastructure_test.go	Managed/brownfield MariaDB + Memcached, unstructured readiness, condition contract, ObservedGeneration, the backing-service sizing of ensureMariaDB / ensureMemcached / ensureRabbitMQ
reconcile_dbcredentials_test.go	Managed ExternalSecret projection (name/store/data/owner-ref), brownfield no-op Ready=True, not-ready requeue + condition contract, distinct per-CP remote key/secret name
reconcile_adminpassword_test.go	Managed ExternalSecret projection (name/store/data/owner-ref), brownfield no-op Ready=True, not-ready requeue + condition contract, distinct per-CP remote key/secret name
reconcile_keystone_test.go	Keystone projection, infra gate, image/rotation/policy projection, condition contract, ObservedGeneration
reconcile_korc_test.go	AC mint, restricted↔unrestricted inversion, hash annotation/re-mint, missing-CRD safety, admin-credential push, catalog, condition contract
reconcile_projected_children_test.go	Prune and sweep of projected satellite children: ownership, name prefix, the Keep set, an absent CRD, the wrapped errors
reconcile_service_messaging_test.go	Bus delivery on the Neutron target, a second target under its own names, serviceMessagingSpec, the CA-mirror reap gate, the teardown stubs
reconcile_cinder_test.go	Cinder projection, the Keystone and registration gates, the satellite projection and prune, the derived Glance/Barbican/internal-tenant fields, the replica pins, the orphan teardown
reconcile_cinder_dbcredentials_test.go	Cinder DB-credential names, OpenBao paths, and the effective mode
reconcile_nova_test.go	Nova projection, the Keystone and Placement gates, the registration gate, the two database blocks and their shared mode, the console-proxy shapes, the replica pins, the compute-config mirror seam and its remote source, the remote-compute projection and reap, the orphan teardown
reconcile_nova_remote_compute_test.go	The remote bus delivery: the projected spec.remoteCompute, the handed URL landing in {controlplane.Name}-nova-remote-messaging at home and on a target, its rotation, the waits and errors, and the reap once the child has converged
reconcile_nova_dbcredentials_test.go	The two Nova DB-credential targets: names, OpenBao roles and paths, and the one effective mode both carry
reconcile_nova_metadata_secret_test.go	The generated metadata shared secret: the Password generator, the non-refreshing ExternalSecret and its shared_secret rewrite, the resolved reference, the pair a reference to the generated Secret keeps, and the reap once metadataSharedSecretRef names another Secret, on the target cluster as well as at home
reconcile_nova_hypervisor_operator_test.go	The hypervisor operator's account: the registration and its name budget, the relayed registration wait, the password wait and read failure, the seven-key auth Secret, one copy per compute cluster, a failed or unresolvable target, a foreign auth Secret refused beside the Nova and on a compute cluster, a Nova in a dedicated namespace or on a target cluster, rotation, and the prune with its ownership and mirror-label gates, its uncached reads and the placed Nova's ExternalSecret
reconcile_nova_metadata_agent_test.go	The metadata shared-secret copy for agents on target clusters: the targets one per cluster in the central's namespace and the agents left out, the neutron-less plane and the unserved kind, the single-key copy with its labels, one copy per cluster, a placed Nova's contract read on its own cluster, no target no write, the waits on the contract and its key, a contract read error, a failed agent list, an unresolvable Nova or agent cluster, a failed write and a foreign Secret, a failing cluster that holds back no other, rotation, and NovaReady waiting on the copy in reconcileNova
builtin_registrations_test.go	The shared registration leg: the projected child, the gate, the credential mirror, the reclaim of foreign spec fields, and the per-service roles
reconcile_credentialrotation_test.go	Nudge model, one-per-namespace resolution, bootstrap, deferred scheduled fields, target enum
credential_invariant_test.go	Security invariants (restricted mint, app-credential Secret not on any workload)
instrumentation_test.go	Wiring smoke test (records through the instrumenter), condition_type drift guard
setupwithmanager_test.go	For/Owns/Watches wiring, field-indexer registration
helpers_test.go	intervalToCron
integration_test.go	Full envtest reconciliation to Ready=True; the deletion scenarios TestIntegration_ControlPlaneDeletion_SweepsProjectedRegistrationsFirst (registration-first teardown order, co-located) and TestIntegration_DedicatedNamespaces (the same in a dedicated namespace, with the cross-namespace sweep) (build tag integration)

---

File Layout

operators/c5c3/
├── main.go                                     Scheme registration + bootstrap wiring, leaderElectionID
├── api/v1alpha1/
│   ├── controlplane_types.go                   ControlPlane CRD types
│   ├── sizing_types.go                         SizingSpec, ControlPlaneSizingSpec, SizingProfile CRD types
│   ├── sizing_profiles.go                      BuiltinSizing: the Minimal and Standard profiles
│   ├── sizing_merge.go                         MergeSizing, ResolveSizing, EffectiveSizingBase
│   ├── sizing_validation.go                    validateSizingSpec, validateResolvedSizing, inert warnings
│   ├── sizingprofile_webhook.go                SizingProfileWebhook (validating + defaulting)
│   ├── credentialrotation_types.go             CredentialRotation CRD types
│   ├── secretaggregate_types.go                SecretAggregate CRD types
│   ├── controlplane_webhook.go                 ControlPlaneWebhook (validating + defaulting)
│   ├── keystoneservice_types.go                KeystoneService CRD types (see the KeystoneService pages)
│   ├── keystoneservice_webhook.go              KeystoneServiceWebhook (see the KeystoneService pages)
│   └── ...
└── internal/
    ├── controller/
    │   ├── controlplane_controller.go          Reconciler struct, Reconcile(), setReadyCondition,
    │   │                                        aggregateReady, setServicesStatus, updateStatus, secret
    │   │                                        field indexer, SetupWithManager
    │   ├── external_mode.go                    IsExternalKeystone predicate + K-ORC external-failure
    │   │                                        classification and import-stall markers
    │   ├── helpers.go                          effective backing-service resolvers, intervalToCron
    │   ├── identity_backends.go                KeystoneIdentityBackend listing + WebSSO/MultiDomain
    │   │                                        projection helpers
    │   ├── instrumentation.go                  instrumenter + drift-guard map
    │   ├── reconcile_sizing.go                 reconcileSizing (SizingReady), effectiveSizing,
    │   │                                       sizingProfileToControlPlaneMapper
    │   ├── sizing_projection.go                shared projection of spec.sizing onto the children
    │   ├── korc_cloudsyaml.go                  clouds.yaml document builders (app-credential + password bootstrap)
    │   ├── korc_eso.go                         PushSecret + clouds.yaml ExternalSecret builders/ensure
    │   ├── korc_imports.go                     admin Domain/User import projection
    │   ├── korc_secrets.go                     app-credential Secret seeding, computeAdminPasswordHash
    │   ├── korc_unlatch.go                     clears latched K-ORC transport errors from child status
    │   ├── reconcile_infrastructure.go         reconcileInfrastructure (MariaDB + Memcached),
    │   │                                        childNamespace, memcachedGVK
    │   ├── reconcile_namespaces.go             reconcileNamespaces (dedicated service namespaces),
    │   │                                        child ownership labels + claim
    │   ├── reconcile_esotenant.go              reconcileESOTenantStore (per-tenant SecretStore + SA +
    │   │                                        mTLS cert), effectiveControlPlaneStoreRef
    │   ├── reconcile_dbcredentials.go          reconcileDBCredentials (per-CP DB-credential ExternalSecret)
    │   ├── reconcile_adminpassword.go          reconcileAdminPassword (per-CP admin-password ExternalSecret),
    │   │                                        effectiveAdminPasswordSecretRef
    │   ├── reconcile_keystone.go               reconcileKeystone projection
    │   ├── reconcile_horizon.go                reconcileHorizon projection (dashboard + WebSSO/MultiDomain)
    │   ├── reconcile_glance.go                 reconcileGlance projection (Glance + GlanceBackend
    │   │                                        children, DB-credential ExternalSecret)
    │   ├── reconcile_placement.go              reconcilePlacement projection (Placement child)
    │   ├── reconcile_placement_dbcredentials.go Placement DB-credential names, OpenBao paths, mode
    │   ├── reconcile_barbican.go               reconcileBarbican projection (Barbican +
    │   │                                        BarbicanSecretStore children, orphan teardown)
    │   ├── reconcile_barbican_openbao.go       Dedicated OpenBao instance and its ensemble
    │   ├── reconcile_barbican_dbcredentials.go Barbican DB-credential names, OpenBao paths, mode
    │   ├── reconcile_ovn.go                    reconcileOVN (mirror the referenced OVNCentral),
    │   │                                        ovnCentralToControlPlaneMapper
    │   ├── reconcile_neutron.go                reconcileNeutron projection (Neutron child, orphan
    │   │                                        teardown)
    │   ├── reconcile_cinder.go                 reconcileCinder projection (Cinder child, its two
    │   │                                        satellite kinds, orphan teardown)
    │   ├── reconcile_nova.go                   reconcileNova projection (Nova child, the
    │   │                                        compute-config mirror, orphan teardown)
    │   ├── reconcile_projected_children.go     Shared prune/sweep of a service's projected
    │   │                                        satellite children (ownership + name prefix)
    │   ├── reconcile_service_messaging.go      Shared-bus delivery on a service target
    │   │                                        (transport-URL Secret + CA mirror)
    │   ├── reconcile_neutron_dbcredentials.go  Neutron DB-credential names, OpenBao paths, mode
    │   ├── reconcile_cinder_dbcredentials.go   Cinder DB-credential names, OpenBao paths, mode
    │   ├── reconcile_nova_dbcredentials.go     The two Nova DB-credential targets: names, OpenBao
    │   │                                        roles and paths, the one shared mode
    │   ├── reconcile_nova_metadata_secret.go   The generated metadata shared secret (Password
    │   │                                        generator + ExternalSecret, resolve and prune)
    │   ├── reconcile_nova_remote_compute.go    The remote bus delivery, the projected
    │   │                                        spec.remoteCompute, and its reap
    │   ├── reconcile_nova_hypervisor_operator.go The hypervisor operator's account: registration,
    │   │                                        auth Secret, its copies, prune
    │   ├── reconcile_nova_metadata_agent.go    The metadata shared-secret copy for the agents
    │   │                                        on target clusters, its watch mapper and predicate
    │   ├── reconcile_korc.go                   reconcileKORC (AC mint/re-mint, drift detection)
    │   ├── reconcile_admincredential.go        reconcileAdminCredential (assemble + push + re-push
    │   │                                        nudges, semantic clouds.yaml gate)
    │   ├── reconcile_catalog.go                reconcileCatalog (mode fork; the managed identity
    │   │                                        Service/Endpoint), korcAvailableUpToDate
    │   ├── reconcile_catalog_external.go       reconcileCatalogExternal (import-first: unmanaged
    │   │                                        identity imports, opt-in entries, stall detection)
    │   ├── reconcile_serviceaccounts.go        reconcileServiceAccounts (folds the built-in
    │   │                                        registrations into ServiceAccountsReady)
    │   ├── builtin_registrations.go            The leg Glance/Placement/Barbican/Neutron/Cinder/
    │   │                                        Nova share: project the KeystoneService child,
    │   │                                        gate, mirror, reclaim; plus the account-only
    │   │                                        builder the Neutron-to-Nova notifier takes
    │   ├── registration_projection.go          K-ORC child + ESO builders shared with the
    │   │                                        KeystoneService controller
    │   ├── keystoneservice_controller.go       KeystoneServiceReconciler (see the KeystoneService pages)
    │   ├── keystoneservice_catalog.go          Its catalog block (see the KeystoneService pages)
    │   ├── keystoneservice_account.go          Its account block (see the KeystoneService pages)
    │   ├── reconcile_delete.go                 reconcileDelete (ORC-teardown finalizer sequencing)
    │   ├── reconcile_credentialrotation.go     CredentialRotationReconciler (nudge model)
    │   ├── requeue_intervals.go                infra/dbCredentials/adminPassword/keystone/korc/credentialRotation backoffs
    │   ├── controlplane_controller_test.go     Orchestration tests
    │   ├── external_mode_test.go               External-mode helper tests
    │   ├── helpers_test.go                     helper-function tests
    │   ├── identity_backends_test.go           Identity-backend projection tests
    │   ├── instrumentation_test.go             Metrics instrumentation + drift guards
    │   ├── korc_cloudsyaml_test.go             clouds.yaml builder tests
    │   ├── korc_unlatch_test.go                transport-error unlatch tests
    │   ├── reconcile_infrastructure_test.go    Infrastructure tests
    │   ├── reconcile_namespaces_test.go        Namespaces tests
    │   ├── reconcile_esotenant_test.go         ESO-tenant-store tests
    │   ├── reconcile_dbcredentials_test.go     DBCredentials tests
    │   ├── reconcile_adminpassword_test.go     AdminPassword tests
    │   ├── reconcile_keystone_test.go          Keystone projection tests
    │   ├── reconcile_horizon_test.go           Horizon projection tests
    │   ├── reconcile_glance_test.go            Glance projection tests
    │   ├── reconcile_placement_test.go         Placement projection tests
    │   ├── reconcile_placement_dbcredentials_test.go Placement DB-credential tests
    │   ├── reconcile_barbican_test.go          Barbican projection tests
    │   ├── reconcile_barbican_openbao_test.go  Dedicated OpenBao ensemble tests
    │   ├── reconcile_barbican_dbcredentials_test.go Barbican DB-credential tests
    │   ├── reconcile_ovn_test.go               OVNCentral mirroring tests
    │   ├── reconcile_neutron_test.go           Neutron projection tests
    │   ├── reconcile_cinder_test.go            Cinder projection tests
    │   ├── reconcile_nova_test.go              Nova projection tests
    │   ├── reconcile_projected_children_test.go Projected-children prune/sweep tests
    │   ├── reconcile_service_messaging_test.go Bus-delivery tests
    │   ├── reconcile_neutron_dbcredentials_test.go Neutron DB-credential tests
    │   ├── reconcile_cinder_dbcredentials_test.go Cinder DB-credential tests
    │   ├── reconcile_nova_dbcredentials_test.go Nova DB-credential tests
    │   ├── reconcile_nova_metadata_secret_test.go Nova metadata shared-secret tests
    │   ├── reconcile_nova_remote_compute_test.go Nova remote bus delivery tests
    │   ├── reconcile_nova_hypervisor_operator_test.go Hypervisor-operator account tests
    │   ├── reconcile_nova_metadata_agent_test.go Metadata-agent shared-secret copy tests
    │   ├── reconcile_korc_test.go              K-ORC mint/re-mint tests
    │   ├── reconcile_admincredential_test.go   AdminCredential tests
    │   ├── reconcile_catalog_test.go           Catalog (managed-mode) tests
    │   ├── reconcile_catalog_external_test.go  External-catalog tests
    │   ├── reconcile_serviceaccounts_test.go   ServiceAccounts tests
    │   ├── builtin_registrations_test.go       Built-in service registration tests
    │   ├── reconcile_delete_test.go            Deletion-sequencing (finalizer) tests
    │   ├── reconcile_credentialrotation_test.go CredentialRotation tests
    │   ├── credential_invariant_test.go        Security-invariant tests
    │   ├── setupwithmanager_test.go            Watch/Owns/indexer wiring tests
    │   ├── setupwithmanager_integration_test.go SetupWithManager envtest wiring (tag: integration)
    │   └── integration_test.go                 Envtest integration test (tag: integration)
    └── testutil/                                c5c3 envtest setup helpers

The c5c3_operator_* duration/error metric vectors are registered by the shared internal/common/instrumentation package (the former per-operator internal/metrics package was folded into it); instrumentation.go supplies only the c5c3_operator prefix and the name → condition_type map.

Migration: legacy flat paths → per-ControlPlane paths

Earlier releases wrote the admin / K-ORC credentials to cluster-global, flat OpenBao paths that assumed a single control plane per cluster. The operator now writes every credential family onto a per-CR path keyed by the owning ControlPlane's (or projected Keystone CR's) {namespace}/{name}, so multiple control planes (one per namespace; see Multi-instance) never collide in OpenBao. This is a one-time operator runbook to migrate an existing cluster; new clusters need no migration.

Switching a ControlPlane's secret store (from the default shared cluster store to a namespaced per-tenant SecretStore, or between stores) is a separate operation from this path migration and is documented in the Multi-Tenant Deployment guide → Per-ControlPlane secret stores and OpenBao identities. Switching the ref moves each PushSecret in place — unchanged name and remote key — so the irreplaceable key material is relocated, never re-created.

The new RemoteKey lands the moment the operator is upgraded — the next reconcile of each CR emits the per-CR path — so re-apply the OpenBao ACLs first or concurrently with the operator upgrade. Without the updated policies ESO returns 403 on the backup/push and the corresponding Ready conditions flip False (AdminCredentialReady for the admin AC; PasswordRotationReady for the Model-B admin password; FernetKeysReady / CredentialKeysReady for the signing keys).

Path mapping (legacy → per-CR):

Credential family	Legacy flat path	Per-CR path
Admin application credential (K-ORC)	openstack/keystone/admin/app-credential	openstack/keystone/{namespace}/{name}/admin/app-credential
Admin bootstrap password (Model B)	bootstrap/keystone-admin	bootstrap/{namespace}/{name}/admin
Fernet / credential keys (boundary-4)	openstack/keystone/{name}/{fernet,credential}-keys	openstack/keystone/{namespace}/{name}/{fernet,credential}-keys
Service-account passwords	— (new)	openstack/keystone/{namespace}/{name}/service-accounts/credentials, where {namespace}/{name} is the KeystoneService CR's

For the admin AC the {namespace}/{name} is the ControlPlane CR's (adminAppCredentialRemoteKeyFor); for the admin password and the Fernet / credential keys it is the projected Keystone CR's ({cp.Name}-keystone). The Fernet / credential move adds the namespace segment on top of the prior flat→per-name migration (see the keystone reconciler's Migration note: legacy flat paths); this change only adds the leading {namespace}/ segment.

One-time copy (preserve the last-pushed value so nothing is locked out):

# admin application credential (per ControlPlane <ns>/<cp>)
bao kv get kv-v2/openstack/keystone/admin/app-credential
bao kv put kv-v2/openstack/keystone/<ns>/<cp>/admin/app-credential clouds.yaml=@-
# admin bootstrap password (per Keystone CR <ns>/<name>, name = <cp>-keystone)
bao kv get kv-v2/bootstrap/keystone-admin
bao kv put kv-v2/bootstrap/<ns>/<name>/admin password=<value>
# fernet / credential keys (per Keystone CR <ns>/<name>)
bao kv get kv-v2/openstack/keystone/<name>/fernet-keys
bao kv put kv-v2/openstack/keystone/<ns>/<name>/fernet-keys value=<value>
bao kv get kv-v2/openstack/keystone/<name>/credential-keys
bao kv put kv-v2/openstack/keystone/<ns>/<name>/credential-keys value=<value>

Re-apply the OpenBao ACLs. Re-run deploy/openbao/bootstrap/setup-policies.sh (the kind/dev path; also invoked by hack/deploy-infra.sh), or for production clusters managed outside the bootstrap flow apply the updated policy files directly with bao policy write …:

Policy file	Grants write to
eso-tenant.hcl	the per-tenant admin AC, admin-password, fernet-keys, credential-keys, and service-account paths, each templated to the caller's own namespace (…/keystone/{namespace}/+/… and bootstrap/{namespace}/+/admin)

The three former wildcard write policies (push-keystone-keys.hcl, push-keystone-admin.hcl, push-app-credentials.hcl) were retired: they matched every tenant's paths behind a +/+ glob, so a leaked shared-identity token could overwrite any tenant's key material. Per-tenant secret traffic now authenticates as the eso-tenant role through the operator-provisioned per-tenant openbao-tenant-store, and eso-tenant.hcl scopes every writable path to the caller's own namespace.

Until the eso-tenant policy is re-applied, ESO's push to the new path returns 403 and the credential's Ready condition stays False.

Orphaned but harmless. After migration the legacy flat paths are orphaned but harmless: the live control plane no longer reads or refreshes them, no live PushSecret references them, and they are otherwise inert. Operators who want a clean OpenBao state can purge them once the per-CR paths are confirmed populated and Ready:

bao kv metadata delete kv-v2/openstack/keystone/admin/app-credential
bao kv metadata delete kv-v2/bootstrap/keystone-admin
bao kv metadata delete kv-v2/openstack/keystone/<name>/fernet-keys
bao kv metadata delete kv-v2/openstack/keystone/<name>/credential-keys

metadata delete removes the current version and all historical versions at the path — the canonical KV-v2 purge and the right inverse of the now-superseded write. (The Fernet / credential families were previously migrated flat→per-name in an earlier release, and the boundary-4 change layers the namespace segment on top.)