KeystoneService Reconciler Architecture

Reference documentation for the KeystoneServiceReconciler, the second reconciler the c5c3 operator registers. It owns the KeystoneService CR lifecycle and is the single writer of its status: the teardown finalizer, the per-block projection of K-ORC children, the OpenBao round-trip behind the account's consumer Secret, and the CatalogReady / AccountReady / Ready conditions.

For CRD type definitions, the field-level contracts, and the webhooks, see KeystoneService CRD API Reference. For the ControlPlane's own control loop, its admin-credential chain, and the shared controller-manager bootstrap both reconcilers reuse, see ControlPlane Reconciler Architecture.

Two properties shape everything below.

K-ORC is the single writer of OpenStack state. The reconciler adds no identity-API client of its own; it projects K-ORC CRs (Service, Endpoint, Region, User, Project, Domain, Role, RoleAssignment) and reads their conditions back. That is why a registration works identically against a Managed and an External Keystone, and why every wait condition ultimately reports what K-ORC reported.

Delivery is management-cluster only. The reconciler carries no cluster Resolver, unlike the ControlPlane's sub-reconcilers. A registration delivers its credentials into its own namespace, beside the workload that consumes them; only the ControlPlane-projected built-in registrations keep the placed-service delivery legs.

The ControlPlane projects a KeystoneService child per built-in service (Glance, Placement, Barbican, Neutron, Cinder, Nova) and two account-only ones, for the network service's compute notifier and for the hypervisor operator, so those registrations run through this same reconciler. Their projection is the ControlPlane's, documented with it.

Controller Registration

if err := (&controller.KeystoneServiceReconciler{
    Client:                  mgr.GetClient(),
    Scheme:                  mgr.GetScheme(),
    Recorder:                mgr.GetEventRecorderFor("keystoneservice-controller"),
    MaxConcurrentReconciles: maxConcurrentReconciles,
}).SetupWithManager(mgr); err != nil { ... }

SetupWithManager delegates to setupWithOptions, which takes the controller options as a parameter so the integration suite registers this exact watch chain with SkipNameValidation, not a hand-built copy that drifts.

Before any watch leg is wired, it registers the field indexer the ControlPlane watch depends on:

Index	Key	Extractor
KeystoneServiceControlPlaneRefIndexKey (spec.controlPlaneRef)	the resolved <namespace>/<name> of the reference	keystoneServiceControlPlaneRefExtractor

The key is namespace-qualified for two reasons: a ControlPlane event must not wake registrations pointing at a same-named plane in another namespace, and the qualification is what lets the mapper's List stay cluster-wide now that the allowlist admits foreign namespaces. A CR with an empty reference name (one that bypassed admission) indexes nothing, not a dangling key.

Watches

Resource	Watch Type	Effect
KeystoneService	For()	Filtered by watch.CRUpdatePredicate() so the controller's own status writes do not re-wake it
K-ORC Service, Endpoint, Region, User, Domain, Project, Role, RoleAssignment	Watches()	Mapped back to the owning CR by the ownership labels every child carries (keystoneServiceChildToRequest). These children live in the ControlPlane's namespace, so a registration from any other namespace can hold no owner reference to them and Owns() would miss it; one label mapper per kind covers both placements
Secret	Watches()	Same label mapper: it reaches the generation-scoped password Secret beside the User and the assembled source Secret beside the consumer
ExternalSecret, PushSecret	Owns()	The delivery objects stay in the CR's own namespace, where a controller reference is legal and the garbage collector reaps them
ControlPlane	Watches()	Index-backed fan-out (controlPlaneToKeystoneServicesMapper) with no generation predicate: the AdminCredentialReady flip the shared gate waits on and the allowlist edits that admit or de-list a namespace both arrive as ControlPlane updates, and both must re-enqueue at watch latency rather than at the next poll
Secret (consumer)	Watches()	credentialsSecretToKeystoneServiceMapper maps a Secret back through the <cr-name>-credentials name contract. The consumer Secret is ESO-owned (CreationPolicy: Owner), so no owner reference points at the CR and the Owns() legs never fire for it

The label mapper reaches both placements at once: a co-located child is stamped with the same labels on top of its owner reference, so one leg per kind covers every registration. An object carrying neither label belongs to something else (the ControlPlane's own children share these namespaces) and maps to nothing.

Watch delivery is the fast path, not the only one. Waiting states requeue on korcRequeueAfter (10s), so a K-ORC child that converges without an event still resolves.

RBAC

On its own kind the controller is narrow:

// +kubebuilder:rbac:groups=c5c3.io,resources=keystoneservices,verbs=get;list;watch;update
// +kubebuilder:rbac:groups=c5c3.io,resources=keystoneservices/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=c5c3.io,resources=keystoneservices/finalizers,verbs=update

It reads registrations and updates them only to install and release the teardown finalizer; it never creates or deletes a KeystoneService. Every other kind it touches (the K-ORC kinds and their status subresources, the ESO kinds, Secrets, events, ControlPlane reads) is already granted by the ControlPlane and CredentialRotation marker blocks, and the Helm chart mirrors the deduplicated union.

Reconciliation Flow

The finalizer is installed before any projection, so a deletion issued between that write and the next pass funnels through reconcileDelete and finds every child.

The undeclared blocks are stamped first, so both sub-condition types exist however the gates below resolve. Without it, a gate failure on a single-block CR would leave the other type absent and the aggregate Ready permanently False with nothing on the object explaining which sub-condition was missing.

The shared gates

Each gate writes its condition onto every declared block and returns. An undeclared block keeps its NotDeclared True: a gate the registration never reaches for that block is not a failure of it.

Gate	Reason	Behaviour
spec.controlPlaneRef resolves	ControlPlaneNotFound	A status condition, not an admission error, because GitOps may apply the registration before the plane. Any read failure other than NotFound propagates instead, so the workqueue backs off instead of reporting a dangling reference it did not observe
Namespace consent	NamespaceNotAllowed	keystoneServiceNamespaceAllowed admits the plane's own namespace, its dedicated service namespaces, then spec.korc.serviceRegistrations.allowedNamespaces. A nil block and an empty list are identical: both admit nothing beyond the implicit two. Nothing is projected
AdminCredentialReady on the plane	WaitingForAdminCredential	K-ORC cannot talk to Keystone before the admin credential is minted
Role allowlist	RoleNotAllowed	Account block only. keystoneServiceRoleRefusal looks up the management-cluster spec.namespaceAssignments entry for the CR's namespace and refuses every account role outside its allowedRoles. The plane's own namespace and its dedicated service namespaces are never bound, and a namespace without an entry stays unlimited. ensureAccount is skipped and the catalog block runs. The condition is written after the sweep, so the sweep's WaitingForServiceAccounts cannot mask it, and the gate adds no requeue: the ControlPlane watch brings the edit that lifts it

The role allowlist freezes the account the same way: the sweep keeps every child the spec still declares, so a role assignment granted earlier stays.

De-listing a namespace freezes its registrations here instead of tearing them down. The gate stops the projection; it never deletes what earlier passes created. Teardown happens only through deletion of the CR itself, so an allowlist edit can never destroy credentials a running service depends on.

Credential references

Past the gates, the reconciler resolves two clouds.yaml references, and which child gets which matters:

• credRef — the ControlPlane's admin credential, used by the read-only collision probes and the unmanaged imports.
• managedCredRef — the operator-owned admin password cloud, used by every managed child. The ApplicationCredential's K-ORC finalizer revokes the credential at teardown, so a managed child still authenticating through it would get a 404 on its own DELETE and orphan the Keystone resource.

Block independence

The two blocks are independent: a catalog failure never defers the account and vice versa. Each writes its own condition, and the pass requeues if either asked for it. Only a Kubernetes-level error short-circuits the pass, and the condition explaining it is already written by the time it propagates.

Each block runs through the shared instrumenter, so its duration and errors land on the c5c3_operator metric vectors under its own sub_reconciler label. See Metrics Instrumentation.

Pruning

sweepChildren removes every child this CR owns that the current spec no longer declares (a removed role, a removed endpoint interface, a whole removed block) and reports what it swept, split by the block that owns the kind. The split is by kind, which is exact here: Service, Endpoint and Region are only ever catalog children, and User / Project / Domain / Role / RoleAssignment and the delivery objects only ever account children.

A swept name is reported on the pass that issued its Delete, not once the object is gone: a K-ORC child stays Terminating behind its finalizer while the Keystone row is removed. The owning block's condition is therefore held (WaitingForCatalog or WaitingForServiceAccounts) until a later pass no longer lists it.

Two safeguards keep the sweep from reaching too far. Both the ownership test and the name test must pass, so a foreign object sharing the namespace can never be swept, which matters because the K-ORC children sit beside the plane's own children and those of every other registration. And the generation-scoped password Secrets are guarded by prefix, not by the declared-name set: their lifecycle belongs to the user projection, and a sweep must not race it into deleting the generation the managed User currently references.

Catalog Projection

ensureCatalog projects one managed K-ORC Service for the catalog row, an unmanaged Region import, and one managed Endpoint per declared interface, registered in that region.

It is mode-independent, and unlike the ControlPlane's own catalog reconciler. That one imports instead of creating in External mode, because the identity row it owns already exists in a brownfield catalog. A KeystoneService registers a service the plane does not carry, so it creates its row against a Managed and an External Keystone alike.

1. Collision gate. A short-lived unmanaged Service import filtered by type and effective name probes whether the row already exists. The verdict is read by the shared interpretProbe. Resolved without catalog.adopt gives ServiceCollision; still resolving gives ProbingForCollision; absent, or adopt consent, or a managed Service the operator already owns, proceeds. The probe covers the service row only: an endpoint is scoped to its service, so once the row is ours the endpoints under it are ours too.
2. Projection. The Service, the Region import and the Endpoints are pure projections of the spec, applied through Server-Side Apply. The import resolves the ControlPlane's spec.region (the region K-ORC's clouds.yaml names), and every Endpoint names it as its regionRef. A client that sets region_name, as every nova client section and the neutron notifier do, finds only a row registered in that region: keystoneauth skips a region-less one and answers EndpointNotFound. The region is imported per registration rather than referenced from the plane's own adopted Region, because K-ORC holds a Region CR while an Endpoint names it, which would tie the plane's teardown to every registration, and an External plane adopts no Region.
3. Terminal errors. The Service's and the Region import's terminal errors are reported before the Endpoints', so the root stuck dependency surfaces instead of an Endpoint merely blocked behind it. Either gives CatalogFailed: K-ORC has stopped retrying, so a bounded wait would never resolve. A latched transport failure is cleared ahead of this check, so K-ORC retries and the block reports WaitingForCatalog; a clear that fails gives TransportErrorRetryFailed. See Latched transport errors.
4. Availability. Registering the CRs only instructs K-ORC to create the rows. The block reports ready only once every child is Available for its current generation, or a failing registration would read Ready while the catalog stayed empty. An unresolved Region import reports WaitingForCatalog naming the region.
5. Retiring region-less rows. Earlier versions registered each interface without a region, under the discriminator endpoint-<interface>. K-ORC neither updates an endpoint's region nor accepts a changed regionRef, so the regioned row is a new CR, region-ep-<interface>. The region-less CR stays until its replacement is Available, and only then does retireLegacyCatalogEndpoints delete it, so the catalog never goes without the interface. The block holds WaitingForCatalog until K-ORC has removed the old row. The sweep keeps a declared interface's region-less CR for exactly that hand-over and removes an undeclared one's at once.

Account Projection

ensureAccount projects the Keystone identity and delivers its credentials. A CR carries one account, so the precedence between outcomes is a chain of early returns, not a pass over a list.

(0) Admin-identity refusal. A registration whose effective user name and domain resolve to the plane's own admin identity is refused with ServiceAccountCollision, changing nothing. It sits ahead of everything else, including the collision probe, because account.adopt: true short-circuits that probe and adopt is the documented remedy for every other collision, so a probe-side rule would be opt-out by design. Admission cannot carry it either: it resolves against the referenced ControlPlane, which the CRD contract allows to be absent at admission time.

The comparison goes through keystoneNameKey, which folds case and accents because Keystone's identity tables use a *_general_ci collation. A narrower comparison would wave a variant spelling past the refusal while Keystone still resolved it onto the admin user. The remedy in the message is chosen by reading the live managed User, not status.account.userID: the child is created a full pass before status records its ID, and a lookup failure counts as owned, so the destructive advice is never what an error falls back to.

(1) Store gate. The delivery leg is gated on the ControlPlane's secret store being ready in the CR's own namespace, so an ESO or OpenBao outage surfaces as SecretStoreNotReady promptly instead of at the next hourly refresh. The gate is scoped to the account block: a catalog-only registration touches no secret machinery and must not wait for a store it never reads.

(2) Domain handle. When the effective domain is the plane's admin domain, the reconciler reuses the admin Domain import, which the ControlPlane created and owns, so this CR's prune never sweeps it. Any other domain gets its own unmanaged import.

(3) Project. create: false references a pre-existing project through an unmanaged import the operator never creates or deletes. create: true creates a managed Project behind the same fail-loudly probe as the user, because K-ORC's managed create would otherwise silently adopt a same-named project. The collision message points at create: false, not at adopt: project takeover is expressed as referencing.

(4) User. A probe decides exists/absent before any managed User is created; account.adopt is the consent that skips it. The managed User then carries a generation-scoped password Secret (…-password-v<N>). K-ORC's user actuator re-applies a password only when the reference name changes, so a rotation is a Secret-name flip and never an in-place edit. The password value is generated once per generation and preserved across passes.

A registration created with rotation.mode: Scheduled emits a ScheduledRotationDeferred Normal event: the mode is accepted by the schema but its scheduling logic is deferred, and the deferral is not silent.

(5) Roles. Per declared role, one unmanaged Role import filtered by name (Keystone roles are global, so the import carries no domain and reading one never mutates it) and one managed RoleAssignment binding that role to this account's user on its project. The imports are per CR, never shared: registrations are independently owned and deleted, so a shared import would have no single owner and the first CR deleted would pull it out from under the others. A role that does not exist in Keystone never resolves, and the wait message says so.

Role readiness is folded into the block's readiness, so an account is never reported provisioned before the roles it needs are bound.

(6) Publish. Once K-ORC confirms the current password is applied to the user, the reconciler assembles the source Secret, mirrors it to the per-CR OpenBao path through a PushSecret, and materializes the consumer Secret through an ExternalSecret. A content-hash annotation (c5c3.io/keystoneservice-push-hash) forces an immediate re-push and re-sync on rotation instead of waiting for the refresh interval. The block reports ready only once the materialized password matches the current generation, so a rotated-away password never reads ready.

A terminal K-ORC error on the Project, the User, a Role import or a RoleAssignment gives ServiceAccountsFailed, because K-ORC has stopped retrying. A latched transport failure is cleared first, and the block reports WaitingForServiceAccounts while K-ORC retries. A clear that fails gives TransportErrorRetryFailed, on a role child as much as on the Project or the User: applyAccountRole owns no condition, so ensureAccount recognises the error it relays rather than reporting it under a second reason. See Latched transport errors.

The bounded waits in steps 3 through 6 all report WaitingForServiceAccounts, with a message naming the stuck dependency in dependency order (project, then user, then the password application), so the condition points at the real blocker instead of at whatever is merely waiting behind it.

External-mode failure classification

Against an External ControlPlane, a K-ORC import that cannot reach the external Keystone reports a non-terminal TransientError. Without help, an authentication failure, a TLS error, and a catalog mismatch would all read as "registered but not yet Available" forever, with nothing naming the cause.

keystoneServiceWaitOrClassify therefore prefers a classifiable failure over the generic wait reason on every bounded wait in both blocks, surfacing AuthenticationFailed, CredentialDrift, EndpointUnreachable, TLSVerificationFailed, or CatalogEndpointMismatch with the external auth URL and K-ORC's own message.

Latched transport errors

A K-ORC create that fails at the transport layer (dial tcp …: connect: connection refused) can land on the child as a terminal Progressing=False with reason InvalidConfiguration. K-ORC's ShouldReconcile then skips that child for as long as the condition stands at the current generation, across controller restarts and upgrades, and reconciles it again once the condition is absent. The K-ORC commit this repo pins retries transport errors itself, so a latch of this class was written by an earlier version or carries an error K-ORC does not classify as transport.

unlatchKORCTransportErrors removes the condition when two things hold: the Progressing condition is terminal at the current generation with reason InvalidConfiguration, and the message carries no such host, connection refused, dial tcp or i/o timeout. Every other terminal error stays. A message that also names x509, 401 or Unauthorized classifies as TLS or authentication first. No retry repairs either one, so that latch keeps reporting what a human has to fix.

The retries are paced by the operator's own clock, not by K-ORC's. Each clear stamps c5c3.io/korc-transport-unlatched-at on the child, and a child cleared less than korcTransportUnlatchBackoff (30s, plus that child's own offset of up to another 30s) ago is left alone. The first latch the operator sees is therefore handed back at once — K-ORC has already given up on it — and every retry after that is spaced. The condition's own lastTransitionTime is deliberately not read: it is written by whoever wrote the condition, so pacing on it would compare this pod's wall clock against K-ORC's. A stamp that does not parse counts as due, and so does one dated in the future — a clock this operator cannot have written from — so no reading of the annotation can hold a latch indefinitely. The stamp is written after the clear and records only a clear that happened, so a clear the apiserver denies keeps being reported on every pass instead of being paced away by its own failure. The stamp is retried inline and never reported: the repair it paces has already succeeded, so a stamp that still cannot be written is logged rather than raised as TransportErrorRetryFailed, and the children behind that one are cleared in the same pass. The clear itself is a status merge patch carrying the child's resourceVersion, so a child whose status K-ORC rewrote in between is skipped and re-evaluated on the next 10s pass.

Nothing is deleted. K-ORC's delete path on a managed child with no status.id looks the resource up by name and deletes what it finds, which under account.adopt: true is the pre-existing Keystone user, and the User's deletion-guard finalizer on its Project can leave the Project Terminating. There is no attempt cap and no growth: an unreachable Keystone costs one K-ORC retry per child per 30s — far less than K-ORC's own retry rate for a transport error it has not latched — and the condition keeps reporting the failure between retries. The offset is derived from the child's UID, so it holds still across passes and spreads the children of one plane — which latch inside the same window and are cleared in the same pass — over several 10s passes instead of handing them back in one synchronized burst. One log line per cleared child is the record.

A clear the operator cannot perform at all reports TransportErrorRetryFailed on the block's condition and returns the error; that is a Forbidden from RBAC predating the status-subresource grant, not a passing apiserver failure. A 429, 503, server timeout or etcd timeout is deferred to the next pass instead: the repair is opportunistic, and reporting it would overwrite the K-ORC failure the block relays. The patches need patch on the nine K-ORC <kind>/status resources, granted by the ControlPlane marker block, and the annotation is written with the patch the same block already grants on the resources themselves.

Child Naming and Placement

The K-ORC children and the generation-scoped password Secrets live in the ControlPlane's namespace, never the registration's. K-ORC reads the clouds.yaml named by a child's cloudCredentialsRef from that child's own namespace, and the admin credential is materialized once, in the plane's namespace, and stays there: copying it into every allowlisted namespace would hand each tenant the cloud-admin credential and undo the escalation the allowlist exists to prevent. So the children go to the credential, not the credential to the children.

Only the delivery objects stay in the registration's own namespace: the assembled source Secret, the PushSecret, the ExternalSecret, and the consumer Secret the service reads.

The figure shows both namespaces and the path of the password. Its numbers are the steps below.

In the figure {reg} and {reg-ns} stand for the registration's name and namespace, and {prefix} for the child-name prefix of Name composition, written <prefix> below.

1. resolveControlPlane reads the ControlPlane that spec.controlPlaneRef names, and keystoneServiceNamespaceAllowed admits the registration when its namespace is the ControlPlane's own, one of its dedicated service namespaces, or listed in spec.korc.serviceRegistrations.allowedNamespaces. A CR from any other namespace reports NamespaceNotAllowed on every declared block and projects nothing. Past the gate the registration waits with WaitingForAdminCredential until the ControlPlane reports AdminCredentialReady.
2. The operator writes the children into the ControlPlane namespace: the Secret <prefix>password-v<N> with a generated password under the key password, the account children (User, Project, one Role import and one RoleAssignment per role) and the catalog children (Service, the Region import, one Endpoint per interface). Each child carries the labels c5c3.io/keystoneservice-name and c5c3.io/keystoneservice-namespace, because an owner reference cannot cross the namespace.
3. K-ORC creates the user in Keystone with the password its User names in passwordRef. K-ORC reads the clouds.yaml a child names in the child's own namespace: managed children authenticate with {controlplane.Name}-admin-password-cloud, probes and imports with k-orc-clouds-yaml.
4. Once the User reports the current password Secret as applied and the role assignments are Available, the operator copies the password into the Secret <prefix>source in the registration namespace and adds username, project_name, user_domain_name, project_domain_name, auth_url, region_name and a clouds.yaml. This copy is the only step that crosses the namespace.
5. The PushSecret <prefix>backup pushes the source Secret through the secret store of the registration namespace, by default the SecretStoreopenbao-tenant-store that reconcileRegistrationTenantStores provisions there, to openstack/keystone/<namespace>/<name>/service-accounts/credentials. Its deletion policy is Delete.
6. The ExternalSecret <metadata.name>-credentials reads the properties password and clouds.yaml from that path into the Secret of the same name, with a refresh of 1h. AccountReady turns True with reason AccountProvisioned once the materialized password equals the generated one.
7. The service mounts <metadata.name>-credentials. It is the one child whose name carries no prefix.

Ownership

Placement	Mechanism
Same namespace as the CR	Controller owner reference, so the garbage collector reaps the child
Another namespace	The labels c5c3.io/keystoneservice-name and c5c3.io/keystoneservice-namespace, since Kubernetes rejects an owner reference across namespaces

Both halves of the label pair are required: a name is only unique within its namespace. Every child is stamped the moment it is created, so it is recognizable on the pass that writes it, not on a later one.

Before adopting a name that already exists outside the CR's namespace, the apply refuses anything this registration did not create. Without that check the apply would overwrite a stranger's spec and the teardown would then delete it.

Name composition

Every child name is composed from a per-CR prefix:

<metadata.name>-<8 hex>-registration-<discriminator>

The hash is taken over <namespace>/<name>, not over the name alone: the children live in the ControlPlane's namespace, so two same-named CRs in different namespaces would otherwise project onto the same child names, one silently reshaping and then deleting the other's Keystone identity. The readable base stays in front so kubectl get user still reads as the registration it belongs to.

Discriminator	Child
service, service-probe	Catalog Service and its collision probe
region	Import of the ControlPlane's Keystone region the endpoints are registered in
region-ep-<interface>	One Endpoint per declared interface, in that region
endpoint-<interface>	The region-less Endpoint an earlier version registered; retired once its region-ep-<interface> replacement is Available
user, user-probe	Managed User and its collision probe
project, project-probe	Project and its collision probe
domain	Domain import, when the effective domain is not the plane's admin domain
role-<slug>, assign-<slug>	Role import and RoleAssignment, per role
password-v<N>	Generation-scoped password Secret
source, backup	Assembled source Secret and its PushSecret

The consumer Secret is the one child that does not carry the prefix. It is named <metadata.name>-credentials, because it is the documented contract a service reads its credentials from and has to stay predictable from the CR's name alone. The sweep's name test admits it alongside the prefix, and the consumer-Secret watch maps it back through that same suffix.

The prefix is also why the CRD's validating webhook bounds metadata.name: the composed child name has to stay within the API server's 253-byte object-name limit.

Deletion and Teardown

reconcileDelete runs the c5c3.io/keystoneservice-teardown finalizer. It reuses the same sweepChildren the per-pass prune uses, passing an empty keep set so everything goes, and issues the deletes dependents-first: assignments before roles, endpoints before the region import and the service they reference, all before the user and project they bind. K-ORC enforces its own ordering through finalizers, but issuing them in dependency order keeps the intent legible and avoids a guaranteed retry.

With the ControlPlane present the teardown is patient. K-ORC's finalizers take the catalog rows and the Keystone user out of the identity plane, so the finalizer is held until no owned child is listed any more. There is no stall escape: a child K-ORC cannot delete (a 403 on DELETE, an unreachable Keystone) leaves the CR visibly Terminating instead of silently orphaning a Keystone identity nobody is tracking any more.

With the ControlPlane gone the teardown fails open. K-ORC has no credentials left to reach Keystone with, so waiting could only hold the CR hostage to a plane that no longer exists. The Kubernetes children are still deleted, since they would otherwise leak, but their outcome no longer gates the finalizer. For a registration the ControlPlane projected, that path only ever meets finalizer-free children: the ControlPlane teardown strips its children's K-ORC and ESO finalizers before it releases itself.

The children are looked for in the resolved controlPlaneRef namespace, which is where they are whether or not the plane itself is still readable.

For which Keystone resources a deletion destroys, see Deletion Semantics on the CRD page.

Metrics Instrumentation

Both block projections run through the package-scope instrumenter the ControlPlane reconciler shares, so they observe the same c5c3_operator_reconcile_duration_seconds{sub_reconciler=…} histogram and c5c3_operator_reconcile_errors_total{sub_reconciler=…,condition_type=…} counter. The behavioural contract of that helper is documented in the ControlPlane Reconciler reference.

This controller contributes two entries to subReconcilerConditionTypes:

sub_reconciler	condition_type
KeystoneServiceCatalog	CatalogReady
KeystoneServiceAccount	AccountReady

The names carry the CR kind as a prefix because Catalog and ServiceAccounts already label the ControlPlane's own legs (the identity row and the registration aggregation), and the two must stay distinguishable per metric series. The condition type strings are shared with the ControlPlane's CatalogReady, which is harmless: they live on different objects.

The drift guard TestSubReconcilerConditionTypesCoversAllNames asserts that every mapped condition_type is a member of the ControlPlane's subConditionTypes or of keystoneServiceSubConditionTypes. A sub_reconciler name reaching the instrumenter without a key here resolves to the sentinel condition_type=UNKNOWN, so the drift is visible in dashboards instead of silent.

The aggregate Ready carries no sub_reconciler of its own: it is the roll-up, re-derived from both sub-conditions on every status persist by SetAggregateReady, which requires every listed type to be present and True.

Testing

Layer	Location
Controller unit tests	operators/c5c3/internal/controller/keystoneservice_controller_test.go
Catalog projection	operators/c5c3/internal/controller/keystoneservice_catalog_test.go
Account projection	operators/c5c3/internal/controller/keystoneservice_account_test.go
Transport-error unlatch	operators/c5c3/internal/controller/korc_unlatch_test.go
CRD schema and webhook	operators/c5c3/api/v1alpha1/keystoneservice_types_test.go, keystoneservice_webhook_test.go
Own-namespace e2e	tests/e2e/c5c3/keystone-service/
Cross-namespace e2e	tests/e2e/c5c3/keystone-service-foreign-namespace/
Admission rejection corpus	tests/e2e/c5c3/invalid-keystoneservice-cr/

Unit tests drive a fake client and envtest carries K-ORC as schema without a controller, so neither sees Keystone, OpenBao, ESO, or cert-manager. The collision postures of both blocks are asserted end to end only in the own-namespace suite. See ControlPlane E2E Test Suites.