Keystone CRD API Reference

Reference documentation for the Keystone Custom Resource Definition. The Keystone CRD is the reference implementation for all CobaltCore service operators — the patterns established here (types, webhooks, generation, scheme registration) will be replicated for Nova, Neutron, Glance, and other OpenStack service operators.

API Group and Version

Field	Value
Group	keystone.openstack.c5c3.io
Version	v1alpha1
Kind	Keystone
List Kind	KeystoneList
Scope	Namespaced

Import path:

import keystonev1alpha1 "github.com/c5c3/cobaltcore/operators/keystone/api/v1alpha1"

Scheme registration:

The init() function in keystone_types.go registers Keystone and KeystoneList with the SchemeBuilder. Operator main.go calls AddToScheme to register the types with the manager's scheme.

---

Sub-Resource Naming Convention

All operator-managed sub-resources for a Keystone CR are named after the CR itself with no -api suffix. For a Keystone CR named keystone in namespace openstack, the operator creates:

Sub-resource	Name	Cluster-internal DNS
Deployment	keystone	—
Service (ClusterIP)	keystone	keystone.openstack.svc.cluster.local
HorizontalPodAutoscaler	keystone	—
PodDisruptionBudget	keystone	—
NetworkPolicy	keystone	—
HTTPRoute	keystone	—
Container & named port	keystone	port 5000

This convention replaces the historical form that appended an -api suffix to each sub-resource (so the same CR would have produced -api-suffixed sub-resources). The change aligns the internal Service DNS with the public Gateway hostname posture and removes the redundant suffix that no longer reflected a meaningful split — the Keystone CR has only ever owned the API role.

For migration semantics (catalog refresh, ownerReference cascade GC of legacy sub-resources, and operator workflows for upgrading a pre-rename cluster), see the Keystone Upgrade Flow reference.

---

Resource Shape

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  database:
    clusterRef:
      name: mariadb
    database: keystone
    secretRef:
      name: keystone-db-credentials
      key: password
  cache:
    backend: dogpile.cache.pymemcache
    clusterRef:
      name: memcached
  fernet:
    rotationSchedule: "0 0 * * 0"
    maxActiveKeys: 3
  credentialKeys:
    rotationSchedule: "0 0 * * 0"
    maxActiveKeys: 3
  trustFlush:
    schedule: "0 * * * *"
  autoscaling:
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilization: 80
  networkPolicy:
    ingress:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openstack
  uwsgi:
    processes: 4
    threads: 4
    httpKeepAlive: true
  bootstrap:
    adminUser: admin
    adminPasswordSecretRef:
      name: keystone-admin
      key: password
    region: RegionOne
    publicEndpoint: https://keystone.example.com/v3
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AllReady
      message: All sub-resources are ready
      lastTransitionTime: "2026-03-09T00:00:00Z"
    - type: KeystoneAPIReady
      status: "True"
      reason: APIHealthy
      message: "Keystone API is responding at http://keystone.openstack.svc.cluster.local:5000/v3"
      lastTransitionTime: "2026-03-09T00:00:00Z"
  endpoint: http://keystone.openstack.svc.cluster.local:5000/v3
  installedRelease: "2025.2"

Printer Columns

kubectl get keystones displays these columns:

Column	JSON Path	Type
Ready	.status.conditions[?(@.type=='Ready')].status	string
Endpoint	.status.endpoint	string
Release	.status.installedRelease	string
Age	.metadata.creationTimestamp	date

---

KeystoneSpec

Field	Type	Required	Default	Description
deployment	DeploymentSpec	No	See below	Pod-level knobs for the Keystone API Deployment (replicas, resources, rollout strategy, graceful-termination timings, scheduling constraints). Grouping keeps the spec root legible as future affinity/tolerations/nodeSelector knobs are added.
image	ImageSpec	Yes	—	Keystone container image reference.
database	DatabaseSpec	Yes	—	MariaDB connection configuration. Includes the optional tls sub-block that opts in to TLS / mTLS for the connection; when nil, the connection is plaintext TCP — preserving the previous behavior for all existing CRs.
cache	CacheSpec	Yes	—	Memcached cache configuration.
fernet	FernetSpec	No	See below	Fernet key rotation configuration.
credentialKeys	CredentialKeysSpec	No	See below	Credential-key rotation configuration. Drives the per-CR CronJob that rotates and credential_migrates the credential keys used for encrypting application credentials.
passwordRotation	*PasswordRotationSpec	No	nil (feature off)	Optionally enables scheduled rotation of the admin password. Day-2 rotation lives at the spec root beside the fernet/credential key rotation config. Nil leaves the feature off and the PasswordRotation sub-reconciler is a clean no-op.
trustFlush	*TrustFlushSpec	No	{schedule: "0 * * * *", suspend: false} (materialized by the defaulting webhook)	Trust flush CronJob configuration. Default-on: when the field is omitted, the defaulting webhook populates an hourly schedule so keystone-manage trust_flush runs by default; there is no nil-back path on a webhook-enabled cluster (a kubectl patch ... 'spec/trustFlush'='null' round-trips through admission and is re-materialized). To pause without deleting the CronJob, set suspend: true — the resource and TrustFlushReady=True condition are preserved.
federation	*FederationSpec	No	nil	Federation sidecar knobs (the proxy image). Federation activates by attaching an OIDC KeystoneIdentityBackend, not by this block.
bootstrap	BootstrapSpec	Yes	—	Initial Keystone bootstrap parameters.
middleware	[]MiddlewareSpec	No	nil	WSGI middleware filters for api-paste.ini.
plugins	[]PluginSpec	No	nil	Service plugins/drivers to configure.
policyOverrides	*PolicySpec	No	nil	Custom oslo.policy rules.
secretStoreRef	*SecretStoreRefSpec	No	nil	Selects the External Secrets store this Keystone routes its ExternalSecrets and backup PushSecrets through. When omitted, the shared cluster-scoped ClusterSecretStore named openbao-cluster-store is used, so existing deployments are unchanged. Set to a namespaced SecretStore in this Keystone's own namespace to reach OpenBao as a per-tenant identity. Normally projected from the owning ControlPlane rather than set here.
targetClusterRef	*TargetClusterRefSpec	No	—	Names the registered target cluster that receives this Keystone's children: the Deployment, the ConfigMaps, the Secrets, and the database CRs. The CR itself does not move, and neither do its status, its finalizers, or the webhooks that admit it. When omitted, the children are created on the local cluster the operator runs on, so an existing CR keeps its behavior without an edit. name must be a non-empty DNS-1123 subdomain matching the registration Secret. Immutable (CEL transition rule): adding, removing, or renaming the ref strands the children already created on the previously selected cluster; delete and recreate instead. See Target Clusters.
autoscaling	*AutoscalingSpec	No	nil	Horizontal pod autoscaling configuration. When set, an HPA is created targeting the {name} Deployment. When removed, the HPA is deleted.
networkPolicy	*NetworkPolicySpec	No	nil	Network isolation for Keystone API pods. When set, a NetworkPolicy restricting ingress to TCP 5000 and auto-deriving egress rules for DNS, MariaDB, and Memcached is created. When nil, no NetworkPolicy is managed and traffic is unrestricted.
gateway	*GatewaySpec	No	nil	Gateway API HTTPRoute configuration. When set, an HTTPRoute is created targeting the {name} Service on port 5000 and attached to the referenced pre-existing Gateway; status.endpoint is updated to https://{hostname}/v3. When removed, the HTTPRoute is deleted and status.endpoint reverts to the cluster-local Service URL.
uwsgi	*UWSGISpec	No	nil	uWSGI application server parameters. When set, the operator uses these values for the Deployment container command. When nil, hardcoded defaults (processes=2, threads=1, httpKeepAlive=true) are used in the reconciler.
logging	*LoggingSpec	No	See below	oslo.log configuration for the Keystone API container. When nil, the defaulting webhook materializes a baseline (format=text, level=INFO, debug=false, no per-logger overrides) so downstream reconciler code never sees a nil pointer. When set, zero-valued sub-fields are partially filled with the same baseline.
extraConfig	map[string]map[string]string	No	nil	Free-form INI sections for additional configuration. The render-time merge follows plugins < operator defaults < extraConfig (each stage merged key-wise), so extraConfig wins over both. Overrides of operator-owned keys are honored but reported via the ExtraConfigHealthy condition and an ExtraConfigOwnedKeyOverride Warning event. Option names are validated at admission against a per-release option catalog embedded in the operator (release derived from spec.image.tag; a digest-pinned image or a release with no embedded catalog skips the check with an admission warning). Sections declared by spec.plugins and operator-owned keys are exempt; an unknown section or option is rejected with the section and key named, and a deprecated-but-accepted option is admitted with a warning naming its replacement. Values are never checked, and the rule is webhook-only with no CEL backstop.

DeploymentSpec

Groups the pod-level knobs for the Keystone API Deployment under spec.deployment.

Field	Type	Required	Default	Description
replicas	int32	No	3	Number of Keystone API replicas. Minimum: 1. The webhook provides a secondary default of 3 when zero.
resources	*corev1.ResourceRequirements	No	See below	CPU and memory requests and limits for the Keystone API container. When unset, the defaulting webhook injects sensible defaults to ensure Burstable QoS class and enable HPA utilization calculations.
topologySpreadConstraints	[]corev1.TopologySpreadConstraint	No	See below	Scheduler hints for spreading pods across zones and nodes. nil injects two defaults (zone + hostname, MaxSkew=1, ScheduleAnyway); a non-nil value (including []) is used verbatim.
priorityClassName	*string	No	nil	PriorityClass attached to the Keystone API pod spec. When set, the webhook verifies the class exists; when unset, no priority class is configured.
terminationGracePeriodSeconds	*int64	No	nil	Grace period (seconds) granted to Keystone API pods between SIGTERM and SIGKILL during rolling updates. When nil, the reconciler applies 30 (the CRD schema emits no default: so pre-existing CRs are not mutated on operator upgrade). Minimum: 10. Must be strictly greater than preStopSleepSeconds. Drives the PodSpec terminationGracePeriodSeconds. See Graceful-termination fields.
preStopSleepSeconds	*int64	No	nil	Sleep duration (seconds) of the preStop lifecycle hook, covering the window between EndpointSlice removal and kube-proxy/ingress propagation. When nil, the reconciler applies 5 (the CRD schema emits no default: so pre-existing CRs are not mutated on operator upgrade). Minimum: 0. Must be strictly less than terminationGracePeriodSeconds. See Graceful-termination fields.
strategy	*appsv1.DeploymentStrategy	No	RollingUpdate(maxSurge=1, maxUnavailable=0)	Overrides the Deployment rollout strategy. When nil, the reconciler injects RollingUpdate with maxUnavailable=0 and maxSurge=1 so available capacity never drops below spec.deployment.replicas during an image-tag patch. Set to customize surge/unavailable counts or switch to Recreate.

CEL Validation Rules

The CRD includes structural validation rules enforced by the API server before webhooks are invoked. Rules that reference oldSelf are transition rules: they are evaluated only on UPDATE and enforce field immutability. Because they are enforced by the API server itself, they keep protecting the field even when the validating webhook is unavailable.

Field	Rule	Error Message
spec.database	has(self.clusterRef) != has(self.host)	"exactly one of clusterRef or host must be set"
spec.database	!has(self.credentialsMode) || self.credentialsMode != 'Dynamic' || has(self.clusterRef)	"credentialsMode Dynamic requires clusterRef (managed mode)"
spec.database	!has(self.tls) || self.tls.mode == '' || self.tls.mode == 'disabled' || (self.tls.caBundleSecretRef.name != '' && self.tls.clientCertSecretRef.name != '')	"when database.tls is enabled (mode is neither empty nor 'disabled'), both database.tls.caBundleSecretRef.name and database.tls.clientCertSecretRef.name must be set"
spec.database	self.database == oldSelf.database (UPDATE only)	"database name is immutable"
spec.database	has(self.clusterRef) == has(oldSelf.clusterRef) (UPDATE only)	"database mode (managed clusterRef vs brownfield host) is immutable"
spec.database.tls.mode	Enum: disabled, prefer, require, verify-ca, verify-full	—
spec.bootstrap.adminUser	self == oldSelf (UPDATE only)	"bootstrap.adminUser is immutable"
spec.bootstrap.region	self == oldSelf (UPDATE only)	"bootstrap.region is immutable"
spec.cache	has(self.clusterRef) != (has(self.servers) && size(self.servers) > 0)	"exactly one of clusterRef or servers must be set"
spec.policyOverrides	(has(self.rules) && size(self.rules) > 0) || self.configMapRef != null	"at least one of rules or configMapRef must be set"
spec.policyOverrides.rules	!has(self.rules) || self.rules.all(k, size(k) > 0)	"policy rule name must not be empty"
spec.policyOverrides.rules	!has(self.rules) || self.rules.all(k, size(self.rules[k]) > 0)	"policy rule value must not be empty"
spec.autoscaling	has(self.targetCPUUtilization) || has(self.targetMemoryUtilization)	"at least one of targetCPUUtilization or targetMemoryUtilization must be set"
spec.autoscaling	!has(self.minReplicas) || self.minReplicas <= self.maxReplicas	"minReplicas must not exceed maxReplicas"
spec.networkPolicy	size(self.ingress) > 0	"at least one ingress source must be specified"
spec.deployment	drain window: effective preStopSleepSeconds (default 5) must be < effective terminationGracePeriodSeconds (default 30)	"preStopSleepSeconds must be strictly less than terminationGracePeriodSeconds"
spec.uwsgi	!has(self.httpKeepAliveTimeout) || !has(self.httpKeepAlive) || self.httpKeepAlive	"httpKeepAliveTimeout may only be set when httpKeepAlive is true"
spec.logging.perLoggerLevels	self.all(k, k != '')	"logger name must not be empty"
spec.logging.perLoggerLevels	self.all(k, self[k] in ['DEBUG','INFO','WARNING','ERROR','CRITICAL'])	"per-logger level must be one of DEBUG, INFO, WARNING, ERROR, CRITICAL"
spec.plugins	list-map keyed by configSection (x-kubernetes-list-type: map)	"Duplicate value" (a repeated configSection is rejected by the API server)
spec	has(self.targetClusterRef) == has(oldSelf.targetClusterRef) (UPDATE only)	"targetClusterRef is immutable"
spec	!has(self.targetClusterRef) || !has(oldSelf.targetClusterRef) || self.targetClusterRef.name == oldSelf.targetClusterRef.name (UPDATE only)	"targetClusterRef is immutable"
spec.bootstrap.publicEndpoint	Pattern: ^https?://	(non-URL value rejected by API server)
spec.deployment.replicas	Minimum: 1	—
spec.fernet.maxActiveKeys	Minimum: 3	—
spec.credentialKeys.maxActiveKeys	Minimum: 3	—
spec.autoscaling.maxReplicas	Minimum: 1	—
spec.autoscaling.minReplicas	Minimum: 1	—
spec.autoscaling.targetCPUUtilization	Range: 1–100	—
spec.autoscaling.targetMemoryUtilization	Range: 1–100	—
spec.uwsgi.processes	Minimum: 1	—
spec.uwsgi.threads	Minimum: 1	—
spec.uwsgi.harakiri	Minimum: 1	—
spec.uwsgi.httpKeepAliveTimeout	Minimum: 1	—
spec.deployment.terminationGracePeriodSeconds	Minimum: 10	—
spec.deployment.preStopSleepSeconds	Minimum: 0	—
spec.gateway.hostname	MinLength: 1	(empty string rejected by API server)
spec.gateway.parentRef.name	MinLength: 1	(empty string rejected by API server)

Known limitation: spec.uwsgi.processes and spec.uwsgi.threads have no upper-bound validation. A user could set an extremely high value (e.g., processes: 10000), causing the Deployment to request more workers than the node can sustain. A +kubebuilder:validation:Maximum marker should be added once the team agrees on a safe ceiling. Track this as a follow-up product decision.

---

AutoscalingSpec

Configures horizontal pod autoscaling for the Keystone API Deployment. This is a pointer field (*AutoscalingSpec) on KeystoneSpec — when nil, no HPA is created and the HPAReady condition is set to True with reason HPANotRequired. When set, a HorizontalPodAutoscaler (autoscaling/v2) is created targeting the {name} Deployment. Removing the field deletes the existing HPA.

While spec.autoscaling is set, the operator leaves the Deployment's .spec.replicas unmanaged (nil) so the HPA owns the replica count and the reconciler does not reset it on each pass. spec.deployment.replicas is then used only as the initial replica count and as the minReplicas default when autoscaling.minReplicas is unset.

Field	Type	Required	Default	Description
minReplicas	*int32	No	spec.deployment.replicas	Lower bound for the number of replicas. Minimum: 1. Defaults to spec.deployment.replicas when unset, allowing the HPA to scale down to the static replica count.
maxReplicas	int32	Yes	—	Upper bound for the number of replicas. Minimum: 1.
targetCPUUtilization	*int32	No*	—	Target average CPU utilization as a percentage. Range: 1–100. At least one of targetCPUUtilization or targetMemoryUtilization must be set.
targetMemoryUtilization	*int32	No*	—	Target average memory utilization as a percentage. Range: 1–100. At least one of targetCPUUtilization or targetMemoryUtilization must be set.

* At least one of targetCPUUtilization or targetMemoryUtilization is required (enforced by CEL XValidation).

HPA Resource Mapping

The HPA created from this spec has the following shape:

HPA Field	Value
metadata.name	{name}
metadata.labels	commonLabels (name, instance, managed-by)
spec.scaleTargetRef.apiVersion	apps/v1
spec.scaleTargetRef.kind	Deployment
spec.scaleTargetRef.name	{name}
spec.minReplicas	autoscaling.minReplicas (or spec.deployment.replicas if unset)
spec.maxReplicas	autoscaling.maxReplicas
spec.metrics	CPU and/or memory Resource metrics based on which targets are set
ownerReferences	Points to the Keystone CR (controller: true)

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  autoscaling:
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilization: 80
    targetMemoryUtilization: 70

---

UWSGISpec

Configures the uWSGI application server parameters for the Keystone API container. This is a pointer field (*UWSGISpec) on KeystoneSpec — when nil, the reconciler uses hardcoded defaults (processes=2, threads=1, httpKeepAlive=true) and the webhook does not inject a default UWSGISpec. When set (even as uwsgi: {}), the webhook defaults zero-valued sub-fields and the reconciler reads from the spec.

Field	Type	Required	Default	Description
processes	int32	No	2	Number of uWSGI worker processes. Minimum: 1. Maps to --processes in the container command.
threads	int32	No	1	Number of threads per uWSGI worker process. Minimum: 1. Maps to --threads in the container command.
httpKeepAlive	*bool	No	true	Enables the --http-keepalive flag on the uWSGI process. When false, the flag is omitted. A nil-preserving pointer: unset means the documented default (true) is restored by the webhook. See HTTPKeepAlive defaulting.
harakiri	*int32	No	nil (flag omitted)	Caps the per-request worker lifetime (seconds) via --harakiri. Minimum: 1. The webhook additionally enforces harakiri < terminationGracePeriodSeconds − preStopSleepSeconds so the worst-case per-request kill fits inside the shutdown drain window. See Graceful-termination fields.
httpKeepAliveTimeout	*int32	No	nil (flag omitted)	Idle timeout (seconds) for keep-alive connections via --http-keepalive-timeout. Minimum: 1. Emitted only when httpKeepAlive=true (the webhook rejects a non-nil timeout combined with httpKeepAlive=false). Recommended to set ≤ preStopSleepSeconds so idle sockets close before SIGTERM reaches uWSGI. See Graceful-termination fields.

Deployment Command Mapping

The reconciler's uwsgiCommand() helper constructs the container command from spec.uwsgi (or defaults when nil). Fixed flags are always present regardless of configuration:

Command Flag	Source
uwsgi	Binary name (always first)
--http :5000	Fixed — Keystone API listen port
--http-keepalive	Included when httpKeepAlive is true (or default); omitted when false
--wsgi-file /var/lib/openstack/bin/keystone-wsgi-public	Fixed — Keystone WSGI entry point
--master	Fixed — enables uWSGI master process
--lazy-apps	Fixed — loads apps in each worker after fork
--need-app	Fixed — exits if no WSGI app is found
--processes <N>	spec.uwsgi.processes (default: 2)
--threads <N>	spec.uwsgi.threads (default: 1)
--pyargv=--config-dir=/etc/keystone/keystone.conf.d/	Fixed — passes config directory to Keystone

HTTPKeepAlive Defaulting

httpKeepAlive is a nil-preserving *bool, so "unset" is distinguishable from an explicit false. The defaulting webhook restores the documented default (true) only when the pointer is nil, and preserves an explicit true or false verbatim. This means:

• uwsgi: {} → processes=2, threads=1, httpKeepAlive=true (all webhook-defaulted)
• uwsgi: {processes: 4} → processes=4, threads=1, httpKeepAlive=true
• uwsgi: {httpKeepAlive: false} → httpKeepAlive stays false (explicit value is preserved)

Bypass paths (e.g., kubectl create against a cluster where the admission webhooks are temporarily unavailable) may leave httpKeepAlive nil. The uwsgiCommand function in the controller falls back to the same default (true) when the pointer is nil, so keep-alive stays enabled even in bypass scenarios.

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  uwsgi:
    processes: 4
    threads: 4
    httpKeepAlive: false

---

Graceful-termination fields

Five CR fields control the shutdown envelope applied during Keystone rolling updates — spec.deployment.terminationGracePeriodSeconds, spec.deployment.preStopSleepSeconds, spec.deployment.strategy, spec.uwsgi.harakiri, and spec.uwsgi.httpKeepAliveTimeout. Each field is listed in its owning section (top-level KeystoneSpec or UWSGISpec); this section consolidates their semantics, interaction rules, and defaulting behavior.

Field Summary

Field	Type	Default	Minimum	Effect
spec.deployment.terminationGracePeriodSeconds	*int64	30	10	PodSpec terminationGracePeriodSeconds — total envelope between SIGTERM and SIGKILL.
spec.deployment.preStopSleepSeconds	*int64	5	0	Sleep duration of the preStop hook (/bin/sh -c 'sleep <n>'). Covers the EndpointSlice / kube-proxy propagation window.
spec.deployment.strategy	*appsv1.DeploymentStrategy	RollingUpdate(maxSurge=1, maxUnavailable=0)	—	Deployment rollout strategy. Default guarantees surge-before-remove so capacity never dips below spec.deployment.replicas.
spec.uwsgi.harakiri	*int32	unset (flag omitted)	1	Per-request worker kill bound (--harakiri <n>). Prevents a single stuck request from holding a worker past the shutdown envelope.
spec.uwsgi.httpKeepAliveTimeout	*int32	unset (flag omitted)	1	Idle keep-alive socket timeout (--http-keepalive-timeout <n>). Only emitted when httpKeepAlive=true.

Interaction Rules Enforced by the Webhook

The validating webhook enforces the following cross-field invariants so that the shutdown envelope is always internally consistent. Violations are returned as field.Invalid errors.

Rule
preStopSleepSeconds < terminationGracePeriodSeconds (with nil pointers resolved to defaults 5 / 30)
harakiri < terminationGracePeriodSeconds − preStopSleepSeconds (only when harakiri is set)
httpKeepAliveTimeout requires httpKeepAlive=true
strategy.type=Recreate must not carry a strategy.rollingUpdate block

Operator Guidance (not webhook-enforced)

• httpKeepAliveTimeout ≤ preStopSleepSeconds — when the keep-alive timeout exceeds the preStop sleep, a client may still hold a warm keep-alive socket to the Pod when SIGTERM fires, returning a connection reset on the client's next request. Tune httpKeepAliveTimeout at or below preStopSleepSeconds to close idle sockets before the kubelet signals uWSGI and preserve the zero-reset SLO. The webhook does not enforce this because slow clients may legitimately need a longer keep-alive window at the cost of occasional resets on rollout.

Reconciler Fallbacks

The reconciler applies internal defaults when the CR field is nil so older CRs continue to reconcile without the fields set:

Field	Fallback when nil
spec.deployment.terminationGracePeriodSeconds	PodSpec receives 30
spec.deployment.preStopSleepSeconds	preStop command is sleep 5
spec.deployment.strategy	RollingUpdate with maxUnavailable=0, maxSurge=1
spec.uwsgi.harakiri	--harakiri flag is omitted
spec.uwsgi.httpKeepAliveTimeout	--http-keepalive-timeout flag is omitted

These fallbacks live in internal/controller/reconcile_deployment.go (terminationGracePeriodSeconds, preStopSleepCommand, deploymentStrategy, uwsgiCommand) and are the single source of truth for the no-op upgrade path.

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
    terminationGracePeriodSeconds: 60
    preStopSleepSeconds: 10
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  uwsgi:
    processes: 4
    threads: 4
    httpKeepAlive: true
    httpKeepAliveTimeout: 10
    harakiri: 45

---

LoggingSpec

Configures oslo.log output for the Keystone API container. This is a pointer field (*LoggingSpec) on KeystoneSpec. When nil, the defaulting webhook materializes a baseline LoggingSpec{Format: "text", Level: "INFO", Debug: false} (no per-logger overrides) so downstream reconciler code never sees a nil pointer — matching the documented production baseline (stdout/stderr, oslo.log line format, no debug noise). When set (even as logging: {}), the webhook partially fills zero-valued sub-fields with the same baseline values and the validating webhook enforces the enum constraints described below.

The reconciler always emits [DEFAULT] use_stderr=true and [DEFAULT] debug=<spec.logging.debug> into keystone.conf. When spec.logging.format == "json", an additional logging.conf ConfigMap entry is rendered (oslo.log JSON formatter wired to a stderr StreamHandler) and [DEFAULT] log_config_append=/etc/keystone/keystone.conf.d/logging.conf is appended; toggling format back to text drops the logging.conf key. [DEFAULT] use_stderr=true is an operator-owned default. A spec.extraConfig override of it is honored — the value is rendered — but surfaces through the informational ExtraConfigHealthy status condition (Reason=OwnedKeysOverridden, with a message noting that container logs will no longer reach kubectl logs) and the gated ExtraConfigOwnedKeyOverride Warning event. The condition is intentionally not aggregated into the top-level Ready condition so an explicit operator override is honoured rather than blocking the rollout. See keystone-events.md, Configuration for the full event/condition contract.

Field	Type	Required	Default	Description
format	string	No	text	On-wire layout of oslo.log records. text emits the standard oslo.log line format; json emits one JSON object per record for direct ingest by Loki/OpenSearch. Enforced as +kubebuilder:validation:Enum=text;json.
level	string	No	INFO	Root logger level applied to oslo.log. One of DEBUG, INFO, WARNING, ERROR, CRITICAL. Enforced as +kubebuilder:validation:Enum=DEBUG;INFO;WARNING;ERROR;CRITICAL.
debug	*bool	No	false	Toggles oslo.log [DEFAULT] debug=true. Independent of level because oslo.log gates several extra-verbose code paths on the debug flag specifically (SQL echo, auth-backend tracing). A nil-preserving pointer: unset means the documented default (false) is restored by the webhook.
perLoggerLevels	map[string]string	No	nil	Overrides the level of named loggers, mirroring oslo.log's default_log_levels. Each value must be one of DEBUG/INFO/WARNING/ERROR/CRITICAL and every logger name must be non-empty — enforced by CRD CEL XValidation rules (a plain enum on additionalProperties is not expressible in CRD v1, so the value constraint is written as an in [...] CEL rule) and by the validating webhook. Rendered into [DEFAULT].default_log_levels in deterministic alphabetical order to keep ConfigMap content-hashes stable across reconciles.

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  logging:
    format: json
    level: INFO
    debug: false
    perLoggerLevels:
      sqlalchemy.engine: WARNING
      keystone.middleware: DEBUG

---

FernetSpec

Configures Fernet token key rotation.

Field	Type	Required	Default	Description
rotationSchedule	string	No	"0 0 * * 0"	Cron expression (5-field standard format) for key rotation. Validated by robfig/cron/v3 ParseStandard.
maxActiveKeys	int32	No	3	Maximum number of active Fernet keys. Minimum: 3.
suspend	bool	No	false	Suspends the Fernet rotation CronJob without deleting it. Maps to the CronJob spec.suspend field. Set true to pause key rotation during an incident; the schedule is unchanged so resuming is churn-free.

---

CredentialKeysSpec

Configures credential-key rotation. Credential keys encrypt the application-credential passwords stored in the database. Rotation uses the same 32-byte base64url format as Fernet but runs keystone-manage credential_migrate after generating a new primary key so that existing rows stay readable after the old key is purged. Rotation is driven by a CronJob that pushes the regenerated key set back to the {name}-credential-keys Secret via a minimally-scoped ServiceAccount. The Secret is also mirrored to OpenBao through a PushSecret.

Field	Type	Required	Default	Description
rotationSchedule	string	No	"0 0 * * 0"	Cron expression (5-field standard format). Validated by robfig/cron/v3 ParseStandard in the webhook.
maxActiveKeys	int32	No	3	Maximum number of active credential keys. Minimum: 3. Exposed to keystone-manage via the OS_credential__max_active_keys environment variable on the rotation CronJob.
suspend	bool	No	false	Suspends the credential rotation CronJob without deleting it. Maps to the CronJob spec.suspend field. Set true to pause key rotation during an incident; the schedule is unchanged so resuming is churn-free.

---

TrustFlushSpec

Configures periodic purging of expired trust delegations. This is a pointer field (*TrustFlushSpec) on KeystoneSpec, but on a webhook-enabled cluster it is default-on: the defaulting webhook materializes {schedule: "0 * * * *", suspend: false} whenever the field is omitted (or patched to null), so the operator always creates a CronJob named {name}-trust-flush running keystone-manage trust_flush and the TrustFlushReady condition is set to True with reason TrustFlushReady.

There is no nil-back path on a webhook-enabled cluster — a kubectl patch ... 'spec/trustFlush'='null' round-trips through admission and is re-materialized, preserving the existing CronJob (no delete/recreate). To pause the schedule without deleting the CronJob, set suspend: true — the resource and TrustFlushReady=True condition are preserved while suspended.

The pointer shape is retained for envtest fixtures and other webhook-less clusters where the defaulting webhook is not wired up. In that legacy bypass posture the reconciler logs a warning, deletes any existing CronJob, and sets TrustFlushReady=True with reason TrustFlushNotRequired and a message identifying the bypass — see reconcileTrustFlush.

For brownfield CRs that omit spec.trustFlush at the time of an operator upgrade and the recommended pre-upgrade actions on clusters with very large trust tables, see Default-on Trust Flush at Upgrade Time in the upgrade-flow reference.

Field	Type	Required	Default	Description
schedule	string	No	"0 * * * *"	Cron expression (5-field standard format) for trust flush. Validated by robfig/cron/v3 ParseStandard. Default is hourly.
suspend	bool	No	false	Suspends the CronJob without deleting it. Maps to the CronJob spec.suspend field. The CronJob resource and TrustFlushReady=True condition are preserved while suspended.
args	[]string	No	nil	Additional CLI flags appended after keystone-manage trust_flush. Flags such as --keystone-user, --keystone-group, --date are passed through verbatim.

CronJob Resource Mapping

The CronJob created from this spec has the following shape. Field values sourced from trustFlush.* are populated either by the user or — when the field was omitted on submission — by the defaulting webhook, which materializes {schedule: "0 * * * *", suspend: false} before the reconciler ever sees the object.

CronJob Field	Value
metadata.name	{name}-trust-flush
metadata.labels	commonLabels (name, instance, managed-by)
spec.jobTemplate.spec.template.metadata.labels	commonLabels + app.kubernetes.io/component=trust-flush, which keeps the pods out of the API Service
spec.schedule	trustFlush.schedule (webhook-defaulted to "0 * * * *" when omitted)
spec.suspend	&trustFlush.suspend (pointer to bool; webhook-defaulted to false when omitted)
spec.jobTemplate.spec.template.spec.restartPolicy	OnFailure
Container name	trust-flush
Container image	{spec.image.repository}:{spec.image.tag}
Container command	["keystone-manage", "--config-dir=/etc/keystone/keystone.conf.d/", "trust_flush"] + args
Container securityContext	restrictedSecurityContext() (PSS Restricted)
ownerReferences	Points to the Keystone CR (controller: true)

Volume Mounts

The trust-flush container mounts the same configuration and key volumes as the Deployment, all read-only:

Volume Name	Mount Path	Source	ReadOnly
config	/etc/keystone/keystone.conf.d/	ConfigMap {configMapName}	Yes
fernet-keys	/etc/keystone/fernet-keys	Secret {name}-fernet-keys	Yes
credential-keys	/etc/keystone/credential-keys	Secret {name}-credential-keys	Yes

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  trustFlush:
    schedule: "30 2 * * 0"
    args: ["--date", "2024-01-01"]

---

NetworkPolicySpec

Configures network isolation for the Keystone API pods. This is a pointer field (*NetworkPolicySpec) on KeystoneSpec — when nil, no NetworkPolicy is managed and the NetworkPolicyReady condition is set to True with reason NetworkPolicyNotRequired. When set, the operator creates a NetworkPolicy that restricts ingress on TCP 5000 to the declared sources — plus the operator's own namespace (so the operator health check can reach the API) and, when spec.gateway is set, the gateway namespace — and auto-derives egress rules for DNS, the kube-apiserver, the database, and the cache. Removing the field deletes the NetworkPolicy on the next reconcile.

Field	Type	Required	Default	Description
ingress	[]NetworkPolicyIngressSource	Yes	—	Sources allowed to reach Keystone API on TCP 5000. At least one entry required (enforced by CEL and webhook).
additionalEgress	[]networkingv1.NetworkPolicyEgressRule	No	nil	Extra egress rules appended after the auto-derived rules. The auto-derived rules already cover the database and cache in both managed and brownfield modes, so reserve this for external integrations beyond those backends.

NetworkPolicyIngressSource

Field	Type	Required	Description
namespaceSelector	metav1.LabelSelector	Yes	Label selector for source namespaces, supporting both matchLabels and set-based matchExpressions. All pods in matching namespaces may reach Keystone on TCP 5000 unless podSelector narrows the set.
podSelector	metav1.LabelSelector	No	Optional label selector restricting allowed pods within the selected namespaces (AND logic within a single peer). Supports matchLabels and matchExpressions.

Auto-added Ingress peers

Beyond the declared ingress sources, the operator appends these ingress peers on the TCP 5000 rule:

Peer	Trigger	Notes
Operator namespace	Operator namespace resolvable	Selected by kubernetes.io/metadata.name. The operator's health check GETs the Keystone Service on TCP 5000; without this peer KeystoneAPIReady would flip False permanently for a healthy deployment. Omitted only when the operator namespace cannot be determined.
Gateway namespace	spec.gateway set	Selected by kubernetes.io/metadata.name of parentRef.namespace (the CR namespace when empty), so the Gateway data plane can reach the API.

Auto-derived Egress

The operator appends the following egress rules before additionalEgress. All rules are port-only — the destination is unrestricted (tightening to backend pod labels is deferred). Rule order is deterministic: DNS, apiserver, database, cache.

Rule	Trigger	Notes
DNS UDP+TCP 53	Always	Destination is unrestricted because CoreDNS may run in any namespace (e.g. NodeLocal DNSCache).
kube-apiserver TCP 443+6443	Always	The fernet/credential/admin-password rotation CronJob pods share this policy's pod selector and PATCH the rotated keys back to a Secret via kubernetes.default.svc; without apiserver egress every scheduled rotation stops at its first run. 443 is the ClusterIP Service port; 6443 covers the post-DNAT kube-apiserver pod port on enforcing CNIs.
Database TCP dbPort	Always	Port from spec.database.port (default 3306), emitted in both managed (database.clusterRef) and brownfield (database.host) modes — the readiness probe TCP-connects to exactly this port, so an enforcing CNI would otherwise depool every pod.
Cache TCP (derived)	cache.clusterRef or cache.servers set	Managed mode → 11211; brownfield mode → the distinct ports parsed from the cache.servers host:port strings (default 11211 when a server omits the port).

A defensive guard in the reconciler refuses to create a NetworkPolicy with an empty ingress list, even if CEL validation was bypassed (stored objects, disabled webhooks, direct etcd writes) — the operator fails closed rather than open.

Example

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  # ... required fields ...
  networkPolicy:
    ingress:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openstack
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: ingress-gateway
        podSelector:
          matchLabels:
            app.kubernetes.io/name: envoy
    additionalEgress:
      - to:
          - ipBlock:
              cidr: 10.0.0.0/24
        ports:
          - protocol: TCP
            port: 443

---

GatewaySpec

GatewaySpec is a shared type from internal/common/types (imported as commonv1), the single source of truth for the Gateway API HTTPRoute shape. Both the Keystone operator and the c5c3 ControlPlane reuse it instead of maintaining their own field-for-field copies; keystonev1alpha1.GatewaySpec is a type alias to commonv1.GatewaySpec, so existing references compile unchanged. The field table below is the canonical reference that the c5c3 ControlPlane CRD doc links back to. The reconciler behavior described in this section is Keystone-specific.

Configures external exposure of the Keystone API via a Gateway API HTTPRoute. This is a pointer field (*GatewaySpec) on KeystoneSpec — when nil, no HTTPRoute is created and the HTTPRouteReady condition is set to True with reason HTTPRouteNotRequired. When set, an HTTPRoute (from gateway.networking.k8s.io/v1) is created in the Keystone CR's namespace, attached to the referenced pre-existing Gateway, and pointing to the {name} Service on port 5000. Removing the field deletes the existing HTTPRoute.

The operator plays the application-developer role in the Gateway API model: it manages only the HTTPRoute. The referenced Gateway (and its GatewayClass) are platform-team concerns and must be pre-provisioned — this operator does not create or reconcile them. Cross-namespace parentRef references additionally require a ReferenceGrant in the target namespace, which is out of scope for this operator.

Gateway API CRD prerequisite: the gateway.networking.k8s.io/v1 HTTPRoute CRD must be installed in the cluster before the Keystone operator starts. The operator probes for the CRD at startup (via the manager RESTMapper); when the CRD is missing it disables the HTTPRoute watch so Keystone CRs without spec.gateway still reconcile, and reports HTTPRouteReady=False with reason GatewayAPINotInstalled for any CR that sets spec.gateway. A Keystone with spec.targetClusterRef is re-probed against its target cluster on every reconcile, so installing Gateway API there heals the CR on the next pass; restoring the HTTPRoute drift watch on that cluster still needs a rotation of its kubeconfig Secret (see Target Clusters). For a Keystone that names no target cluster the startup probe is the whole answer, and installing the CRD after the operator has started requires restarting the operator for the watch to become active. The quickstart stack (make deploy-infra) installs the upstream Gateway API standard CRDs for this reason; the pinned version is set via GATEWAY_API_VERSION in hack/deploy-infra.sh and tracks sigs.k8s.io/gateway-api in operators/keystone/go.mod. That pin is v1.6.1 (standard channel), which is also the floor for the route timeout below: spec.rules[].timeouts reached the HTTPRoute schema after the versions the stack shipped earlier, and an older CRD prunes the stanza silently, leaving the implementation default in force with nothing logged or rejected. When live CRDs carry a gateway.networking.k8s.io/bundle-version annotation that differs from the pin, the script upgrades them in place; a live set without that annotation is warned about and skipped, since it cannot be compared.

Field	Type	Required	Default	Description
parentRef	GatewayParentRefSpec	Yes	—	Gateway the HTTPRoute attaches to.
hostname	string	Yes	—	Externally reachable hostname (SNI / Host header) matched by the HTTPRoute. Used for both route hostname matching and deriving status.endpoint (https://{hostname}/v3). Minimum length: 1.
path	string	No	"/"	URL path prefix matched by the HTTPRoute. The reconciler applies the default when the field is empty. Uses PathPrefix match type.
annotations	map[string]string	No	nil	Annotations passed through verbatim to the HTTPRoute metadata.annotations, allowing implementation-specific configuration (rate limits, CORS). The route timeout is operator-managed and not annotation-driven — see spec.rules[0].timeouts.request below. Operator-managed labels are preserved: user annotations do not shadow them.

GatewayParentRefSpec

References the pre-existing Gateway that the operator attaches the HTTPRoute to. Like GatewaySpec, this is a shared commonv1 type reused by both the Keystone operator and the c5c3 ControlPlane; keystonev1alpha1.GatewayParentRefSpec aliases commonv1.GatewayParentRefSpec.

Field	Type	Required	Default	Description
name	string	Yes	—	Gateway resource name. Minimum length: 1.
namespace	string	No	CR namespace	Namespace of the referenced Gateway. When empty, the Gateway is assumed to live in the Keystone CR's namespace. Cross-namespace references require a ReferenceGrant.
sectionName	string	No	""	Targets a specific listener on the Gateway (e.g., "https") when the Gateway defines multiple listeners. When empty, the HTTPRoute attaches to all compatible listeners.

HTTPRoute Resource Mapping

The HTTPRoute created from this spec has the following shape (gateway.networking.k8s.io/v1, Kind: HTTPRoute):

HTTPRoute Field	Value
metadata.name	{name} (matches the backend Service, Deployment, HPA, NetworkPolicy naming)
metadata.namespace	Keystone CR namespace
metadata.labels	commonLabels (name, instance, managed-by)
metadata.annotations	Merged from spec.gateway.annotations
spec.parentRefs[0].name	spec.gateway.parentRef.name
spec.parentRefs[0].namespace	spec.gateway.parentRef.namespace when non-empty; omitted otherwise
spec.parentRefs[0].sectionName	spec.gateway.parentRef.sectionName when non-empty; omitted otherwise
spec.hostnames[0]	spec.gateway.hostname
spec.rules[0].matches[0].path.type	PathPrefix
spec.rules[0].matches[0].path.value	spec.gateway.path (or "/" when empty)
spec.rules[0].backendRefs[0].kind	Service
spec.rules[0].backendRefs[0].name	{name}
spec.rules[0].backendRefs[0].port	5000
spec.rules[0].timeouts.request	Not rendered; the gateway implementation's default applies. Glance is the exception, and renders "4h" (see below)
ownerReferences	Points to the Keystone CR (controller: true) — enables garbage collection

The route timeout is operator-managed on every service that renders an HTTPRoute, and no CRD exposes it. Keystone and Horizon omit the stanza and inherit the implementation's default, 15 s on the Envoy Gateway the kind stack deploys, which their short identity and dashboard requests fit inside. The Glance route raises it to timeouts.request: "4h", because an image upload or import legitimately streams for hours.

It is raised, not disabled. "0s" is the Gateway API spelling of no timeout, and the rule it would land on matches a bare / prefix, so it covers every path of the service behind the route. The route timeout is then the only request-duration cap in front of the backend: a stream idle timeout resets on every byte, so it never fires for a client that trickles one. A finite bound far above any legitimate transfer still caps how long a stalled request holds what it holds, while letting a multi-gibibyte transfer finish. It bounds duration only, not how many requests may hold a worker at once; see Gateway requirements for the upload path for what that leaves uncovered. Rendering the stanza needs the Gateway API CRD version named in the prerequisite above.

status.endpoint Derivation

status.endpoint reflects the externally reachable Keystone API URL and is recomputed on every reconcile:

spec.gateway	status.endpoint Value
nil	http://{name}.{namespace}.svc.cluster.local:5000/v3 (cluster-local fallback)
Set	https://{hostname}/v3 — HTTPS is fixed because Gateways are the public-ingress hop and terminate TLS

status.endpoint does not include spec.gateway.path. The /v3 suffix is appended unconditionally because Keystone API v3 is served at that fixed path; the PathPrefix match on the HTTPRoute routes any prefix under spec.gateway.path to the backend. spec.publicEndpoint (if set) still takes precedence over the gateway-derived URL for the --bootstrap-public-url argument passed to keystone-manage bootstrap; the precedence is unchanged from earlier behavior.

Interaction with NetworkPolicy

When both spec.gateway and spec.networkPolicy are configured, the operator automatically appends an extra ingress peer to the managed NetworkPolicy so that the Gateway's data-plane pods can reach Keystone on TCP 5000:

• Peer selector: namespaceSelector matching kubernetes.io/metadata.name={gatewayNamespace}. The gateway data plane's pod labels are implementation-specific (Kong/Envoy/NGINX/…) and not known to this operator, so selection is by entire gateway namespace rather than by pod labels.
• Namespace source: spec.gateway.parentRef.namespace when set; otherwise the Keystone CR's own namespace (mirroring the ParentRef lookup semantics).
• Removal: Clearing spec.gateway removes the extra peer on the next reconcile.
• networkPolicy nil: When spec.networkPolicy is nil, no NetworkPolicy is managed at all and no extra peer is added (gateway-only deployments rely on the namespace's default network policy or absence thereof).

Example — Basic Gateway Exposure

kind Quick Start note: a ready-made Gateway/openstack-gw ships in the kind overlay (deploy/kind/base/openstack-gateway.yaml) and is reachable on the host at https://keystone.127-0-0-1.nip.io/v3 — see the Quick Start (Extended) / Access Keystone section. On a Quick Start cluster, setting spec.gateway.parentRef.name: openstack-gw plus hostname: keystone.127-0-0-1.nip.io makes status.endpoint = https://keystone.127-0-0-1.nip.io/v3 actually resolve from your workstation — no /etc/hosts edit, no kubectl port-forward. Production overlays do not ship openstack-gw; operators pick their own Gateway implementation and parent reference there.

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  deployment:
    replicas: 3
  image:
    repository: c5c3/keystone
    tag: "2025.1"
  # ... other required fields ...
  gateway:
    parentRef:
      name: public-gateway
      namespace: istio-ingress
      sectionName: https
    hostname: keystone.example.com
    path: /identity
    annotations:
      konghq.com/plugins: rate-limit-sha

Resulting status.endpoint: https://keystone.example.com/v3.

Example — Gateway with NetworkPolicy

apiVersion: keystone.openstack.c5c3.io/v1alpha1
kind: Keystone
metadata:
  name: keystone
  namespace: openstack
spec:
  # ... required fields ...
  gateway:
    parentRef:
      name: public-gateway
      namespace: istio-ingress
    hostname: keystone.example.com
  networkPolicy:
    ingress:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openstack

The operator-managed NetworkPolicy allows ingress from:

1. The openstack namespace (user-declared).
2. The istio-ingress namespace (auto-added because spec.gateway is set).

---

TopologySpreadConstraints

spec.deployment.topologySpreadConstraints attaches scheduler spread hints to the Keystone API Deployment's pod template. Uses the upstream corev1.TopologySpreadConstraint type verbatim, except that the webhook restricts labelSelector to exact matchLabels matching the Deployment selector (see below).

spec.deployment.topologySpreadConstraints	Effect
nil (unset)	Operator injects two defaults: topology.kubernetes.io/zone and kubernetes.io/hostname, both MaxSkew=1 with ScheduleAnyway, selecting pods via app.kubernetes.io/name=keystone + app.kubernetes.io/instance={name}.
[] (empty slice)	Defaults disabled; no spread constraints configured. Explicit opt-out.
Non-empty slice	User value is applied verbatim; no defaults merged.

Webhook Constraint

Each entry must set labelSelector.matchLabels equal to the Deployment selector (app.kubernetes.io/name=keystone, app.kubernetes.io/instance={CR name}). matchExpressions is rejected. This prevents constraints that widen or narrow beyond the Deployment's intent, which would otherwise silently produce wrong spread behavior.

Example

spec:
  # ... required fields ...
  deployment:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: keystone
            app.kubernetes.io/instance: keystone

---

PriorityClassName

spec.deployment.priorityClassName (pointer) passes through to pod.spec.priorityClassName on the Keystone API pods. Uses the standard scheduling.k8s.io/v1 PriorityClass resource model.

Value	Effect
nil	No priority class is configured; the cluster default applies.
"" (empty string)	No priority class — explicit opt-out, useful when clearing a previously set value via kubectl patch.
Non-empty string	Value is written to the Deployment PodSpec. The webhook performs a direct (uncached) cluster-scoped Get of the PriorityClass at admission time and rejects unknown names with field.NotFound.

The rotation CronJobs (Fernet, credential) reuse the same priorityClassName to stay co-scheduled with the API pods.

---

FederationSpec

Carries the Keystone-side federation knobs. Federation itself is activated by attaching a federation-typed KeystoneIdentityBackend (type: OIDC) — not by this block: when at least one OIDC backend is projected, the operator injects the mod_auth_openidc reverse-proxy sidecar, binds uWSGI to localhost (with the federation-sized --buffer-size), and switches the Service targetPort to the proxy. This spec only configures how that sidecar runs.

Field	Type	Required	Default	Description
proxyImage	*ImageSpec	No	nil	The Apache/mod_auth_openidc sidecar image. Standalone Keystone installations must set it (mirroring the required spec.image); the managed ControlPlane path projects the ghcr.io/c5c3/keystone-federation-proxy default. When a federation backend is attached and no proxy image is configured, the backends stay pending with a FederationProxyImageMissing Warning — no hidden default is assumed. The webhook rejects a set proxyImage without a repository.
trustedDashboards	[]string	No	nil	Dashboard origins Keystone will POST a WebSSO token back to after a successful federated login. Keystone matches the origin the dashboard sends verbatim, so each entry must reproduce it exactly — scheme, host, non-default port, and the trailing slash (e.g. https://horizon.example.com/auth/websso/). Rendered as repeated [federation] trusted_dashboard lines, one per origin (an oslo MultiStrOpt). Unlike proxyImage it is independent of an attached backend: the [federation] section renders as soon as an origin is declared. The managed ControlPlane path projects its Horizon child's origin; standalone installations set it directly. Max 8 entries, each matching ^https?://[^\s]*$ — the pattern is anchored at both ends because entries render into keystone.conf unescaped, and RE2 anchors ^ at start-of-text, so a prefix-only pattern would let an embedded newline inject a second INI option. The webhook rejects duplicates and rejects declaring trusted_dashboard in both this field and spec.extraConfig (extraConfig wins the merge, which would silently drop the typed list). An http:// origin is accepted but raises an admission warning: Keystone POSTs the unscoped WebSSO token — a bearer token good for the user's full API privileges — to this origin, so cleartext hands it to any on-path observer.

LDAP/AD-backed domains are not part of this block: they ship as KeystoneIdentityBackend CRs too (type: LDAP), where one CR per domain describes the connection, tree layout, and attribute mapping. Both types aggregate into the Keystone CR's IdentityBackendsReady condition.

---

BootstrapSpec

Configures the initial Keystone bootstrap.

Field	Type	Required	Default	Description
adminUser	string	No	"admin"	Admin username for the bootstrap. Immutable after create (CEL transition rule): re-bootstrapping with a different admin user duplicates or strands catalog entries.
adminPasswordSecretRef	SecretRefSpec	Yes	—	Secret containing the admin password.
region	string	No	"RegionOne"	Keystone region name. Immutable after create (CEL transition rule): changing the region strands catalog entries under the old region.
publicEndpoint	string	No	Cluster-local service DNS	Externally routable Keystone endpoint URL. Used for the --bootstrap-public-url argument passed to keystone-manage bootstrap. Required by external clients (CLI users, Horizon, federation partners) that cannot resolve the cluster-local service DNS. When set, it must be an HTTP(S) URL (+kubebuilder:validation:Pattern=^https?://), enforced unconditionally by the CRD schema; when spec.gateway is also set the webhook additionally requires the host to equal spec.gateway.hostname.

PasswordRotationSpec

Configures scheduled admin-password rotation. Unlike TrustFlushSpec, the defaulting webhook does not materialize this block when it is absent — scheduled rotation is strictly opt-in, so upgrading a CR that never set passwordRotation never silently enables it. The webhook only fills the leaf defaults once enabled is true; the +kubebuilder:default markers below remain as defense-in-depth for callers that bypass the webhook. The rotated password is pushed to the per-CR OpenBao key bootstrap/{namespace}/{name}/admin, so enabling rotation on multiple CRs does not collide on a shared object.

Field	Type	Required	Default	Description
enabled	bool	No	false	Turns on scheduled admin-password rotation. Disabling it tears down every rotation resource.
schedule	string	No	"0 0 1 * *"	Cron expression controlling when a new admin password is generated (monthly at midnight on the 1st).
suspend	bool	No	false	Pauses the CronJob without deleting it or any sibling resource, matching TrustFlushSpec.suspend semantics.
passwordLength	int32	No	32	Length of the generated password. Minimum 24 (+kubebuilder:validation:Minimum=24).

---

KeystoneStatus

Field	Type	Description
conditions	[]metav1.Condition	Latest available observations of the Keystone state. A listType=map strategic-merge list keyed by type, so concurrent writers (the controller and external tooling) merge per-condition under server-side apply instead of clobbering the whole list.
observedGeneration	int64	The .metadata.generation the controller last reconciled, so a stale status is distinguishable from one reflecting the current spec without scanning conditions.
endpoint	string	Keystone API endpoint URL (set by the controller when ready). Defaults to http://{name}.{namespace}.svc.cluster.local:5000/v3.
installedRelease	string	OpenStack release version currently deployed. Set by the controller after a successful db_sync; reflects the value extracted from spec.image.tag. Left unset when the image is pinned by digest (digest mode disables release tracking).
targetRelease	string	Upgrade target release during an active upgrade. Set while upgradePhase is one of Expanding/Migrating/RollingUpdate/Contracting; cleared after Contracting completes.
upgradePhase	UpgradePhase	Current phase of an active database upgrade. Empty outside upgrades.

The status subresource is enabled via +kubebuilder:subresource:status.

UpgradePhase

UpgradePhase is a string enum (+kubebuilder:validation:Enum=Expanding;Migrating;RollingUpdate;Contracting) representing the current phase of a sequential release upgrade driven by reconcileDatabase. Phase transitions follow the expand-migrate-contract pattern:

Value	Meaning
Expanding	Additive schema migrations running (new columns/tables). Old pods keep serving.
Migrating	Backfill/data-migration jobs running against the expanded schema.
RollingUpdate	New image is rolling out; old and new pods read the expanded schema side-by-side.
Contracting	Destructive schema migrations running (drop old columns/tables) after the rollout completes.

spec.image.tag must be parseable by ParseRelease (YYYY.N or YYYY.N-patch). Sequential upgrades are limited to one minor step (2025.1 → 2025.2) or a year-boundary crossing (2025.2 → 2026.1); downgrades and skip-level upgrades are rejected by the reconciler.

---

Shared Types (from internal/common/types)

The following types are imported as commonv1 from github.com/c5c3/cobaltcore/internal/common/types. They are shared across all CobaltCore operator CRDs.

ImageSpec

Exactly one of tag or digest must be set, enforced by a type-level XValidation rule (and mirrored by the validating webhook). A pinned digest closes the supply-chain gap where a mutable tag can be re-pushed behind a stable name.

Digest-mode disables release tracking. Keystone's release tracking and the expand-migrate-contract upgrade flow key entirely on the image tag. When the image is pinned by digest (no tag), status.installedRelease is left unset and no upgrade is ever detected. The managed ControlPlane path always projects a tag, so it is unaffected.

Field	Type	Required	Description
repository	string	Yes	Container image repository (e.g., c5c3/keystone). Must be non-empty (MinLength=1) and match a permissive OCI reference Pattern (^[a-z0-9]+([._:/-][a-z0-9]+)*$) that accepts registry-host and host:port forms.
tag	string	No (exactly one of tag/digest)	Image tag (e.g., 2025.1). When present, must match the OCI tag grammar Pattern (^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$).
digest	string	No (exactly one of tag/digest)	Immutable content digest (e.g., sha256:<64 hex>). Must match Pattern (^sha256:[a-f0-9]{64}$). Pinning by digest disables release tracking/upgrades.

DatabaseSpec

Field	Type	Required	Description
clusterRef	*corev1.LocalObjectReference	No	Reference to a MariaDB CR (managed mode).
credentialsMode	string	No	How the secretRef credential is provisioned. Static (default when empty) — operator-managed MariaDB User/Grant with a long-lived password. Dynamic — short-lived credentials issued by an external secrets engine; secretRef carries both a username and password, no User/Grant CRs are managed, and no long-lived DB password remains at rest. Enum Static;Dynamic; Dynamic is only valid in managed mode (CEL rule).
host	string	No	Database hostname (brownfield mode). When set, MinLength=1 plus a permissive host Pattern (^[a-zA-Z0-9._:-]+$) that accepts DNS names, IPv4, and IPv6 literals.
port	int32	No	Database port (brownfield mode, default 3306). When set, range Minimum=1/Maximum=65535; omitted (managed mode) leaves it unset.
database	string	Yes	Database name. Constrained to the MySQL identifier set and length: MinLength=1, MaxLength=64, Pattern=^[A-Za-z0-9_]+$. Immutable after create (CEL transition rule): renaming re-points db_sync at a fresh, empty schema and silently orphans the existing data.
secretRef	SecretRefSpec	Yes	Secret with database credentials.
tls	*DatabaseTLSSpec	No	Optional TLS/mTLS configuration. The pointer keeps the field opt-in and non-mutating: a nil tls means plaintext TCP, preserving the previous behavior for all existing CRs.
replicas	int32	No	Number of managed MariaDB replicas provisioned in fresh-create mode (default 3, minimum 1). This field is shared by the c5c3 ControlPlane, whose managed-mode projection derives the MariaDB/Galera topology from it. The keystone operator ignores it: Keystone adopts an existing MariaDB (managed mode references it via clusterRef, brownfield via host) and never provisions its own, so replicas has no effect on a standalone Keystone CR.

Exactly one of clusterRef or host must be set (enforced by CEL validation). The database name and the clusterRef ↔ host mode are immutable after create (CEL transition rules, evaluated only on UPDATE).

DatabaseTLSSpec

Configures opt-in TLS (and mutual TLS) for the Keystone-to-database connection. Referenced as an optional pointer from DatabaseSpec; a nil value preserves the previous plaintext behavior.

The single mode enum is the on/off discriminator. A present tls block means "on" (the defaulting webhook materializes an empty mode to "require"), and TLS is enabled exactly when mode is neither empty nor "disabled". The "disabled" value lets an operator keep the certificate references while turning verification off, without deleting the block.

Field	Type	Required	Default	Description
mode	string	No	"require" (materialized by the defaulting webhook when tls is non-nil and mode is empty)	Verification strength applied to the connection. Enum: disabled, prefer, require, verify-ca, verify-full. disabled turns TLS off (certificate references are ignored); prefer/require encrypt the connection only (no peer verification); verify-ca additionally verifies the server certificate chain against the trusted CA bundle; verify-full additionally verifies that the server hostname matches the certificate identity. When TLS is enabled the operator provisions the client certificate (<name>-db-client), appends the ssl_* DSN parameters, and mounts the certificate material into every workload that opens a connection.
caBundleSecretRef	SecretRefSpec	Yes (when enabled)	—	Secret holding the server CA bundle the client trusts when verifying the database endpoint. Required by both the CRD CEL rule and the validating webhook when TLS is enabled (mode is neither empty nor "disabled").
clientCertSecretRef	SecretRefSpec	Yes (when enabled)	—	Secret holding the client keypair presented to the database for mutual TLS. In managed mode (database.clusterRef set) the operator provisions a cert-manager Certificate into a Secret named <name>-db-client; in brownfield mode (database.host set) the keypair must be supplied out-of-band. Required by both the CRD CEL rule and the validating webhook when TLS is enabled.

Mode → connect-args mapping

The reconciler's reconcile_dbconnection_secret.go appends ssl_* query parameters to the database DSN according to mode. The mapping is implemented by modeToSSLParams in operators/keystone/internal/controller/dbtls_mode.go. The on-pod paths come from the read-only volume db-tls mounted at /etc/keystone/db-tls/:

mode	ssl_ca	ssl_cert	ssl_key	ssl_verify_cert	ssl_verify_identity
prefer	/etc/keystone/db-tls/ca.crt	/etc/keystone/db-tls/tls.crt	/etc/keystone/db-tls/tls.key	—	—
require	/etc/keystone/db-tls/ca.crt	/etc/keystone/db-tls/tls.crt	/etc/keystone/db-tls/tls.key	—	—
verify-ca	/etc/keystone/db-tls/ca.crt	/etc/keystone/db-tls/tls.crt	/etc/keystone/db-tls/tls.key	true	—
verify-full	/etc/keystone/db-tls/ca.crt	/etc/keystone/db-tls/tls.crt	/etc/keystone/db-tls/tls.key	true	true

Parameters are emitted via url.Values.Encode(), which sorts keys lexically — so the resulting query string is deterministic across reconciles regardless of the insertion order shown in this table. Any other mode value is rejected by modeToSSLParams (and earlier by the CRD enum and validating webhook) before the DSN is assembled, so a partially-formed DSN can never reach a workload.

Status condition

reconcileDatabaseTLS reports its outcome via the DatabaseTLSReady status condition using these typed reasons:

Reason	When
NotRequired	spec.database.tls is nil or enabled=false — plaintext connection.
ExternallyManaged	enabled=true but the database is brownfield (spec.database.host set, no clusterRef) — the operator does not own the trust domain and expects the client keypair to be supplied out-of-band via clientCertSecretRef.
CertificatePending	Managed mode; the operator has created the cert-manager Certificate but cert-manager has not yet issued it. The condition is False until issuance completes.
CertificateIssued	Managed mode; the client Certificate is issued into Secret <name>-db-client and ready for mount.
CertificateError	Managed mode; applying the Certificate to the cluster the children are written to failed — a target cluster without cert-manager answers no matches for kind Certificate. The condition is False and the pass returns the error.
CapabilityProbeFailed	The cluster the children are written to could not be asked whether it serves the cert-manager.io/v1 Certificate kind (target API server unreachable, or throttling the discovery request). The condition is False and the pass returns the error, so a DatabaseTLSReady left at its previous verdict cannot report the CR converged at a spec that was never applied.

CacheSpec

Field	Type	Required	Description
clusterRef	*corev1.LocalObjectReference	No	Reference to a Memcached CR (managed mode).
backend	string	Yes	Cache backend (e.g., dogpile.cache.pymemcache).
servers	[]string	No	Cache server endpoints (brownfield mode).

Warning — one memcached per Keystone. Keystone instances backed by different databases must not share a memcached: oslo.cache/dogpile keys carry no per-deployment discriminator (they are built from function name plus arguments only), so two Keystones sharing one memcached read and write the same keys for name-based lookups such as get_project_by_name("admin", "default"). A cache entry written by one instance then resolves to an object UUID that does not exist in the other instance's database, surfacing as spurious 404 Not Found (or 401) responses during token issuance. Point each Keystone's clusterRef / servers at a dedicated memcached, or disable keystone-side caching via spec.extraConfig: cache: {enabled: "false"}. Sharing one memcached is only safe for multiple replicas/servers of the same Keystone deployment.

SecretRefSpec

Field	Type	Required	Description
name	string	Yes	Name of the Kubernetes Secret. Must be a non-empty DNS-1123 subdomain (MinLength=1 plus Pattern=^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$). Tightening the shared type rejects an empty Secret name on every consumer (database/admin/messaging/TLS refs).
key	string	No	Key within the Secret's data.

SecretStoreRefSpec

Selects the External Secrets store a consumer routes its ExternalSecrets (and backup PushSecrets) through. When the ref is nil, the operator defaults to the shared cluster-scoped ClusterSecretStore named openbao-cluster-store, so existing deployments are unchanged. A namespaced SecretStore is always resolved in the consuming CR's own namespace — there is no namespace field.

Field	Type	Required	Description
kind	SecretStoreRefKind (ClusterSecretStore | SecretStore)	No	Which External Secrets store kind name refers to. Enum ClusterSecretStore;SecretStore; defaulted to ClusterSecretStore by the +kubebuilder:default marker.
name	string	Yes	Name of the store. Must be a non-empty DNS-1123 subdomain (MinLength=1 plus Pattern=^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$).

PolicySpec

Field	Type	Required	Description
rules	map[string]string	No	Inline policy rule overrides. Keys are oslo.policy rule names; values are rule definitions. Inline rules take precedence over ConfigMap rules. Both the key and the value of every rule must be non-empty.
configMapRef	*corev1.LocalObjectReference	No	Reference to a ConfigMap containing a policy.yaml key with rule overrides.

When policyOverrides is set on KeystoneSpec, at least one of rules or configMapRef must be provided. Every rules entry must have a non-empty name and a non-empty value — an empty value previously passed admission and reached oslo.policy. All three constraints are enforced by both CEL validation on the shared PolicySpec type and the validating webhook.

PluginSpec

Field	Type	Required	Description
name	string	Yes	Plugin name (e.g., keystone-keycloak-backend).
configSection	string	Yes	INI section name (e.g., keycloak). Must be unique across all plugins — enforced structurally by the CRD (spec.plugins is an x-kubernetes-list-type: map keyed by configSection), so a duplicate is rejected by the API server before the webhook runs.
config	map[string]string	No	Key-value pairs for the plugin's INI section.

MiddlewareSpec

Field	Type	Required	Description
name	string	Yes	Filter name (e.g., audit).
filterFactory	string	Yes	Python entry point (e.g., audit_middleware:filter_factory).
position	PipelinePosition	Yes	Pipeline insertion point: "before" or "after". Constrained by +kubebuilder:validation:Enum=before;after on the shared PipelinePosition type, so any other value is rejected at admission.
config	map[string]string	No	Key-value pairs for the filter section.

GatewaySpec

Gateway API HTTPRoute exposure configuration, promoted into commonv1 and reused by both the Keystone operator and the c5c3 ControlPlane. See GatewaySpec above for the field table and the Keystone-specific HTTPRoute reconciler behavior, and GatewayParentRefSpec for the parent-reference fields.

---

Webhooks

The KeystoneWebhook struct implements both defaulting and validating admission webhooks via the admission.Defaulter[*Keystone] and admission.Validator[*Keystone] interfaces from controller-runtime.

Registration

func (w *KeystoneWebhook) SetupWebhookWithManager(mgr ctrl.Manager) error

Registers both webhooks with the manager using builder.WebhookManagedBy[*Keystone].

Defaulting Webhook

func (w *KeystoneWebhook) Default(_ context.Context, obj *Keystone) error

Sets spec fields to their documented defaults when they carry zero values. Explicit (non-zero) values are never overridden.

Field	Condition	Default Value
spec.deployment.replicas	== 0	3
spec.fernet.maxActiveKeys	== 0	3
spec.credentialKeys.maxActiveKeys	== 0	3
spec.cache.backend	== ""	"dogpile.cache.pymemcache"
spec.bootstrap.adminUser	== ""	"admin"
spec.bootstrap.region	== ""	"RegionOne"
spec.uwsgi.processes	== 0 (when spec.uwsgi is non-nil)	2 — webhook only; when spec.uwsgi is nil, the reconciler applies this default internally.
spec.uwsgi.threads	== 0 (when spec.uwsgi is non-nil)	1 — same nil-pointer caveat as processes.
spec.uwsgi.httpKeepAlive	Field absent from JSON payload	true — the field is a nil-preserving *bool, so the defaulting webhook restores the documented default when the pointer is nil, while preserving an explicit false. See HTTPKeepAlive defaulting.
spec.deployment.resources	== nil or empty (requests and limits both unset)	{requests: {memory: 256Mi, cpu: 100m}, limits: {memory: 512Mi, cpu: 500m}} — ensures Burstable QoS class and enables HPA utilization calculations.
spec.database.tls.mode	spec.database.tls != nil && mode == ""	"require" — DefaultDatabaseTLSMode in keystone_webhook.go. Only materialized when the tls block is explicitly present; the webhook never materializes the block itself.

Not defaulted by the webhook:

• spec.fernet.rotationSchedule, spec.credentialKeys.rotationSchedule, spec.trustFlush.schedule, spec.autoscaling.minReplicas, spec.deployment.topologySpreadConstraints, spec.deployment.priorityClassName — these rely on CRD schema defaults or reconciler-level fallbacks. For topologySpreadConstraints the reconciler distinguishes nil (inject zone+hostname defaults) from [] (opt out), so the webhook must not materialise a struct.
• spec.database.tls itself — the webhook never materializes the tls block. TLS is strictly opt-in, so an upgrade of a previously plaintext CR cannot silently turn encryption on (which would also trigger Certificate provisioning). The webhook only partial-fills tls.mode (empty → require) when the parent block is explicitly present, mirroring the TrustFlush / UWSGI / Logging non-mutating discipline.

Design note: spec.fernet.rotationSchedule is NOT defaulted by the webhook — it relies solely on the Kubebuilder +kubebuilder:default="0 0 * * 0" marker. The webhook uses conditional checks (== 0 / == "") rather than always-set to cooperate with the remaining Kubebuilder +default markers, which also provide schema-level defaults. Both layers are intentional — schema defaults apply at deserialization time, while webhook defaults catch zero values that bypass schema defaults (e.g., explicit replicas: 0).

Validating Webhook

func (w *KeystoneWebhook) ValidateCreate(_ context.Context, obj *Keystone) (admission.Warnings, error)
func (w *KeystoneWebhook) ValidateUpdate(_ context.Context, _, newObj *Keystone) (admission.Warnings, error)
func (w *KeystoneWebhook) ValidateDelete(_ context.Context, _ *Keystone) (admission.Warnings, error)

• ValidateCreate and ValidateUpdate both delegate to the internal validate() method. There are no create-specific or update-specific rules.
• ValidateDelete always returns nil — deletion is unconditionally allowed.

Validation Rules

The validate() method accumulates all errors in a field.ErrorList and returns a single apierrors.NewInvalid error. It does not short-circuit on the first error.

Rule	Field Path	Error Type	Condition
Replicas minimum	spec.deployment.replicas	field.Invalid	replicas < 1. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
Cache mutual exclusivity	spec.cache	field.Invalid	Both clusterRef and servers set, or neither. Defense-in-depth alongside the CEL XValidation rule.
Database mutual exclusivity	spec.database	field.Invalid	Both clusterRef and host set, or neither. Defense-in-depth alongside the CEL XValidation rule.
Database TLS mode out-of-enum	spec.database.tls.mode	field.NotSupported	tls.mode is non-empty but not one of disabled/prefer/require/verify-ca/verify-full. Defense-in-depth alongside the +kubebuilder:validation:Enum marker. Empty mode is tolerated because Default() materializes "require" before validation in the normal admission path.
Database TLS caBundleSecretRef required	spec.database.tls.caBundleSecretRef.name	field.Required	TLS is enabled (mode is neither empty nor "disabled") but caBundleSecretRef.name is empty. Defense-in-depth alongside the CEL XValidation rule on spec.database.
Database TLS clientCertSecretRef required	spec.database.tls.clientCertSecretRef.name	field.Required	TLS is enabled (mode is neither empty nor "disabled") but clientCertSecretRef.name is empty. Defense-in-depth alongside the CEL XValidation rule on spec.database.
Fernet maxActiveKeys minimum	spec.fernet.maxActiveKeys	field.Invalid	maxActiveKeys < 3. Defense-in-depth alongside the +kubebuilder:validation:Minimum=3 marker.
Fernet schedule required	spec.fernet.rotationSchedule	field.Required	Empty after admission (bypass paths).
Fernet cron expression	spec.fernet.rotationSchedule	field.Invalid	cron.ParseStandard() fails. Error message includes the parse failure details.
CredentialKeys maxActiveKeys minimum	spec.credentialKeys.maxActiveKeys	field.Invalid	maxActiveKeys < 3. Defense-in-depth alongside the +kubebuilder:validation:Minimum=3 marker.
CredentialKeys schedule required	spec.credentialKeys.rotationSchedule	field.Required	Empty after admission (bypass paths).
CredentialKeys cron expression	spec.credentialKeys.rotationSchedule	field.Invalid	cron.ParseStandard() fails.
Duplicate plugin sections	spec.plugins[i].configSection	field.Duplicate	Two or more plugins share the same configSection value.
Policy source required	spec.policyOverrides	field.Required	policyOverrides is set but both rules and configMapRef are nil/empty.
Empty policy rule name	spec.policyOverrides.rules[<key>]	field.Required	A key in the rules map is the empty string. Enforced via the shared policy.ValidatePolicyRules.
Empty policy rule value	spec.policyOverrides.rules[<key>]	field.Required	A value in the rules map is the empty string. Enforced via the shared policy.ValidatePolicyRules.
SecretStoreRef name required / kind enum	spec.secretStoreRef.name, spec.secretStoreRef.kind	field.Required / field.NotSupported	name is empty (Required), or kind is set to a value outside ClusterSecretStore/SecretStore (NotSupported). A nil ref is valid. Defense-in-depth alongside the shared SecretStoreRefSpec MinLength/Enum markers.
Autoscaling maxReplicas minimum	spec.autoscaling.maxReplicas	field.Invalid	maxReplicas < 1. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
Autoscaling minReplicas minimum	spec.autoscaling.minReplicas	field.Invalid	minReplicas < 1 when set. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
Autoscaling min exceeds max	spec.autoscaling.minReplicas	field.Invalid	minReplicas > maxReplicas when set.
Autoscaling maxReplicas vs replicas	spec.autoscaling.maxReplicas	field.Invalid	minReplicas is unset and spec.deployment.replicas > autoscaling.maxReplicas. Would otherwise produce an HPA the API server rejects, because minReplicas defaults to spec.deployment.replicas.
Autoscaling CPU utilization range	spec.autoscaling.targetCPUUtilization	field.Invalid	Value outside 1..100 when set.
Autoscaling memory utilization range	spec.autoscaling.targetMemoryUtilization	field.Invalid	Value outside 1..100 when set.
Autoscaling no metric targets	spec.autoscaling	field.Required	Neither targetCPUUtilization nor targetMemoryUtilization is set. Defense-in-depth alongside the CEL XValidation rule.
NetworkPolicy ingress required	spec.networkPolicy.ingress	field.Required	networkPolicy is set but ingress is empty. Defense-in-depth alongside the CEL XValidation rule.
uWSGI processes minimum	spec.uwsgi.processes	field.Invalid	processes < 1 when spec.uwsgi is non-nil. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
uWSGI threads minimum	spec.uwsgi.threads	field.Invalid	threads < 1 when spec.uwsgi is non-nil. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
uWSGI harakiri minimum	spec.uwsgi.harakiri	field.Invalid	harakiri < 1 when set. Defense-in-depth alongside the +kubebuilder:validation:Minimum=1 marker.
uWSGI keep-alive timeout minimum	spec.uwsgi.httpKeepAliveTimeout	field.Invalid	httpKeepAliveTimeout < 1 when set. A zero value is rejected because uWSGI interprets it as unbounded, defeating the graceful-termination contract.
uWSGI keep-alive timeout without keep-alive	spec.uwsgi.httpKeepAliveTimeout	field.Invalid	httpKeepAliveTimeout is set while httpKeepAlive=false. The --http-keepalive-timeout flag is only emitted when keep-alive is enabled, so the combination is rejected to avoid silently dropping user intent.
TerminationGracePeriodSeconds minimum	spec.deployment.terminationGracePeriodSeconds	field.Invalid	terminationGracePeriodSeconds < 10 when set. Defense-in-depth alongside the +kubebuilder:validation:Minimum=10 marker.
PreStopSleepSeconds minimum	spec.deployment.preStopSleepSeconds	field.Invalid	preStopSleepSeconds < 0 when set. Defense-in-depth alongside the +kubebuilder:validation:Minimum=0 marker.
PreStopSleep ≥ grace period	spec.deployment.preStopSleepSeconds	field.Invalid	Resolved preStopSleepSeconds >= terminationGracePeriodSeconds (nil pointers resolve to defaults 5/30). Guarantees a non-zero drain window between the end of the preStop sleep and SIGKILL.
Harakiri ≥ drain window	spec.uwsgi.harakiri	field.Invalid	harakiri >= terminationGracePeriodSeconds − preStopSleepSeconds (nil pointers resolve to defaults). Guarantees the per-request kill fits inside the shutdown envelope.
Recreate strategy with RollingUpdate	spec.deployment.strategy.rollingUpdate	field.Invalid	strategy.type = Recreate combined with a non-nil strategy.rollingUpdate block. The Deployment controller would reject the object at apply time; the webhook catches the misconfiguration up-front.
Resource request exceeds limit	spec.deployment.resources.requests.<resource>	field.Invalid	A resource request exceeds its corresponding limit (e.g., CPU request 1000m > limit 500m). Checked per resource type when both requests and limits are set.
Trust flush schedule required	spec.trustFlush.schedule	field.Required	trustFlush is set but schedule is empty. Defense-in-depth — the +kubebuilder:default marker normally prevents this, but bypass paths (e.g., kubectl patch) may produce an empty string.
Trust flush cron expression	spec.trustFlush.schedule	field.Invalid	cron.ParseStandard() fails on trustFlush.schedule. Error message includes the parse failure details.
PriorityClass existence	spec.deployment.priorityClassName	field.NotFound / field.InternalError	The webhook performs a direct (uncached) cluster-scoped Get of the referenced scheduling.k8s.io/v1 PriorityClass when the field is non-empty, so a just-created class is never rejected off a stale cache. Missing classes produce NotFound; transient API errors produce InternalError.
TopologySpread labelSelector required	spec.deployment.topologySpreadConstraints[i].labelSelector	field.Required	Entry has no labelSelector.
TopologySpread matchLabels mismatch	spec.deployment.topologySpreadConstraints[i].labelSelector	field.Invalid	matchLabels does not exactly equal {app.kubernetes.io/name: keystone, app.kubernetes.io/instance: {CR name}}.
TopologySpread matchExpressions forbidden	spec.deployment.topologySpreadConstraints[i].labelSelector.matchExpressions	field.Invalid	matchExpressions is non-empty. Only exact matchLabels are allowed.

Error format: All validation errors are returned as a structured apierrors.StatusError with GroupKind{Group: "keystone.openstack.c5c3.io", Kind: "Keystone"}, providing clear, field-specific error messages to the operator.

---

Testing

The Keystone CRD has a three-layer test strategy:

1. Unit tests — fast, in-process tests for webhook logic.
2. Integration tests — envtest-based tests that run a real API server + etcd to validate CRD schema, CEL rules, and webhooks through the full admission pipeline.
3. E2E tests — Chainsaw tests that deploy the operator to a real cluster and verify webhook rejection in a production-like environment.

Running the Tests

Layer	Command	Prerequisites
Unit	go test ./operators/keystone/api/v1alpha1/	None
Integration	go test -tags=integration ./operators/keystone/api/v1alpha1/	KUBEBUILDER_ASSETS set to envtest binaries
E2E	chainsaw test --test-dir tests/e2e/keystone/invalid-cr/	Operator deployed to a cluster with webhooks active

envtest Integration Helper

The operators/keystone/internal/testutil package provides a Keystone-specific envtest setup helper that configures CRD installation and webhook serving for integration tests.

func SetupKeystoneEnvTest(
    t testing.TB,
    addToScheme func(*runtime.Scheme) error,
    registerWebhooks func(ctrl.Manager) error,
) (client.Client, context.Context, context.CancelFunc)

Design decisions:

• Uses a local scheme — SharedScheme() from internal/common is not modified. Only Keystone tests need Keystone types registered.
• Resolves CRD and webhook manifest paths via runtime.Caller(0) relative navigation, matching the pattern in internal/common/testutil/envtest/setup.go.
• Starts a controller-runtime manager with a webhook server bound to the envtest-allocated host, port, and certificate directory.
• Waits for the webhook server TLS endpoint to accept connections before returning.
• Tears down the environment automatically via t.Cleanup().

Parameters:

Name	Type	Description
addToScheme	func(*runtime.Scheme) error	Registers Keystone API types (breaks import cycle between testutil and v1alpha1).
registerWebhooks	func(ctrl.Manager) error	Sets up webhook handlers with the manager.

The SkipIfEnvTestUnavailable guard is re-exported from internal/common/testutil/envtest for convenience.

Integration Test Coverage

All integration tests use the //go:build integration tag and call testutil.SkipIfEnvTestUnavailable(t) as the first statement.

CRD Installation and Valid CR Acceptance

Test	Requirement	Behavior
TestIntegration_CRDInstalled	CRD discoverable	Lists CRDs via apiextensions API; verifies keystones.keystone.openstack.c5c3.io is present.
TestIntegration_ValidCRAccepted	Happy-path admission	Creates a valid Keystone CR (brownfield database mode), verifies HTTP 201 and successful Get.
TestIntegration_ValidCRWithClusterRefAccepted	ClusterRef mode	Creates a valid CR using database.clusterRef and cache.clusterRef, verifies acceptance and readback.

CEL Validation Rejection

Test	Requirement	Trigger	Expected Error
TestIntegration_CELRejectsDBBothClusterRefAndHost	Mutual exclusivity	Both database.clusterRef and database.host set	Invalid/Forbidden containing "database"
TestIntegration_CELRejectsCacheBothClusterRefAndServers	Mutual exclusivity	Both cache.clusterRef and cache.servers set	Invalid/Forbidden containing "cache"
TestIntegration_CELRejectsReplicasBelowMinimum	Minimum constraint	replicas = -1 (note: 0 is converted to 3 by the defaulting webhook, so -1 is used)	Invalid/Forbidden
TestIntegration_CELRejectsMaxActiveKeysBelowMinimum	Minimum constraint	fernet.maxActiveKeys = 1 (below minimum of 3; 0 is defaulted to 3 by webhook)	Invalid/Forbidden
TestIntegration_CELRejectsPolicyOverridesEmpty	Policy source required	policyOverrides set with neither rules nor configMapRef	Invalid/Forbidden containing "policyOverrides"
TestIntegration_CELRejectsPolicyRuleEmptyValue	Non-empty rule values	policyOverrides.rules with a rule whose value is the empty string	Invalid/Forbidden containing "policyOverrides"

Admission pipeline note: In Kubernetes, the admission order is: mutating webhooks then schema validation (CEL) then validating webhooks. The defaulting webhook converts replicas: 0 to 3 and maxActiveKeys: 0 to 3 before CEL validation runs, so these tests use values that bypass defaulting (negative or non-zero-but-below-minimum) to exercise the CRD schema constraints.

Webhook Defaulting

Test	Requirement	Behavior
TestIntegration_WebhookDefaultsSetsZeroValues	Defaults applied	Creates a CR with zero-valued defaultable fields; verifies replicas=3, cache.backend="dogpile.cache.pymemcache", bootstrap.adminUser="admin", bootstrap.region="RegionOne", fernet.maxActiveKeys=3 after admission.
TestIntegration_WebhookDefaultsPreservesExplicit	Explicit values preserved	Creates a CR with replicas=5 and region="EU-West"; verifies these values are not overwritten by the defaulting webhook.
TestIntegration_ResourcesDefaultedWhenNil	Resources defaulted	Creates a CR with spec.deployment.resources unset (nil); verifies the defaulting webhook injects {requests: {memory: 256Mi, cpu: 100m}, limits: {memory: 512Mi, cpu: 500m}}.
TestIntegration_ResourcesPreservedWhenExplicit	Explicit resources preserved	Creates a CR with explicit spec.deployment.resources (1Gi/2Gi memory, 200m/1 CPU); verifies the defaulting webhook does not overwrite them.
TestIntegration_UWSGIDefaultsAppliedWhenEmpty	uWSGI defaults applied	Creates a CR with spec.uwsgi: {} (all zero values); verifies processes=2, threads=1, httpKeepAlive=true after admission.
TestIntegration_UWSGIExplicitValuesPreserved	Explicit uWSGI preserved	Creates a CR with spec.uwsgi.processes=4, threads=4; verifies these values are not overwritten by the defaulting webhook.
TestIntegration_UWSGIPartialDefaulting	Partial uWSGI defaults	Creates a CR with only spec.uwsgi.processes=4; verifies threads=1 is defaulted while processes=4 is preserved.
TestIntegration_UWSGINilPreserved	uWSGI nil preserved	Creates a CR without spec.uwsgi; verifies the field remains nil after admission — webhook does not inject a default struct.

Webhook Validation Rejection

Test	Requirement	Trigger	Expected Error
TestIntegration_ResourcesRequestExceedsLimitRejected	Request must not exceed limit	spec.deployment.resources with CPU request 1000m > limit 500m	Invalid/Forbidden containing "resources".
TestIntegration_UWSGIProcessesBelowMinimumRejected	Processes minimum	spec.uwsgi.processes below minimum (bypassing defaulting)	Invalid/Forbidden containing "uwsgi".
TestIntegration_UWSGIThreadsBelowMinimumRejected	Threads minimum	spec.uwsgi.threads below minimum (bypassing defaulting)	Invalid/Forbidden containing "uwsgi".

Chainsaw E2E Tests

E2E tests live in tests/e2e/keystone/ and use the Chainsaw framework (chainsaw.kyverno.io/v1alpha2). The invalid-cr suite below verifies webhook rejection in a real cluster with the operator deployed. For the full reconciler E2E test suite inventory (basic-deployment, scale, fernet-rotation, credential-rotation, network-policy, topology-spread, priority-class, release-upgrade, schema-drift-detection, events, healthcheck, graceful-shutdown, policy-validation, config-pruning, database-tls, …), see Keystone E2E Test Suites.

invalid-cr Suite

The full webhook + CEL rejection matrix extends the original two-step suite so that every implemented XValidation rule and every webhook.validate() branch in operators/keystone/api/v1alpha1/ is pinned by a Chainsaw step.

Step	Manifest	Requirement	Expected Error
invalid-cron-expression-rejected	00-invalid-cron.yaml	Invalid cron	Error containing "rotationSchedule" and "invalid cron expression"
duplicate-plugin-config-section-rejected	01-duplicate-plugins.yaml	Duplicate configSection	Error containing "configSection" and "Duplicate value"
database-both-modes-rejected	02-database-both-modes.yaml	DatabaseSpec mutual exclusivity	Error containing "spec.database" and "exactly one of clusterRef or host must be set"
cache-both-modes-rejected	03-cache-both-modes.yaml	CacheSpec mutual exclusivity	Error containing "spec.cache" and "exactly one of clusterRef or servers must be set"
autoscaling-no-target-rejected	04-autoscaling-no-target.yaml	AutoscalingSpec target required	Error containing "spec.autoscaling" and "at least one of targetCPUUtilization or targetMemoryUtilization"
policy-overrides-no-source-rejected	05-policy-overrides-no-source.yaml	PolicyOverrides source required	Error containing "spec.policyOverrides" and "at least one of rules or configMapRef must be set"
policy-overrides-empty-rule-key-rejected	06-policy-overrides-empty-rule-key.yaml	Non-empty rule names	Error containing "spec.policyOverrides" and "policy rule name must not be empty"
networkpolicy-empty-ingress-rejected	07-networkpolicy-empty-ingress.yaml	NetworkPolicy ingress required	Error containing "spec.networkPolicy" and "at least one ingress source"
replicas-negative-rejected	09-replicas-negative.yaml	Replicas Minimum=1 (subsumes the dropped 08-replicas-zero.yaml case — see layer-ordering aside)	Error containing "replicas"
hpa-min-greater-than-max-rejected	10-hpa-min-greater-than-max.yaml	minReplicas ≤ maxReplicas	Error containing "spec.autoscaling.minReplicas" and "must not exceed maxReplicas"
fernet-maxactivekeys-below-minimum-rejected	11-fernet-maxactivekeys-below-minimum.yaml	Fernet maxActiveKeys Minimum=3	Error containing "maxActiveKeys"
credentialkeys-maxactivekeys-below-minimum-rejected	12-credentialkeys-maxactivekeys-below-minimum.yaml	CredentialKeys maxActiveKeys Minimum=3	Error containing "maxActiveKeys"
policy-overrides-empty-rule-value-rejected	13-policy-overrides-empty-rule-value.yaml	Non-empty rule values	Error containing "spec.policyOverrides" and "policy rule value must not be empty"
immutable-base-accepted	13-immutable-base.yaml	Valid base CR for the update-rejection cases (applied first)	None (must succeed)
immutable-database-name-rejected	14-immutable-database-name.yaml	spec.database.database immutable on UPDATE	Error containing "database" and "immutable"
immutable-database-mode-rejected	15-immutable-database-mode.yaml	spec.database mode (clusterRef ↔ host) immutable on UPDATE	Error containing "database mode" and "immutable"
immutable-adminuser-rejected	16-immutable-adminuser.yaml	spec.bootstrap.adminUser immutable on UPDATE	Error containing "adminUser" and "immutable"
immutable-region-rejected	17-immutable-region.yaml	spec.bootstrap.region immutable on UPDATE	Error containing "region" and "immutable"
image-empty-tag-rejected	13-image-empty-tag.yaml	ImageSpec.Tag Pattern	Error containing "image.tag"
image-tag-and-digest-rejected	21-image-tag-and-digest.yaml	ImageSpec tag/digest XOR	Error containing "digest"
database-name-invalid-char-rejected	14-database-name-invalid-char.yaml	DatabaseSpec.Database Pattern	Error containing "database.database"
database-secretref-empty-name-rejected	15-database-secretref-empty-name.yaml	SecretRefSpec.Name MinLength=1	Error containing "secretRef.name"
middleware-bad-position-rejected	16-middleware-bad-position.yaml	PipelinePosition Enum=before;after	Error containing "position" and "sideways"
uwsgi-keepalive-timeout-conflict-rejected	17-uwsgi-keepalive-timeout-conflict.yaml	httpKeepAliveTimeout requires httpKeepAlive (CEL)	Error containing "httpKeepAliveTimeout"
prestop-not-less-than-grace-rejected	18-prestop-not-less-than-grace.yaml	drain window preStop < TGPS (CEL)	Error containing "preStopSleepSeconds" and "terminationGracePeriodSeconds"
publicendpoint-not-url-rejected	19-publicendpoint-not-url.yaml	BootstrapSpec.PublicEndpoint Pattern	Error containing "publicEndpoint"
perloggerlevels-invalid-value-rejected	20-perloggerlevels-invalid-value.yaml	perLoggerLevels value enum (CEL)	Error containing "perLoggerLevels"

Steps 14-17 reuse the immutable-fields name from 13-immutable-base.yaml, so each is applied as an UPDATE of the base CR and is rejected by the corresponding CEL transition rule (self == oldSelf, evaluated only on UPDATE). The base CR is brownfield (host mode), so the operator creates no MariaDB CRs and pushes no OpenBao secrets; its finalizer cleanup is a no-op and the ephemeral namespace tears down cleanly.

Each step uses apply with expect to assert that the $error variable is non-null and contains the expected field-level error message. Kubernetes admission evaluates validation in a fixed pipeline — mutating webhook (defaulting) → CRD structural schema (incl. CEL XValidation rules) → validating webhook — and the first layer that rejects an object is the one whose message Chainsaw sees. The mutating step is listed first because it can silently rewrite a value out from under a downstream rule: keystone_webhook.go:80-82 coerces spec.deployment.replicas == 0 to 3 BEFORE the +kubebuilder:validation:Minimum=1 marker is evaluated, so a manifest using spec.deployment.replicas: 0 would be silently accepted. This is the precise reason the 08-replicas-zero.yaml case was dropped from the suite: the 09-replicas-negative.yaml fixture (spec.deployment.replicas: -1) uses a value the defaulter does not touch (the defaulter only fires on == 0) and exercises the same Minimum=1 and webhook-defense-in-depth path. The same trap applies to maxActiveKeys: 0, which is why the maxActiveKeys fixtures use 2 rather than 0.

For most rules the producing layer is unambiguous (CEL emits the exact "exactly one of …", "at least one of …", "must not exceed maxReplicas" wording), so the assertions match the full webhook-equivalent message. The 06-policy-overrides-empty-rule-key.yaml and 07-networkpolicy-empty-ingress.yaml fixtures are the dual-layer exceptions where the fieldPath emitted by CEL is the parent path (spec.policyOverrides / spec.networkPolicy) — the path where the XValidation rule is declared — and NOT the deeper path the validating webhook would emit (…rules / …ingress). Because CEL fails first and short-circuits the admission pipeline, the validating webhook's deeper-path message never reaches Chainsaw, so the assertions match only the parent path. The 11-fernet-maxactivekeys-below-minimum.yaml and 12-credentialkeys-maxactivekeys-below-minimum.yaml fixtures are the field-substring exceptions: they trip the CRD structural schema's Minimum=N first, whose generated wording ("must be greater than or equal to N") differs from the webhook's defense-in-depth wording ("maxActiveKeys must be at least 3"). Both layers carry the field name, so the loose-substring assertion (maxActiveKeys) keeps the tests stable regardless of which layer fires first and across upstream Kubernetes admission-pipeline changes.

The 24 generated fixtures (02-… through 20-…, with the 08-replicas-zero.yaml gap explained above) share an otherwise-identical minimal valid Keystone scaffold. The create-rejection fixtures (02-… through 12-…, 13-policy-overrides-empty-rule-value.yaml, and the validation-marker set 13-image-empty-tag.yaml through 20-perloggerlevels-invalid-value.yaml) differ only by the field under test; the update-rejection fixtures (13-immutable-base.yaml through 17-immutable-region.yaml) share the immutable-fields name so 13-immutable-base.yaml is the base and 14-17 are applied as UPDATEs. The create-rejection, update-rejection, and validation-marker sets deliberately reuse the 13-17 numeric prefixes with distinct filenames, so those prefixes recur across the suite. To prevent that scaffold from drifting across files, the fixtures are generated from a single canonical source in tests/e2e/keystone/invalid-cr/_generate.py. After editing the scaffold or any per-fixture override, regenerate via python3 tests/e2e/keystone/invalid-cr/_generate.py. The verify-invalid-cr-fixtures CI job (and the matching make verify-invalid-cr-fixtures Makefile target) runs _generate.py --check in drift mode and the test_generate.py unit suite (len(FIXTURES) == 25 plus a cross-reference assertion that every Fixture.filename appears as a file: step in chainsaw-test.yaml), so a hand-edit to any generated fixture — or a rename/removal that desynchs FIXTURES from chainsaw-test.yaml — fails the build before the cluster-bound e2e-operator job runs. The 00-invalid-cron.yaml and 01-duplicate-plugins.yaml fixtures predate the generator and are intentionally NOT regenerated.

The following follow-up gaps are intentionally not covered by this suite — they require new validation rules that do not exist yet, and each one is tracked as its own feature ticket:

• topologySpreadConstraints[*].maxSkew: 0 (no CRD-level minimum on the upstream type, no defense-in-depth in the Keystone webhook).
• Mutation of spec.cache mode (clusterRef ↔ servers) on UPDATE — no transition rule yet. The spec.database name and mode and the spec.bootstrap.adminUser/region fields are now immutable via CEL transition rules (see CEL Validation Rules).

uwsgi Suite

The uwsgi suite (tests/e2e/keystone/uwsgi/) validates that spec.uwsgi values propagate to the Deployment container command in a real cluster with the operator deployed and reconciling.

Step	Description	Assertion
Step 1	Apply Keystone CR without explicit spec.uwsgi	CR created
Step 2 (step-2-assert-default-uwsgi-args)	Assert Deployment command contains default uWSGI args	Container command includes --processes 2 --threads 1 --http-keepalive
Step 3	Patch CR with spec.uwsgi: {processes: 3, threads: 3, httpKeepAlive: false}	Patch applied
Step 4 (step-4-assert-custom-uwsgi-args)	Assert Deployment command updated with custom values	Container command includes --processes 3 --threads 3; --http-keepalive is absent

---

CRD Generation

The CRD manifest and DeepCopy methods are generated by controller-gen:

Target	Command	Output
DeepCopy	make generate	operators/keystone/api/v1alpha1/zz_generated.deepcopy.go
CRD YAML	make manifests	operators/keystone/config/crd/bases/keystone.openstack.c5c3.io_keystones.yaml

Both targets are parameterized by operator directory in the Makefile. Generated zz_generated.*.go files are excluded from linting via .golangci.yml.

Generated DeepCopy Types

zz_generated.deepcopy.go provides DeepCopyObject() and DeepCopyInto() for:

• Keystone
• KeystoneList
• KeystoneSpec
• KeystoneStatus
• AutoscalingSpec
• NetworkPolicySpec
• NetworkPolicyIngressSource
• UWSGISpec
• TrustFlushSpec
• FernetSpec
• CredentialKeysSpec
• FederationSpec
• BootstrapSpec
• PasswordRotationSpec
• LoggingSpec

---

File Layout

operators/keystone/
├── api/v1alpha1/
│   ├── groupversion_info.go          GroupVersion, SchemeBuilder, AddToScheme
│   ├── keystone_types.go             CRD types + init() scheme registration
│   ├── keystone_webhook.go           Defaulting + validating webhooks
│   ├── keystone_types_test.go        Type and scheme registration tests
│   ├── keystone_webhook_test.go      Webhook unit tests (table-driven)
│   ├── integration_test.go           envtest integration tests
│   └── zz_generated.deepcopy.go     Generated DeepCopy methods
├── config/crd/bases/
│   └── keystone.openstack.c5c3.io_keystones.yaml  Generated CRD manifest
├── config/webhook/
│   ├── manifests.yaml                Generated webhook configurations
│   └── ...
├── internal/testutil/
│   └── envtest_setup.go              Keystone-specific envtest helper
└── main.go                           Scheme registration + bootstrap + webhook wiring

tests/e2e/keystone/
├── basic-deployment/                 Happy-path reconciliation E2E
├── missing-secret/                   Secret dependency recovery E2E
├── fernet-rotation/                  Fernet key rotation E2E
├── scale/                            Replica scaling E2E
├── deletion-cleanup/                 Garbage collection E2E
├── policy-overrides/                 oslo.policy integration E2E
├── middleware-config/                Middleware pipeline E2E
├── brownfield-database/              External database mode E2E
├── database-tls/                     Database TLS/mTLS E2E
│   ├── chainsaw-test.yaml            Chainsaw E2E test definition
│   └── 00-keystone-cr.yaml           Keystone CR with spec.database.tls (verify-full)
├── image-upgrade/                    Rolling image upgrade E2E
├── uwsgi/                            uWSGI field propagation E2E
│   ├── chainsaw-test.yaml            Chainsaw E2E test definition
│   ├── 00-keystone-cr.yaml           Keystone CR without explicit uWSGI
│   └── 01-patch-custom-uwsgi.yaml    Patch with custom uWSGI values
└── invalid-cr/
    ├── chainsaw-test.yaml                                  Chainsaw E2E test definition
    ├── 00-invalid-cron.yaml                                Invalid cron expression CR manifest
    ├── 01-duplicate-plugins.yaml                           Duplicate plugin configSection CR manifest
    ├── 02-database-both-modes.yaml                         Database clusterRef + host both set
    ├── 03-cache-both-modes.yaml                            Cache clusterRef + servers both set
    ├── 04-autoscaling-no-target.yaml                       Autoscaling without utilization target
    ├── 05-policy-overrides-no-source.yaml                  PolicyOverrides without rules or configMapRef
    ├── 06-policy-overrides-empty-rule-key.yaml             PolicyOverrides rule with empty key
    ├── 07-networkpolicy-empty-ingress.yaml                 NetworkPolicy with empty ingress array
    ├── 09-replicas-negative.yaml                           spec.deployment.replicas: -1 (subsumes the dropped 08-replicas-zero case)
    ├── 10-hpa-min-greater-than-max.yaml                    HPA minReplicas > maxReplicas
    ├── 11-fernet-maxactivekeys-below-minimum.yaml          Fernet maxActiveKeys < 3
    └── 12-credentialkeys-maxactivekeys-below-minimum.yaml  CredentialKeys maxActiveKeys < 3

This layout is the canonical pattern for all CobaltCore operators. New operators should replicate this directory structure.