Shared Library

The shared library at internal/common/ provides reusable building blocks for all CobaltCore operators. It encapsulates patterns for database interaction, config rendering, secret handling, deployment management, and plugin configuration — ensuring consistent behavior and reducing duplication across operators.

Design Rationale

Without a shared library, each operator would independently implement the same patterns: waiting for MariaDB readiness, rendering INI config files, checking ESO secret availability, managing Kubernetes Jobs. This leads to divergence, duplicated bugs, and inconsistent user experience. The shared library centralizes these concerns.

Comparison with openstack-k8s-operators/lib-common

Red Hat's openstack-k8s-operators project uses a separate lib-common repository. CobaltCore takes a different approach:

Aspect	Red Hat (lib-common)	CobaltCore (internal/common)
Repository	Separate repo (openstack-k8s-operators/lib-common)	Monorepo subdirectory
Versioning	Tagged releases, operators pin specific versions	Go Workspace — all operators always use HEAD
Dependency Management	go.mod require with exact version	go.work use directive (local resolution)
Breaking Changes	Requires coordinated version bumps across repos	Single commit updates library + all consumers
CI	Separate CI per repo, cross-repo integration testing	Single CI pipeline tests everything together
Discovery	Separate docs/godoc	Colocated in the same codebase

The monorepo approach trades release independence for development velocity — a breaking change in the shared library is immediately visible in all operator builds within the same CI run.

Package Structure

┌─────────────────────────────────────────────────────────────────────────────┐
│                       internal/common/ PACKAGES                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  bootstrap/     Controller-runtime manager initialization                   │
│  conditions/    Condition management for Status.Conditions                  │
│  config/        INI config rendering pipeline                               │
│  database/      MariaDB CR interaction and db_sync jobs                     │
│  deployment/    Deployment, Service, PDB, and HPA management               │
│  job/           Kubernetes Job and CronJob management                       │
│  secrets/       ESO secret readiness and PushSecret helpers                 │
│  plugins/       Plugin and middleware config rendering                      │
│  policy/        Policy file rendering and validation                       │
│  tls/           cert-manager Certificate CR handling                        │
│  types/         Shared Go struct definitions                                │
│  testutil/      Test utilities (assertions, builders, envtest, simulators)  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

bootstrap/

Provides a shared entrypoint for all operator main.go files, encapsulating controller-runtime manager initialization, scheme registration, leader election, and namespace scoping.

Function	Description
Run(cfg ManagerConfig) error	Initialize and start a controller-runtime manager with the given configuration. Parses flags (--metrics-bind-address, --health-probe-bind-address, --leader-elect, --namespace, --enable-webhooks, sync period), sets up zap logging, health/readiness probes, leader election, and graceful shutdown.

ManagerConfig carries:

type ManagerConfig struct {
    Scheme           *runtime.Scheme
    LeaderElectionID string
    // Namespace restricts the manager cache to a single namespace
    // when set (namespace-scoped mode, CC-0043); cluster-scoped otherwise.
    Namespace string
    // SetupFunc registers controllers and (when webhooks is true) webhooks.
    SetupFunc func(mgr ctrl.Manager, webhooks bool) error
}

Each operator's main.go calls bootstrap.Run() with a ManagerConfig. The SetupFunc receives the manager and a webhooks bool (driven by the --enable-webhooks flag, which defaults to true) so webhook registration can be skipped in environments where it is not wanted. Run validates the config (a non-nil Scheme and a non-empty LeaderElectionID are required) and applies flag defaults --metrics-bind-address :8080, --health-probe-bind-address :8081, --leader-elect false, --sync-period 10m.

conditions/

Manages metav1.Condition entries on operator status objects.

Function	Description
SetCondition(conditions *[]metav1.Condition, condition metav1.Condition)	Set or update a condition by type
IsReady(conditions []metav1.Condition) bool	Check if the Ready condition is True
GetCondition(conditions []metav1.Condition, conditionType string) *metav1.Condition	Retrieve a specific condition
AllTrue(conditions []metav1.Condition, types ...string) bool	Check if all specified conditions are True

database/

Encapsulates interaction with the MariaDB Operator's CRDs.

Function	Description
EnsureDatabase(ctx, client, scheme, owner, db) (bool, error)	Create or verify a MariaDB Database CR. Returns true when ready.
EnsureDatabaseUser(ctx, client, scheme, owner, user, grant) (bool, error)	Create or verify a MariaDB User CR with Grant CR for privileges.
RunDBSyncJob(ctx, client, scheme, owner, job) (bool, error)	Run a db_sync Kubernetes Job using the service image. Returns true when completed.
IsDatabaseReady(db) bool	Check if a MariaDB Database CR reports ready status.
IsUserReady(user) bool	Check if a MariaDB User CR reports ready status.
IsGrantReady(grant) bool	Check if a MariaDB Grant CR reports ready status.

messaging/ planned

Status: planned — not yet implemented. No messaging/ package exists in forge today (Keystone, the only built operator, needs no message bus). The shared types.MessagingSpec is already defined, and this package will be added with the first messaging-dependent operator (Nova/Neutron/Cinder). The design below is the intended shape.

Encapsulates interaction with the RabbitMQ Messaging Topology Operator's CRDs. Analogous to the database/ package — each service operator will use this package to create per-service vhosts, users, and permissions as Kubernetes CRs. In brownfield mode (explicit hosts), no Topology CRs are created.

Function	Description
EnsureVhost(ctx, client, owner, spec) (bool, error)	Create or verify a Topology Vhost CR. Returns true when ready.
EnsureUser(ctx, client, owner, spec) (bool, error)	Create or verify a Topology User CR with credentials from the referenced Secret.
EnsurePermission(ctx, client, owner, spec) (bool, error)	Create or verify a Topology Permission CR granting configure/write/read on the vhost.
IsMessagingReady(ctx, client, namespace, vhostName, userName string) (bool, error)	Check if the Vhost, User, and Permission CRs all report ready status conditions.
ResolveEndpoint(ctx, client, namespace string, msgSpec MessagingSpec) (string, int32, error)	Resolve the AMQP endpoint from either the RabbitmqCluster CR status (managed mode) or explicit hosts (brownfield mode).

Usage in a service operator's reconcileMessaging():

func (r *NovaReconciler) reconcileMessaging(ctx context.Context,
    nova *novav1alpha1.Nova) (ctrl.Result, error) {

    msgSpec := nova.Spec.Messaging

    if msgSpec.ClusterRef != nil {
        // Managed mode: create Topology CRs (Vhost, User, Permission)
        vhostReady, err := messaging.EnsureVhost(ctx, r.Client, nova, messaging.VhostSpec{
            Name:       "nova",
            ClusterRef: msgSpec.ClusterRef,
        })
        if err != nil || !vhostReady {
            conditions.SetCondition(&nova.Status.Conditions, metav1.Condition{
                Type:   "MessagingReady",
                Status: metav1.ConditionFalse,
                Reason: "WaitingForVhost",
            })
            return ctrl.Result{RequeueAfter: 15 * time.Second}, err
        }

        userReady, err := messaging.EnsureUser(ctx, r.Client, nova, messaging.UserSpec{
            ClusterRef: msgSpec.ClusterRef,
            SecretRef:  msgSpec.SecretRef,
        })
        if err != nil || !userReady {
            conditions.SetCondition(&nova.Status.Conditions, metav1.Condition{
                Type:   "MessagingReady",
                Status: metav1.ConditionFalse,
                Reason: "WaitingForUser",
            })
            return ctrl.Result{RequeueAfter: 15 * time.Second}, err
        }

        permReady, err := messaging.EnsurePermission(ctx, r.Client, nova, messaging.PermissionSpec{
            Vhost:      "nova",
            UserRef:    nova.Name + "-rabbitmq-user",
            ClusterRef: msgSpec.ClusterRef,
        })
        if err != nil || !permReady {
            conditions.SetCondition(&nova.Status.Conditions, metav1.Condition{
                Type:   "MessagingReady",
                Status: metav1.ConditionFalse,
                Reason: "WaitingForPermission",
            })
            return ctrl.Result{RequeueAfter: 15 * time.Second}, err
        }
    }
    // Brownfield mode: skip Topology CR creation, use hosts directly

    conditions.SetCondition(&nova.Status.Conditions, metav1.Condition{
        Type:   "MessagingReady",
        Status: metav1.ConditionTrue,
        Reason: "MessagingAvailable",
    })
    return ctrl.Result{}, nil
}

config/

Implements the config generation pipeline documented in Config Generation. This package renders INI configuration files from Go structs — no template language is used.

Function	Description
RenderINI(sections map[string]map[string]string) string	Render a map of sections/keys into INI format
MergeDefaults(userConfig, defaults map[string]map[string]string) map[string]map[string]string	Merge user-provided config with operator defaults (user values take precedence)
CreateImmutableConfigMap(ctx, client, scheme, owner, baseName, namespace string, data map[string]string) (string, error)	Create an immutable ConfigMap whose name carries a content-hash suffix. Returns the generated ConfigMap name.
PruneImmutableConfigMaps(ctx, client, owner, baseName, namespace, currentName string, retain int) error	Garbage-collect superseded immutable ConfigMaps for a base name, keeping currentName plus the most recent retain.
InjectSecrets(config map[string]map[string]string, secrets map[string]string) map[string]map[string]string	Substitute placeholders in config values with resolved secret values; unresolved placeholders are left as-is and the input is never mutated
InjectOsloPolicyConfig(config map[string]map[string]string, policyFilePath string) map[string]map[string]string	Return a copy of the INI config with [oslo_policy] policy_file = <path> set, when policy overrides are present

The config package directly implements the pipeline from Config Generation: CRD spec → resolve secrets → apply defaults → render INI → immutable ConfigMap. Override mechanisms described in Customization (configOverrides, conf.d pattern) are supported via MergeDefaults with user-provided overrides taking precedence. When policyOverrides are configured, InjectOsloPolicyConfig adds the [oslo_policy] section pointing to the rendered policy.yaml file (see policy/ package).

deployment/

Creates and manages Kubernetes Deployments, Services, PodDisruptionBudgets, and HorizontalPodAutoscalers.

Function	Description
EnsureDeployment(ctx, client, scheme, owner, deploy) (bool, error)	Create or update a Deployment. Returns true when available.
EnsureService(ctx, client, scheme, owner, svc) error	Create or update a ClusterIP Service.
EnsurePDB(ctx, client, scheme, owner, pdb) error	Create or update a PodDisruptionBudget.
EnsureHPA(ctx, client, scheme, owner, hpa) error	Create or update a HorizontalPodAutoscaler.
DeleteHPA(ctx, client, namespace, name) error	Delete a HorizontalPodAutoscaler (used when spec.autoscaling is removed).
IsDeploymentReady(deploy *appsv1.Deployment) bool	Check if all replicas are available.

job/

Manages one-shot Jobs and recurring CronJobs.

Function	Description
RunJob(ctx, client, scheme, owner, job) (bool, error)	Create a Job and wait for completion. Returns true when succeeded. Detects PodSpec changes via hash annotation and recreates the Job when the spec changes.
EnsureCronJob(ctx, client, scheme, owner, cronJob) error	Create or update a CronJob.
IsJobComplete(job *batchv1.Job) bool	Check if a Job has completed successfully.
IsJobFailed(job *batchv1.Job) bool	Check if a Job has failed.
PodSpecHash(template *corev1.PodTemplateSpec) string	Compute a SHA-256 hash of a pod template (spec plus metadata/annotations) for change detection, so rotated-credential digests participate in re-run decisions.

secrets/

Provides helpers for ESO-based secret workflows. Operators never interact with OpenBao directly — they work exclusively with Kubernetes Secrets that ESO creates from OpenBao (see Secret Management).

Function	Description
WaitForExternalSecret(ctx, client, key client.ObjectKey) (bool, error)	Check whether the ExternalSecret reports a Ready condition (status True), indicating ESO has synced it. Returns true when ready. (Target-Secret existence and key checks live in IsSecretReady.)
IsSecretReady(ctx, client, key client.ObjectKey, expectedKeys ...string) (bool, error)	Verify that a K8s Secret exists and contains the expected keys.
IsClusterSecretStoreReady(ctx, client, name string) (bool, error)	Verify that the named ESO ClusterSecretStore reports a Ready condition before reading from it.
EnsurePushSecret(ctx, client, scheme *runtime.Scheme, owner client.Object, ps *esov1alpha1.PushSecret) error	Create or update a PushSecret CR to write operator-generated secrets back to OpenBao.
GetSecretValue(ctx, client, key client.ObjectKey, dataKey string) (string, error)	Read a specific key from a K8s Secret.
IsMissingSecretOrKey(err error) bool	Classify an error as "Secret or key not yet present" so callers can requeue rather than fail.

PushSecret pattern: Some secrets are generated by operators at runtime (e.g., Fernet keys). These are written to a Kubernetes Secret and then pushed to OpenBao via a PushSecret CR for backup and cross-cluster distribution. See Credential Lifecycle for the full ESO/PushSecret flow.

plugins/

Provides a generic plugin and middleware configuration framework usable by all OpenStack service operators. All OpenStack services use PasteDeploy for WSGI pipeline configuration, making this framework universally applicable.

Function	Description
RenderPastePipeline(spec PipelineSpec) (map[string]map[string]string, error)	Build the api-paste.ini section map from a declarative pipeline specification. Operators define a base pipeline and add middleware filters (e.g., audit, CORS, rate limiting) from the CRD.
RenderPastePipelineINI(spec PipelineSpec) (string, error)	Convenience wrapper that renders the pipeline section map directly to INI text.
RenderPluginConfig(plugins []types.PluginSpec) (map[string]map[string]string, error)	Generate INI config sections for service plugins (e.g., [keycloak] for keystone-keycloak-backend, [filter:audit] for openstack-audit-middleware).

PipelineSpec is defined in the plugins package; PluginSpec, MiddlewareSpec, and the PipelinePosition enum (PipelinePositionBefore / PipelinePositionAfter) live in the shared types package:

// PluginSpec defines a service plugin/driver configuration (package types).
type PluginSpec struct {
    // Name of the plugin (e.g., "keystone-keycloak-backend")
    Name string `json:"name"`
    // ConfigSection is the INI section name (e.g., "keycloak")
    ConfigSection string `json:"configSection"`
    // Config contains key-value pairs for the plugin's INI section
    Config map[string]string `json:"config,omitempty"`
}

// MiddlewareSpec defines a WSGI middleware filter for api-paste.ini.
type MiddlewareSpec struct {
    // Name of the filter (e.g., "audit")
    Name string `json:"name"`
    // FilterFactory is the Python entry point (e.g., "audit_middleware:filter_factory")
    FilterFactory string `json:"filterFactory"`
    // Position defines where in the pipeline this filter is inserted
    Position PipelinePosition `json:"position"`
    // Config contains key-value pairs for the filter section
    Config map[string]string `json:"config,omitempty"`
}

policy/

Provides oslo.policy file rendering, merging, and validation for OpenStack services. All OpenStack services support a policy.yaml file that overrides the default API authorization rules defined in code (policy-in-code).

Function	Description
RenderPolicyYAML(rules map[string]string) (string, error)	Render a rule map as YAML suitable for oslo.policy consumption
MergePolicies(base, override types.PolicySpec) types.PolicySpec	Merge two PolicySpecs — override rules take precedence over base rules
LoadPolicyFromConfigMap(ctx, client, key client.ObjectKey) (map[string]string, error)	Read and parse the policy.yaml key from a user-provided ConfigMap
ValidatePolicyRules(rules map[string]string, fldPath *field.Path) field.ErrorList	Validate rule syntax (valid YAML, non-empty keys, non-empty values)

tls/

Integrates with cert-manager for TLS certificate provisioning.

Function	Description
EnsureCertificate(ctx, client, scheme *runtime.Scheme, owner client.Object, cert *certmanagerv1.Certificate) (bool, error)	Create or update a cert-manager Certificate CR. Returns true when the certificate is ready.
IsCertificateReady(cert *certmanagerv1.Certificate) bool	Return true if the given cert-manager Certificate has a Ready condition with status True. (Fetch-by-key readiness lives in EnsureCertificate.)
GetTLSSecret(ctx, client, key client.ObjectKey) (certPEM []byte, keyPEM []byte, err error)	Retrieve the tls.crt / tls.key material from the Secret created by cert-manager.

types/

Shared Go struct definitions used across operator CRDs:

// ImageSpec defines a container image reference.
type ImageSpec struct {
    Repository string `json:"repository"`
    Tag        string `json:"tag"`
}

// DatabaseSpec supports managed (ClusterRef) and brownfield (explicit) modes.
// Exactly one of ClusterRef or Host must be set.
type DatabaseSpec struct {
    // ClusterRef references a MariaDB CR in the cluster (managed mode).
    // +optional
    ClusterRef *corev1.LocalObjectReference `json:"clusterRef,omitempty"`
    // Host is the database hostname (brownfield mode).
    // +optional
    Host string `json:"host,omitempty"`
    // Port is the database port (brownfield mode, default 3306).
    // +optional
    Port int32 `json:"port,omitempty"`
    // Database is the database name within the cluster.
    Database string `json:"database"`
    // SecretRef references the K8s Secret with credentials.
    SecretRef SecretRefSpec `json:"secretRef"`
    // TLS optionally enables TLS/mTLS for the database connection (CC-0106).
    // A nil TLS means plaintext TCP — opt-in and non-mutating.
    // +optional
    TLS *DatabaseTLSSpec `json:"tls,omitempty"`
}

// DatabaseTLSSpec configures opt-in TLS (and mutual TLS) for a database
// connection (CC-0106). Referenced as an optional pointer from DatabaseSpec.
type DatabaseTLSSpec struct {
    // Enabled turns on TLS. When true the operator provisions the client
    // certificate, appends the ssl_* DSN parameters, and mounts the
    // certificate material into workloads that open a connection.
    Enabled bool `json:"enabled"`
    // Mode selects verification strength: prefer/require (encrypt only),
    // verify-ca (verify server chain), verify-full (verify chain + hostname).
    // +kubebuilder:validation:Enum=prefer;require;verify-ca;verify-full
    // +optional
    Mode string `json:"mode,omitempty"`
    // CABundleSecretRef references the Secret holding the server CA bundle.
    CABundleSecretRef SecretRefSpec `json:"caBundleSecretRef"`
    // ClientCertSecretRef references the Secret holding the client keypair
    // presented to the database for mutual TLS.
    ClientCertSecretRef SecretRefSpec `json:"clientCertSecretRef"`
}

// MessagingSpec supports managed (ClusterRef) and brownfield (explicit) modes.
// Exactly one of ClusterRef or Hosts must be set.
type MessagingSpec struct {
    // ClusterRef references a RabbitMQ CR in the cluster (managed mode).
    // +optional
    ClusterRef *corev1.LocalObjectReference `json:"clusterRef,omitempty"`
    // Hosts is the list of RabbitMQ endpoints (brownfield mode).
    // +optional
    Hosts []string `json:"hosts,omitempty"`
    // SecretRef references the K8s Secret with credentials.
    SecretRef SecretRefSpec `json:"secretRef"`
}

// CacheSpec supports managed (ClusterRef) and brownfield (explicit) modes.
// Exactly one of ClusterRef or Servers must be set.
type CacheSpec struct {
    // ClusterRef references a Memcached CR in the cluster (managed mode).
    // +optional
    ClusterRef *corev1.LocalObjectReference `json:"clusterRef,omitempty"`
    // Backend is the cache backend (e.g. dogpile.cache.pymemcache).
    Backend string `json:"backend"`
    // Servers is the list of cache server endpoints (brownfield mode).
    // +optional
    Servers []string `json:"servers,omitempty"`
    // Replicas is the number of Memcached pod replicas in the referenced cluster
    // (managed mode). Used to generate the correct number of StatefulSet pod
    // endpoints. Only used when ClusterRef is set.
    // +optional
    // +kubebuilder:default=3
    // +kubebuilder:validation:Minimum=1
    Replicas int32 `json:"replicas,omitempty"`
}

// SecretRefSpec references a Kubernetes Secret.
type SecretRefSpec struct {
    Name string `json:"name"`
    Key  string `json:"key,omitempty"`
}

// PolicySpec defines oslo.policy override configuration for an OpenStack service.
type PolicySpec struct {
    // Rules contains inline policy rule overrides.
    // Keys are oslo.policy rule names (e.g., "compute:create").
    // Values are oslo.policy rule definitions (e.g., "role:admin").
    // Inline rules take precedence over ConfigMap rules.
    // +optional
    Rules map[string]string `json:"rules,omitempty"`

    // ConfigMapRef references a user-provided ConfigMap containing a
    // "policy.yaml" key with rule overrides.
    // +optional
    ConfigMapRef *corev1.LocalObjectReference `json:"configMapRef,omitempty"`
}

testutil/

Provides test infrastructure shared across all operator test suites. Organized into subdirectories:

Subdirectory	Purpose
assertions/	testing.TB-based assertion helpers (AssertCondition, AssertConditionWithReason, AssertConditionMissing, AssertResourceExists, AssertResourceNotExists, EventuallyCondition) — not Gomega matchers
builders/	Fluent builders for test Kubernetes resources (currently SecretBuilder)
envtest/	Shared envtest setup (SetupEnvTest, SkipIfEnvTestUnavailable, SharedScheme) and auto-discovered fake CRDs
fake_crds/	CRD manifests for third-party resources (cert-manager, external-secrets, gateway-api, k-orc, mariadb-operator, memcached-operator, rabbitmq-operator) needed in envtest — k-orc provides the OpenStack Resource Controller CRDs (ApplicationCredential, Service, Endpoint) used by the c5c3-operator (CC-0110)
simulators/	Simulators for external controllers (e.g. SimulateMariaDBReady, SimulateExternalSecretSync, SimulateJobComplete, SimulateCertificateReady)

Secret Flow Design Principle

Operators interact exclusively with Kubernetes Secrets. The full secret lifecycle flows through OpenBao and ESO, but operators are unaware of this — they only see standard Kubernetes Secrets.

┌─────────────────────────────────────────────────────────────────────────────┐
│                       SECRET FLOW                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────┐    ExternalSecret    ┌─────────────┐    Operator reads        │
│  │ OpenBao  │ ──────────────────▶  │ K8s Secret  │ ──────────────────▶      │
│  │          │    (ESO syncs)       │             │    secret values         │
│  └──────────┘                      └─────────────┘                          │
│       ▲                                                                     │
│       │            PushSecret      ┌─────────────┐    Operator writes       │
│       └─────────────────────────── │ K8s Secret  │ ◀──────────────────      │
│              (ESO pushes)          │ (generated) │    generated secrets     │
│                                    └─────────────┘                          │
│                                                                             │
│  Direction 1 (read):                                                        │
│  OpenBao → ESO ExternalSecret → K8s Secret → Operator reads → Config        │
│                                                                             │
│  Direction 2 (write-back):                                                  │
│  Operator generates secret → K8s Secret → ESO PushSecret → OpenBao          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

See Secret Management for OpenBao architecture and policies. See Credential Lifecycle for the bootstrap flow and PushSecret patterns.

Extra Packages / Plugin Installation (Build-Time)

Plugins and middleware (e.g., openstack-audit-middleware, keystone-keycloak-backend) are Python packages that must be installed in the service container image at build time. The operator only configures them at runtime via the CRD.

A new file releases/<release>/extra-packages.yaml defines additional Python packages per service, analogous to the existing source-refs.yaml:

# releases/2025.2/extra-packages.yaml
keystone:
  - openstack-audit-middleware
  - keystone-keycloak-backend
nova:
  - openstack-audit-middleware
neutron:
  - openstack-audit-middleware
glance:
  - openstack-audit-middleware
cinder:
  - openstack-audit-middleware
placement:
  - openstack-audit-middleware

During the container image build (see Build Pipeline), the uv pip install step is extended to include extra packages:

# In the venv-builder stage
RUN uv pip install \
    --constraint /upper-constraints.txt \
    /src/${SERVICE} \
    ${EXTRA_PACKAGES}

The EXTRA_PACKAGES build argument is populated from extra-packages.yaml by the CI pipeline. This pattern is generic — any additional Python package (middleware, driver, backend plugin) can be added to extra-packages.yaml without modifying the Dockerfile.

Usage Example

A simplified example showing how the Keystone reconciler uses shared library packages (Keystone does not use messaging — for a messaging example, see the messaging/ package above):

import (
    "github.com/c5c3/forge/internal/common/conditions"
    "github.com/c5c3/forge/internal/common/config"
    "github.com/c5c3/forge/internal/common/database"
    "github.com/c5c3/forge/internal/common/deployment"
    "github.com/c5c3/forge/internal/common/job"
    // "github.com/c5c3/forge/internal/common/messaging"  // planned — Nova, Neutron, Cinder
    "github.com/c5c3/forge/internal/common/secrets"
    "github.com/c5c3/forge/internal/common/plugins"
    "github.com/c5c3/forge/internal/common/policy"
)

func (r *KeystoneReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    keystone := &keystonev1alpha1.Keystone{}
    if err := r.Get(ctx, req.NamespacedName, keystone); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Check ESO-provided secrets
    ready, err := secrets.WaitForExternalSecret(ctx, r.Client,
        client.ObjectKey{Namespace: keystone.Namespace, Name: keystone.Spec.Database.SecretRef.Name})
    if !ready {
        conditions.SetCondition(&keystone.Status.Conditions, metav1.Condition{
            Type:   "SecretsReady",
            Status: metav1.ConditionFalse,
            Reason: "WaitingForESO",
        })
        return ctrl.Result{RequeueAfter: 15 * time.Second}, nil
    }

    // Ensure database
    dbReady, err := database.EnsureDatabase(ctx, r.Client, r.Scheme, keystone,
        keystone.Spec.Database)
    if !dbReady {
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }

    // Render config with plugin support
    iniConfig := config.MergeDefaults(buildKeystoneConfig(keystone), keystoneDefaults)
    pluginConfig, _ := plugins.RenderPluginConfig(keystone.Spec.Plugins)
    // ... merge pluginConfig into iniConfig ...

    // Returns the generated content-hashed ConfigMap name.
    configMapName, err := config.CreateImmutableConfigMap(ctx, r.Client, r.Scheme, keystone,
        "keystone-config", keystone.Namespace,
        map[string]string{"keystone.conf": config.RenderINI(iniConfig)})

    // Create deployment (referencing configMapName)
    _, err = deployment.EnsureDeployment(ctx, r.Client, r.Scheme, keystone, deploymentSpec)

    return ctrl.Result{}, nil
}