Secret Management
Design Principle
ALL secrets are centrally managed via OpenBao. OpenBao is the single source of truth for all credentials in CobaltCore — bootstrap passwords, service credentials, database credentials, Ceph keys, kubeconfigs, TLS certificates, and messaging credentials.
Integration is done via the External Secrets Operator (ESO), which runs in each cluster and reads secrets from OpenBao. Existing operators continue to read Kubernetes Secrets — no code changes to operators needed. PushSecret CRDs write operator-generated secrets (Ceph Keys, the K-ORC admin Application Credential, and per-pod service-user passwords) back to OpenBao.
OpenBao Architecture
OpenBao runs as an HA cluster in the Management Cluster (namespace openbao-system):
┌─────────────────────────────────────────────────────────────────────────────┐
│ MANAGEMENT CLUSTER — openbao-system │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ OpenBao Node 0 │ │ OpenBao Node 1 │ │ OpenBao Node 2 │ │
│ │ (Leader/Standby)│ │ (Standby) │ │ (Standby) │ │
│ ├──────────────────┤ ├──────────────────┤ ├──────────────────┤ │
│ │ Raft Storage │◀─┼──────────────────┼──▶│ Raft Storage │ │
│ │ Seal: Transit/ │ │ Raft Storage │ │ Seal: Transit/ │ │
│ │ Auto-Unseal│ │ Seal: Transit/ │ │ Auto-Unseal│ │
│ └──────────────────┘ │ Auto-Unseal│ └──────────────────┘ │
│ └──────────────────┘ │
│ │
│ Raft Consensus: 3 Replicas, integrated storage │
│ Listener: HTTPS + mutual TLS on Port 8200 (CC-0107) │
│ Service: openbao.openbao-system.svc.cluster.local │
│ │
└─────────────────────────────────────────────────────────────────────────────┘mTLS admission gate (CC-0107). The OpenBao listener does not merely terminate server TLS — it sets
tls_require_and_verify_client_cert = true, so every client (ESO, the Raftretry_joinpeers, and the in-podbaoCLI) must present a certificate signed by a dedicated OpenBao CA (openbao-ca-issuer). ESO therefore authenticates with a client certificate (eso-openbao-client-tls) on top of Kubernetes auth: a leaked Kubernetes ServiceAccount token alone cannot reach the OpenBao API. See OpenBao Deployment — Deployment via FluxCD.
Secret Engines
| Engine | Mount Path | Purpose | Example Paths |
|---|---|---|---|
| KV v2 | kv-v2/bootstrap/ | Bootstrap credentials | kv-v2/bootstrap/keystone-admin, kv-v2/bootstrap/service-passwords |
| KV v2 | kv-v2/openstack/ | OpenStack service secrets | kv-v2/openstack/admin/app-credential, kv-v2/openstack/nova/pods/nova-api-0/user, kv-v2/openstack/nova/db |
| KV v2 | kv-v2/infrastructure/ | Infrastructure credentials | kv-v2/infrastructure/mariadb, kv-v2/infrastructure/rabbitmq, kv-v2/infrastructure/valkey |
| KV v2 | kv-v2/ceph/ | Ceph auth keys | kv-v2/ceph/client-nova, kv-v2/ceph/client-cinder, kv-v2/ceph/client-glance |
| PKI | pki/ | TLS certificates | pki/issue/openstack-internal, pki/issue/api-external |
| Database | database/mariadb/ | Dynamic DB credentials | database/mariadb/creds/nova-rw, database/mariadb/creds/nova-ro |
Auth Methods
| Auth Method | Mount Path | Usage | Cluster |
|---|---|---|---|
| Kubernetes | kubernetes/management | ESO, FluxCD in Management Cluster | Management |
| Kubernetes | kubernetes/control-plane | ESO in Control Plane Cluster | Control Plane |
| Kubernetes | kubernetes/hypervisor | ESO in Hypervisor Cluster | Hypervisor |
| Kubernetes | kubernetes/storage | ESO in Storage Cluster | Storage |
| AppRole | approle/ci-cd | CI/CD pipelines for secret provisioning | External |
Policies (Least Privilege)
| Role | Allowed Paths | Capabilities |
|---|---|---|
eso-control-plane | kv-v2/data/bootstrap/*, kv-v2/data/openstack/*, kv-v2/data/infrastructure/*, kv-v2/data/ceph/* | read |
eso-hypervisor | kv-v2/data/ceph/client-nova, kv-v2/data/openstack/nova/compute-*, kv-v2/data/openstack/nova/pods/*/user | read |
eso-storage | kv-v2/data/ceph/* | read, create, update |
eso-management | kv-v2/data/bootstrap/*, kv-v2/data/infrastructure/* | read |
push-ceph-keys | kv-v2/data/ceph/* | create, update |
push-admin-app-cred | kv-v2/data/openstack/admin/app-credential | create, update |
push-pod-users | kv-v2/data/openstack/*/pods/*/user | create, update |
The "never in a pod" invariant is enforced at the ExternalSecret/namespace level, not by these coarse per-cluster read policies. The admin password (
kv-v2/bootstrap/keystone-admin) and the admin App Cred (kv-v2/openstack/admin/app-credential) are materialized only into the Keystone bootstrap Job, the c5c3-operator, andorc-system(K-ORC) — never via an ExternalSecret targeting a workload namespace, and never mounted into a workload pod. Per-pod user passwords (…/pods/<pod>/user) are each materialized into exactly one pod. |ci-cd-provisioner|kv-v2/data/*| create, update, read | |db-exporter|database/mariadb/creds/*-ro| read | |pki-issuer|pki/issue/*| create, update |
ESO Integration
The External Secrets Operator (ESO) runs in each cluster and synchronizes secrets from OpenBao. ESO uses ClusterSecretStore resources (cluster-wide scope, as opposed to namespace-scoped SecretStore) to define the connection to OpenBao:
┌─────────────────────────────────────────────────────────────────────────────┐
│ ESO INTEGRATION PATTERN │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Per Cluster: │
│ │
│ ┌────────────────────────┐ ┌────────────────────────┐ │
│ │ ClusterSecretStore │ │ ExternalSecret │ │
│ │ ─────────────────── │ │ ────────────── │ │
│ │ provider: vault │ │ secretStoreRef: │ │
│ │ server: https:// │◀────│ clusterStore │ │
│ │ openbao.mgmt:8200 │ │ target: │ │
│ │ auth: │ │ name: <k8s-sec> │ │
│ │ kubernetes: │ │ data: │ │
│ │ mountPath: ... │ │ - remoteRef: │ │
│ │ role: eso-<cls> │ │ key: <path> │ │
│ └────────────────────────┘ └────────────────────────┘ │
│ │
│ ┌────────────────────────┐ │
│ │ PushSecret │ (Return channel: K8s Secret → OpenBao) │
│ │ ────────── │ │
│ │ selector: │ │
│ │ name: <k8s-sec> │ │
│ │ data: │ │
│ │ - match: │ │
│ │ remoteRef: │ │
│ │ remoteKey: │ │
│ │ <vaultpath> │
│ └────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘Database Secret Engine
The OpenBao Database Secret Engine dynamically generates short-lived database credentials. This is used by the OpenStack Database Exporters to obtain read-only access to individual service databases.
Each OpenStack service database has a corresponding read-only role configured in OpenBao:
| Role | Database | Permissions | Consumer |
|---|---|---|---|
nova-ro | nova | SELECT | openstack-database-exporter-nova |
neutron-ro | neutron | SELECT | openstack-database-exporter-neutron |
cinder-ro | cinder | SELECT | openstack-database-exporter-cinder |
glance-ro | glance | SELECT | openstack-database-exporter-glance |
keystone-ro | keystone | SELECT | openstack-database-exporter-keystone |
heat-ro | heat | SELECT | openstack-database-exporter-heat |
ironic-ro | ironic | SELECT | openstack-database-exporter-ironic |
octavia-ro | octavia | SELECT | openstack-database-exporter-octavia |
placement-ro | placement | SELECT | openstack-database-exporter-placement |
Each role creates a MariaDB user with SELECT-only grants on the respective database. Credentials have a configurable TTL (default: 1h) and are automatically revoked by OpenBao after expiry. ESO refreshes the credentials before expiry to ensure uninterrupted exporter operation.
Complete Secret Inventory
| Secret Type | OpenBao Path | Engine | Consumer(s) | Cluster |
|---|---|---|---|---|
| Keystone Admin Password (root of trust) | kv-v2/bootstrap/keystone-admin | KV v2 | Keystone Bootstrap Job; c5c3-operator (mints/rotates admin App Cred) — never a workload pod | Control Plane |
| Admin Application Credential | kv-v2/openstack/admin/app-credential | KV v2 | K-ORC Controller (orc-system) — never a workload pod | Control Plane |
| Per-Pod Service User Password | kv-v2/openstack/<service>/pods/<pod>/user | KV v2 | exactly one workload pod (OS_KEYSTONE_AUTHTOKEN__*) | Control Plane / Hypervisor |
| MariaDB Root Credentials | kv-v2/infrastructure/mariadb | KV v2 | MariaDB Operator | Control Plane |
| RabbitMQ Credentials | kv-v2/infrastructure/rabbitmq | KV v2 | RabbitMQ Operator | Control Plane |
| Valkey Auth | kv-v2/infrastructure/valkey | KV v2 | Valkey Operator | Control Plane |
| Nova DB Credentials | kv-v2/openstack/nova/db | KV v2 | Nova API | Control Plane |
| Neutron DB Credentials | kv-v2/openstack/neutron/db | KV v2 | Neutron API | Control Plane |
| Glance DB Credentials | kv-v2/openstack/glance/db | KV v2 | Glance API | Control Plane |
| Cinder DB Credentials | kv-v2/openstack/cinder/db | KV v2 | Cinder API | Control Plane |
| Ceph Client Key (Nova) | kv-v2/ceph/client-nova | KV v2 | Nova Compute, Hypervisor Node Agent | Hypervisor |
| Ceph Client Key (Cinder) | kv-v2/ceph/client-cinder | KV v2 | Cinder Volume | Control Plane |
| Ceph Client Key (Glance) | kv-v2/ceph/client-glance | KV v2 | Glance API | Control Plane |
| Nova Compute Credentials | kv-v2/openstack/nova/compute-config | KV v2 | Nova Compute Agent | Hypervisor |
| OVN Config | kv-v2/openstack/ovn/config | KV v2 | ovn-controller | Hypervisor |
| Kubeconfig Control Plane | kv-v2/infrastructure/kubeconfig-cp | KV v2 | FluxCD | Management |
| Kubeconfig Hypervisor | kv-v2/infrastructure/kubeconfig-hv | KV v2 | FluxCD | Management |
| Kubeconfig Storage | kv-v2/infrastructure/kubeconfig-st | KV v2 | FluxCD | Management |
| DB Exporter RO Creds (Nova) | database/mariadb/creds/nova-ro | DB | openstack-database-exporter-nova | Control Plane |
| DB Exporter RO Creds (Neutron) | database/mariadb/creds/neutron-ro | DB | openstack-database-exporter-neutron | Control Plane |
| DB Exporter RO Creds (Cinder) | database/mariadb/creds/cinder-ro | DB | openstack-database-exporter-cinder | Control Plane |
| DB Exporter RO Creds (Glance) | database/mariadb/creds/glance-ro | DB | openstack-database-exporter-glance | Control Plane |
| DB Exporter RO Creds (Keystone) | database/mariadb/creds/keystone-ro | DB | openstack-database-exporter-keystone | Control Plane |
| DB Exporter RO Creds (Heat) | database/mariadb/creds/heat-ro | DB | openstack-database-exporter-heat | Control Plane |
| DB Exporter RO Creds (Ironic) | database/mariadb/creds/ironic-ro | DB | openstack-database-exporter-ironic | Control Plane |
| DB Exporter RO Creds (Octavia) | database/mariadb/creds/octavia-ro | DB | openstack-database-exporter-octavia | Control Plane |
| DB Exporter RO Creds (Placement) | database/mariadb/creds/placement-ro | DB | openstack-database-exporter-placement | Control Plane |
| TLS Certificates | pki/issue/openstack-internal | PKI | OpenStack APIs | Control Plane |
Multi-Cluster Secret Distribution
┌─────────────────────────────────────────────────────────────────────────────┐
│ MULTI-CLUSTER SECRET DISTRIBUTION │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────┐ │
│ │ OpenBao │ │
│ │ (Management Cluster) │ │
│ │ │ │
│ │ kv-v2/bootstrap/* │ │
│ │ kv-v2/openstack/* │ │
│ │ kv-v2/infrastructure/* │ │
│ │ kv-v2/ceph/* │ │
│ │ pki/* │ │
│ └───┬─────┬─────┬─────────┘ │
│ │ │ │ │
│ ┌────────┘ │ └────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ ESO │ │ ESO │ │ ESO │ │
│ │ Control Plane │ │ Hypervisor │ │ Storage │ │
│ ├────────────────┤ ├────────────────┤ ├────────────────┤ │
│ │ ExternalSecret │ │ ExternalSecret │ │ PushSecret │ │
│ │ → K8s Secrets: │ │ → K8s Secrets: │ │ (Ceph Keys → │ │
│ │ - DB Creds │ │ - Ceph Keys │ │ OpenBao) │ │
│ │ - SvcPasswords│ │ - Nova Config │ │ │ │
│ │ - AppCreds │ │ - OVN Config │ │ ExternalSecret │ │
│ │ - Ceph Keys │ │ │ │ → K8s Secrets: │ │
│ └────────────────┘ └────────────────┘ │ - Ceph Config │ │
│ └────────────────┘ │
│ │
│ ESO in Management Cluster itself: │
│ ExternalSecret → K8s Secrets: Kubeconfigs, FluxCD Secrets │
│ │
└─────────────────────────────────────────────────────────────────────────────┘Ceph Key Flow
The flow for Ceph credentials from Rook Operator via OpenBao to the Libvirt daemon:
┌─────────────────────────────────────────────────────────────────────────────┐
│ CEPH KEY FLOW (via OpenBao) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ STORAGE CLUSTER │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 1. Rook Operator creates CephClient + K8s Secret │ │
│ │ Secret: rook-ceph-client-openstack-nova (key: AQBxxxx==) │ │
│ │ │ │
│ │ 2. PushSecret writes key to OpenBao │ │
│ │ PushSecret → kv-v2/ceph/client-nova │ │
│ └───────────────────────────────┬─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ MANAGEMENT CLUSTER │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 3. OpenBao stores Ceph key │ │
│ │ Path: kv-v2/data/ceph/client-nova │ │
│ └───────────────────────────────┬─────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ HYPERVISOR CLUSTER │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 4. ESO ExternalSecret reads from OpenBao │ │
│ │ → Creates K8s Secret: ceph-client-nova (namespace: openstack) │ │
│ │ │ │
│ │ 5. Nova Compute DaemonSet mounts secret │ │
│ │ → /etc/ceph/ceph.client.nova.keyring │ │
│ │ │ │
│ │ 6. Hypervisor Node Agent creates Libvirt Secret on each node │ │
│ │ → virsh secret-define + virsh secret-set-value │ │
│ │ │ │
│ │ 7. Libvirt uses secret for RBD access │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘Bootstrap Sequence
The bootstrap sequence with OpenBao as central secret store:
| Phase | Description | Secret Source |
|---|---|---|
| Phase 0 | Initialize OpenBao, Unseal, configure Secret Engines + Auth Methods | Manual / CI-CD |
| Phase 1 | Write bootstrap secrets to OpenBao (Admin PW, Service Passwords) | CI-CD → OpenBao |
| Phase 2 | Deploy ESO in all clusters, configure ClusterSecretStores | FluxCD |
| Phase 3 | ESO creates K8s Secrets from OpenBao in all clusters | ESO → OpenBao |
| Phase 4 | Infrastructure Operators start (MariaDB, RabbitMQ, Valkey) | K8s Secrets |
| Phase 5 | Keystone Bootstrap with Admin credentials from OpenBao | K8s Secrets |
| Phase 6 | c5c3-operator mints the admin App Cred, then creates Keystone Services, Endpoints, and per-pod Service Users (via K-ORC) | Keystone API |
| Phase 7 | PushSecrets write the admin App Cred and per-pod user passwords back to OpenBao | PushSecret → OpenBao |
| Phase 8 | ESO distributes all secrets to target clusters, services start | ESO → K8s Secrets |
Credentials Never Live in ConfigMaps
A cross-cutting rule for all service operators (CC-0080): resolved credentials are never rendered into the immutable config ConfigMap. The Keystone Operator materializes the full database connection URL (including the password) into a derived <name>-db-connection Secret and injects it into every workload via the OS_DATABASE__CONNECTION environment variable. The keystone.conf ConfigMap carries no password. This keeps the at-rest credential exposure confined to objects treated as Secrets.
Credential Rotation
Three distinct rotation mechanisms exist; do not conflate them:
1. Keystone cryptographic-key rotation (built, keystone-operator). The operator rotates Fernet token keys and credential encryption keys using a split-compute-write boundary (CC-0081): the rotation CronJob — holding only narrow get+patch RBAC — writes new key material to a dedicated staging Secret; the operator then validates it (correct length, no duplicates, count in range) and applies it to the production Secret with its own privileged ServiceAccount. This keeps token-forgery primitives out of the CronJob's reach. Keys are projected into pods and rotate in place (no rolling restart, CC-0074), and credential-key rotation additionally runs keystone-manage credential_migrate to re-encrypt stored credentials. Backups are pushed per-CR to OpenBao at openstack/keystone/{name}/{fernet,credential}-keys (CC-0093), and an OpenBao finalizer purges them on deletion (CC-0079). A separate trust-flush CronJob purges expired trust delegations (CC-0057).
2. Workload service-user rotation = pod recreation (planned, c5c3-operator). Per-pod service users are not rotated in place; a fresh credential is produced by recreating the pod-identity (issue #30, D4). Any deploy / rollout / reschedule / crash-restart (ephemeral) or intentional recreation (per-replica), and maxAge-driven rolling recreation for long-lived pods, deletes the old Keystone user (finalizer revokes its tokens) and provisions a new one with a freshly generated password. There is no per-service Application Credential rotation. See Credential Lifecycle — Credential Rotation.
2b. Admin Application Credential rotation (planned, c5c3-operator). The single admin App Cred — K-ORC's only credential — is rotated by restricted + password-driven re-mint: the operator mints a fresh restricted App Cred from the admin password (OpenBao), writes it (PushSecret → OpenBao → ESO → k-orc-clouds-yaml), and deletes the old one after a short grace window. This is the only target of the CredentialRotation CRD.
3. Admin credential rotation (planned, keystone-operator). The Keystone admin password — written once during bootstrap — has no rotation path today. The planned design re-runs the idempotent keystone-manage bootstrap Job whenever ESO syncs a new password from kv-v2/bootstrap/keystone-admin: the bootstrap pod template carries a hash of the password so a rotation changes the Job's PodSpec hash and RunJob (CC-0005) re-runs it, applying the new password to Keystone. The new password is produced either externally in OpenBao (default) or by an opt-in operator-scheduled CronJob using the same split-compute-write boundary as key rotation (CC-0081). The admin password is single-valued (a hard cutover, no grace window) but low blast radius — running services use their own application credentials, not the admin password. See Admin Credential Rotation for the full mechanism.
For the CRD definitions of SecretAggregate and CredentialRotation, see CRDs.
Further Reading
- OpenBao Deployment — Deployment, initialization, secret engines, auth methods, policies, and bootstrap automation
- Credential Lifecycle — GitOps-driven credential flow with FluxCD
- C5C3 Operator — CredentialRotation CRD implementation and rotation flow
- Brownfield Integration — Secret management for existing OpenStack deployments