Implementation
CobaltCore's architecture is documented across the preceding chapters — from the modular operator architecture and CRD definitions to the config generation pipeline and secret management. This chapter bridges the gap between architecture and code by documenting the concrete implementation of operators using Operator SDK and controller-runtime.
The implementation follows a Keystone-first strategy: the Keystone Operator is built first as a complete reference implementation, establishing patterns and shared libraries that all subsequent operators will reuse.
Implementation Philosophy
| Principle | Description |
|---|---|
| One Operator per Service | Each OpenStack service has a dedicated operator with its own reconciliation loop, CRD, and release lifecycle |
| Shared Library (Monorepo) | Common patterns (database, config, secrets, conditions) live in internal/common/ and are shared via Go Workspace |
| Operator SDK + controller-runtime | Standard tooling — Kubebuilder markers, controller-gen for CRD/RBAC generation, envtest for integration tests |
| Go over Templates | Configuration files are rendered from Go structs, not template languages — enabling type safety and testability |
| Secrets via ESO | Operators read Kubernetes Secrets (created by ESO from OpenBao) — they never interact with OpenBao directly |
Implementation Roadmap
text
┌─────────────────────────────────────────────────────────────────────────────┐
│ IMPLEMENTATION ROADMAP │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Phase 1: Foundation │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Go Workspace + Monorepo Setup │ │
│ │ Shared Library (internal/common/) │ │
│ │ CI/CD Pipeline + Helm Chart Skeleton │ │
│ │ E2E Test Framework (Chainsaw) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Phase 2: Keystone Operator (Reference Implementation) │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Keystone CRD (v1alpha1) + Webhooks │ │
│ │ Keystone Reconciler (DB, Config, Fernet, Deployment, Bootstrap) │ │
│ │ Keystone Dependencies (MariaDB, Memcached, ESO Secrets) │ │
│ │ Full Test Suite (Unit + envtest + Chainsaw E2E) │ │
│ │ Helm Chart + FluxCD Integration │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Phase 3a: c5c3-operator │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ ControlPlane CRD + Orchestration Reconciler │ │
│ │ Infrastructure CR creation + Readiness waiting │ │
│ │ Service CR projection (ControlPlane → per-service CRs) │ │
│ │ K-ORC integration (bootstrap imports, Services, Endpoints, Users) │ │
│ │ SecretAggregate + CredentialRotation CRDs │ │
│ │ Phased rollout strategy (updatePhase tracking) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Phase 3b: Remaining Service Operators │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Glance Operator (MariaDB, Keystone, Ceph) │ │
│ │ Placement Operator (MariaDB, Keystone) │ │
│ │ Nova Operator (MariaDB, RabbitMQ, Keystone, Ceph, Cells) │ │
│ │ Neutron Operator (MariaDB, RabbitMQ, Keystone, OVN) │ │
│ │ Cinder Operator (MariaDB, RabbitMQ, Keystone, Ceph) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘Technology Stack
| Component | Version | Purpose |
|---|---|---|
| Go | 1.25+ | Operator implementation language |
| Operator SDK | 1.38+ | Project scaffolding and code generation |
| controller-runtime | 0.23+ | Reconciler framework, manager, caching |
| Kubebuilder | 4.x | Code generation markers for CRDs, RBAC, webhooks |
| Chainsaw | 0.2+ | Declarative E2E testing for Kubernetes operators |
| Helm | 3.x | Operator packaging and deployment |
| GitHub Actions | — | CI/CD pipeline |
Why Keystone First
Keystone is the ideal starting point for implementation:
- Simplest dependency graph — Keystone requires only MariaDB and Memcached. No RabbitMQ, no Valkey, no Ceph. This minimizes the infrastructure needed for development and testing.
- Foundation for all other services — Every OpenStack service authenticates against Keystone. Building it first unblocks all subsequent operators.
- Non-trivial reconciliation patterns — Fernet key rotation (generation, CronJob, rolling restart, OpenBao backup via PushSecret) exercises the full reconciliation lifecycle without excessive complexity.
- Config generation reference — The config generation pipeline can be validated end-to-end with
keystone.confbefore tackling more complex services like Nova (multiple config files, cell architecture). - Plugin/middleware pattern — Keystone's
api-paste.inipipeline and domain-specific configs (e.g., Keycloak backend) establish the generic plugin framework that all services will use.
Further Reading
- Project Setup — Go workspace, monorepo layout, Makefile targets
- Shared Library —
internal/common/package design, OpenBao/ESO integration - CRD Implementation — Go types, Kubebuilder markers, webhooks
- Keystone Reconciler — Sub-reconciler pattern, error handling
- Keystone Dependencies — Secret flow, MariaDB, Memcached, Fernet rotation
- Testing — Unit, integration (envtest), E2E (Chainsaw)
- CI/CD & Packaging — GitHub Actions, Helm charts, FluxCD
- C5C3 Operator — ControlPlane CRD, orchestration reconciler, rollout strategy
- OpenBao Deployment — Deployment, initialization, secret engines, policies
For the configuration lifecycle concepts that inform this implementation, see Service Configuration.