CRDs

CobaltCore defines several Custom Resource Definitions (CRDs) for declarative management of hypervisors, evictions, migrations, storage arbiters, network status, Keystone identity resources (via K-ORC), control plane orchestration, secret aggregation, and credential rotation. The following CRDs form the central API interface of the system. For Cortex scheduling CRDs, see Cortex Scheduling.

CRD API Version Maturity: CRDs within the same API group may have different version levels reflecting their maturity. The Hypervisor and Eviction CRDs (hypervisor.c5c3.io/v1) have reached stable status after extensive production use, while the Migration CRD (hypervisor.c5c3.io/v1alpha1) is newer and its API surface is still evolving. New CRDs generally start at v1alpha1 and are promoted to v1beta1 then v1 via conversion webhooks as they stabilize.

Hypervisor CRD (hypervisor.c5c3.io/v1)

Represents a hypervisor node in the cluster. For the complete hypervisor state machine, see Hypervisor Lifecycle.

apiVersion: hypervisor.c5c3.io/v1
kind: Hypervisor
metadata:
  name: hypervisor-001
spec:
  version: "1.0.0"                    # Desired OS version
  reboot: false                       # Request reboot after upgrade
  evacuateOnReboot: true              # Evacuation before reboot
  lifecycleEnabled: true              # Lifecycle management enabled
  skipTests: false                    # Skip onboarding tests
  customTraits: []                    # Custom OpenStack traits
  aggregates: []                      # Host aggregates
  allowedProjects: []                 # Allowed OpenStack projects
  highAvailability: true              # HA handling enabled
  createCertManagerCertificate: false # Create Cert-Manager certificate
  installCertificate: true            # Certificate installation via agent
  maintenance: ""                     # manual|auto|ha|termination
status:
  libVirtVersion: "8.0.0"
  operatingSystem:
    version: "1.0.0"
    variantID: "garden-linux"
    kernelRelease: "6.1.0"
    hardwareVendor: "Dell"
    hardwareModel: "PowerEdge R750"
  capabilities:
    cpuArch: "x86_64"
    memory: "512Gi"
    cpus: "128"
  domainCapabilities:
    arch: "x86_64"
    hypervisorType: "kvm"             # "kvm" or "cloud-hypervisor"
    supportedDevices: ["video", "network"]
    supportedCpuModes: ["host-passthrough"]
    supportedFeatures: ["sev", "sgx"]
  instances:
    - id: "abc123"
      name: "vm-001"
      active: true
  numInstances: 5
  hypervisorId: "hv-001"
  serviceId: "svc-001"
  conditions:
    - type: Ready
      status: "True"
      reason: Ready
    - type: Onboarding
      status: "False"

Condition Types:

Type	Description
Onboarding	Node is undergoing initial configuration and OpenStack registration
Offboarded	Node has completed the offboarding process and is no longer active
Ready	All onboarding steps and tests passed; node can accept workloads
Terminating	Node is being terminated and resources are being cleaned up
Tainted	Node has been flagged with a problem preventing normal scheduling
TraitsUpdated	OpenStack traits have been synchronized from spec to Nova
AggregatesUpdated	Host aggregate assignments have been synchronized from spec to Nova

Maintenance Modes:

Mode	Description
manual	Manual maintenance mode by external user
auto	Automatic maintenance mode
ha	High availability maintenance mode
termination	Internal mode during termination

For maintenance mode details, see Hypervisor Lifecycle -- Maintenance Mode. For HA-triggered maintenance, see High Availability.

Eviction CRD (hypervisor.c5c3.io/v1)

Represents an eviction request for a hypervisor. For the full eviction flow, see Hypervisor Lifecycle -- Eviction Process.

apiVersion: hypervisor.c5c3.io/v1
kind: Eviction
metadata:
  name: eviction-001
spec:
  hypervisor: "hypervisor-001"    # Name of hypervisor to evacuate
  reason: "Planned maintenance"   # Reason for eviction
status:
  hypervisorServiceId: "svc-001"
  outstandingRamMb: 16384
  outstandingInstances:
    - "vm-001"
    - "vm-002"
  conditions:
    - type: Evicting
      status: "True"
      reason: Running

Eviction Condition Types:

Type	Description
MigratingInstance	Migration status of a server
PreflightChecksSucceeded	Preflight checks successful
HypervisorReEnabled	Hypervisor re-enabled
HypervisorDisabled	Hypervisor disabled
Evicting	Eviction status

Migration CRD (hypervisor.c5c3.io/v1alpha1)

Represents an ongoing VM migration with detailed metrics.

apiVersion: hypervisor.c5c3.io/v1alpha1
kind: Migration
metadata:
  name: migration-vm-001
spec: {}
status:
  origin: "hypervisor-001"
  destination: "hypervisor-002"
  type: "live"
  started: "2024-01-15T10:00:00Z"
  operation: "running"

  # Timing metrics
  timeElapsed: "45s"
  timeRemaining: "30s"
  setupTime: "5s"
  downtime: "0s"

  # Data transfer
  dataTotal: "32Gi"
  dataProcessed: "20Gi"
  dataRemaining: "12Gi"

  # Memory metrics
  memTotal: "16Gi"
  memProcessed: "10Gi"
  memRemaining: "6Gi"
  memBps: "1Gi"
  memDirtyRate: "100Mi"
  memIteration: 3
  memPageSize: "4Ki"
  memNormal: 2621440
  memConstant: 1048576
  memPostcopyRequests: 0

  # Disk metrics
  diskTotal: "16Gi"
  diskProcessed: "10Gi"
  diskRemaining: "6Gi"
  diskBps: "500Mi"

  # Additional information
  autoConvergeThrottle: "0"
  errMsg: ""

RemoteCluster CRD (ceph.c5c3.io/v1alpha1)

Defines access to a remote Kubernetes cluster for external arbiter deployment. For the storage cluster integration, see Storage Architecture.

apiVersion: ceph.c5c3.io/v1alpha1
kind: RemoteCluster
metadata:
  name: arbiter-site
spec:
  # Namespace in remote cluster for arbiter deployment
  namespace: external-arbiter
  # Reference to secret with kubeconfig
  accesskeyRef:
    name: arbiter-kubeconfig
    key: "kubeconfig.yaml"
  # Interval for health checks
  checkInterval: 1m
  # Request timeout for remote client
  timeout: 10s
status:
  state: Ready  # Init|Progressing|Error|Ready|Deleting
  message: "Cluster reachable and permissions verified"
  conditions:
    - type: SecretAvailable
      status: "True"
    - type: ConfigValid
      status: "True"
    - type: ClusterReachable
      status: "True"
    - type: HasEnoughPermissions
      status: "True"

RemoteArbiter CRD (ceph.c5c3.io/v1alpha1)

Defines a Ceph Monitor (arbiter) to be deployed in a RemoteCluster.

apiVersion: ceph.c5c3.io/v1alpha1
kind: RemoteArbiter
metadata:
  name: stretched-cluster-arbiter
spec:
  # Reference to RemoteCluster or inline spec
  remoteCluster:
    name: arbiter-site  # Name of RemoteCluster in same namespace
  # Reference to Rook-managed CephCluster
  cephCluster:
    name: my-cluster
    namespace: rook-ceph
  # Prefix for monitor ID (e.g., "ext-a", "ext-b")
  monIdPrefix: "ext-"
  # Interval for health checks
  checkInterval: 1m
  # Optional: Service configuration for arbiter exposure
  service:
    type: NodePort
    nodeIp: 10.10.0.1
  # Optional: Pod configuration
  deployment:
    nodeSelector:
      node-role: arbiter
status:
  state: Ready  # Init|Progressing|Error|Ready|Deleting
  monId: "ext-a"  # Reserved monitor ID
  message: "Arbiter deployed and joined quorum"
  conditions:
    - type: RemoteClusterReady
      status: "True"
    - type: CephClusterReady
      status: "True"
    - type: ArbiterDeploymentReady
      status: "True"

Use Case for Stretched Cluster:

┌─────────────────────┐     ┌─────────────────────┐     ┌─────────────────────┐
│    Datacenter A     │     │    Datacenter B     │     │   Arbiter Site      │
│    (Storage Cluster)│     │    (Storage Cluster)│     │   (Arbiter Cluster) │
│                     │     │                     │     │                     │
│  MON + OSDs         │     │  MON + OSDs         │     │  MON only           │
│  Ext. Arbiter Op ───┼─────┼─────────────────────┼─────┼─▶ (Tiebreaker)      │
│                     │     │                     │     │                     │
└─────────────────────┘     └─────────────────────┘     └─────────────────────┘

OVSNode CRD (ovs.c5c3.io/v1alpha1)

Represents the OVS status of a hypervisor node. Automatically created and updated by the OVS Agent. For the OVS bridge layout and OVN architecture, see Network Architecture.

apiVersion: ovs.c5c3.io/v1alpha1
kind: OVSNode
metadata:
  name: hypervisor-node-01
  namespace: ovn-system
spec:
  nodeRef:
    name: hypervisor-node-01
status:
  ovsVersion: "3.4.1"
  dpdkEnabled: true
  dpdkVersion: "23.11.1"
  bridges:
    - name: br-int
      ports: 156
      flows: 2847
      status: Active
    - name: br-ex
      ports: 2
      flows: 45
      status: Active
  interfaces:
    bonds:
      - name: bond0
        mode: balance-tcp
        status: Active
        members: 2
  ovnController:
    connected: true
    chassisId: "a1b2c3d4-..."
  conditions:
    - type: Ready
      status: "True"
    - type: OVSDBConnected
      status: "True"
    - type: OVNControllerConnected
      status: "True"

K-ORC Keystone CRDs (openstack.k-orc.cloud/v1alpha1)

K-ORC (Kubernetes OpenStack Resource Controller) provides CRDs for declarative management of Keystone resources. These CRDs are essential for the bootstrap process --- without them, OpenStack services cannot register in the service catalog or authenticate.

Common Fields:

All K-ORC CRDs share the following fields:

Field	Description
spec.cloudCredentialsRef.cloudName	Cloud name from clouds.yaml
spec.cloudCredentialsRef.secretName	Kubernetes Secret containing clouds.yaml
spec.managementPolicy	managed (full lifecycle) or unmanaged (read-only import)

Service CRD

Registers an OpenStack service in the Keystone service catalog.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Service
metadata:
  name: nova-service
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: nova
    type: compute
    description: "OpenStack Compute Service"

Endpoint CRD

Registers a service endpoint (public or internal) in the Keystone service catalog.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Endpoint
metadata:
  name: nova-public
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    serviceRef: nova-service
    interface: public
    url: "https://compute.example.com"
    region: RegionOne

---
apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Endpoint
metadata:
  name: nova-internal
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    serviceRef: nova-service
    interface: internal
    url: "http://nova-api.openstack.svc:8774"
    region: RegionOne

User CRD

Creates a service user in Keystone for service-to-service authentication.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: User
metadata:
  name: nova-service-user
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: nova
    domainRef: default
    passwordSecretRef:
      name: openstack-service-passwords
      key: nova-password

ApplicationCredential CRD

Creates an Application Credential for secure, restricted authentication.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: ApplicationCredential
metadata:
  name: k-orc-app-credential
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: k-orc-app-credential
    userRef: k-orc-service-user
    roles:
      - admin
    expiresAt: "2027-04-15T00:00:00Z"
    secretRef:
      name: k-orc-app-credential-secret

Domain CRD

Manages Keystone identity domains.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Domain
metadata:
  name: default-domain
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: Default
    description: "Default domain"
    enabled: true

Project CRD

Manages Keystone projects within a domain.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Project
metadata:
  name: service-project
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: service
    domainRef: default-domain
    description: "Service project for OpenStack services"
    enabled: true
    tags: ["infrastructure", "service"]

Role CRD

Manages RBAC roles in Keystone.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Role
metadata:
  name: admin-role
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: admin
    description: "Admin role"

Group CRD

Manages user groups in Keystone.

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Group
metadata:
  name: service-admins
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: service-admins
    domainRef: default-domain
    description: "Group for service administrators"

ControlPlane CRD (c5c3.io/v1alpha1)

The ControlPlane CRD is the top-level API for an entire OpenStack deployment. Users or GitOps apply a single CR, and the c5c3-operator handles everything downstream --- infrastructure provisioning, service CR creation, and phased rollout.

apiVersion: c5c3.io/v1alpha1
kind: ControlPlane
metadata:
  name: production
  namespace: openstack
spec:
  openStackRelease: "2025.2"         # Target OpenStack release
  region: RegionOne
  infrastructure:
    database:
      replicas: 3                    # MariaDB Galera cluster size
      storageClass: fast-ssd
      storageSize: 50Gi
    messaging:
      replicas: 3                    # RabbitMQ cluster size
    cache:
      replicas: 3                    # Memcached replicas
  services:
    keystone:
      enabled: true
      replicas: 3
      fernet:
        maxActiveKeys: 3
        rotationInterval: 24h
    nova:
      enabled: true
      replicas:
        api: 3
        scheduler: 2
        conductor: 2
      policyOverrides:                     # Per-service policy overrides
        rules:
          "compute:create": "role:member"
          "compute:delete": "role:admin"
    neutron:
      enabled: true
      replicas: 3
    glance:
      enabled: true
      replicas: 3
    cinder:
      enabled: true
      replicas:
        api: 3
        scheduler: 2
        volume: 2
    placement:
      enabled: true
      replicas: 3
  global:
    tls:
      enabled: true
      issuerRef:
        name: letsencrypt-prod
        kind: ClusterIssuer
    policyOverrides:                       # Global policy rules for all services
      rules:
        "admin_required": "role:admin"
  korc:
    cloudCredentialsRef:
      cloudName: openstack
      secretName: k-orc-clouds-yaml
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AllServicesReady
    - type: InfrastructureReady
      status: "True"
    - type: KeystoneReady
      status: "True"
  updatePhase: Complete              # Idle|Validating|UpdatingInfra|UpdatingKeystone|UpdatingServices|Verifying|Complete|RollingBack
  services:
    keystone:
      ready: true
      version: "28.0.0"
    nova:
      ready: true
      version: "32.1.0"

Update Phases:

Phase	Description
Idle	No update in progress
Validating	Pre-flight checks (CRD schema, dependency resolution)
UpdatingInfra	Infrastructure CRs being updated (MariaDB, RabbitMQ, Memcached)
UpdatingKeystone	Keystone CR updated and rolling out
UpdatingServices	Remaining service CRs updated and rolling out
Verifying	Post-update health checks
Complete	Update finished successfully
RollingBack	Reverting to previous state due to failure

For the full orchestration flow and Go type definitions, see C5C3 Operator.

SecretAggregate CRD (c5c3.io/v1alpha1)

Merges multiple Kubernetes Secrets into a single aggregated Secret. This is useful when a service needs credentials from multiple sources in a single mount.

apiVersion: c5c3.io/v1alpha1
kind: SecretAggregate
metadata:
  name: nova-all-credentials
  namespace: openstack
spec:
  sources:
    - secretRef:
        name: nova-db-credentials
    - secretRef:
        name: nova-rabbitmq-credentials
    - secretRef:
        name: nova-app-credential
      keys:                              # Select specific keys (empty = all)
        - application_credential_id
        - application_credential_secret
  target:
    name: nova-aggregated-credentials
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AllSourcesAvailable
    - type: SourcesResolved
      status: "True"

The c5c3-operator watches source Secrets and re-aggregates whenever any source changes. For the Go type definitions, see C5C3 Operator -- SecretAggregate.

CredentialRotation CRD (c5c3.io/v1alpha1)

Automates Application Credential rotation for OpenStack services. Works in coordination with K-ORC and the OpenBao/ESO pipeline.

apiVersion: c5c3.io/v1alpha1
kind: CredentialRotation
metadata:
  name: nova-credential-rotation
  namespace: openstack
spec:
  targetServiceUser: nova-service-user   # K-ORC User CR reference
  rotationType: applicationCredential
  schedule:
    intervalDays: 90                     # Rotate every 90 days
    preRotationDays: 7                   # Create new credential 7 days before expiry
  gracePeriodDays: 1                     # Overlap period with both credentials valid
status:
  lastRotation: "2025-12-01T00:00:00Z"
  nextRotation: "2026-03-01T00:00:00Z"
  currentCredentialRef:
    name: nova-app-credential-v3
  conditions:
    - type: Ready
      status: "True"
      reason: CredentialValid
    - type: RotationScheduled
      status: "True"

Rotation Lifecycle:

Day 0: New Application Credential created
       ├── K-ORC creates AppCred in Keystone
       ├── Written to K8s Secret → PushSecret → OpenBao
       └── ESO distributes to all consumers

Day 83: Pre-rotation (7 days before expiry)
       ├── New Application Credential created
       └── Old credential still valid

Day 90: Grace period ends
       └── Old Application Credential deleted from Keystone

For the full rotation flow and Go type definitions, see C5C3 Operator -- CredentialRotation. For the secret management architecture, see Secret Management.