CRDs

CobaltCore defines several Custom Resource Definitions (CRDs) for declarative management of hypervisors, evictions, migrations, storage arbiters, network status, Keystone identity resources (via K-ORC), control plane orchestration, secret aggregation, and credential rotation. The following CRDs form the central API interface of the system. For Cortex scheduling CRDs, see Cortex Scheduling.

CRD API Version Maturity: CRDs within the same API group may have different version levels reflecting their maturity. The Hypervisor and Eviction CRDs (hypervisor.c5c3.io/v1) have reached stable status after extensive production use, while the Migration CRD (hypervisor.c5c3.io/v1alpha1) is newer and its API surface is still evolving. New CRDs generally start at v1alpha1 and are promoted to v1beta1 then v1 via conversion webhooks as they stabilize.

Hypervisor CRD (`hypervisor.c5c3.io/v1`)

Represents a hypervisor node in the cluster. For the complete hypervisor state machine, see Hypervisor Lifecycle.

yaml

apiVersion: hypervisor.c5c3.io/v1
kind: Hypervisor
metadata:
  name: hypervisor-001
spec:
  version: "1.0.0"                    # Desired OS version
  reboot: false                       # Request reboot after upgrade
  evacuateOnReboot: true              # Evacuation before reboot
  lifecycleEnabled: true              # Lifecycle management enabled
  skipTests: false                    # Skip onboarding tests
  customTraits: []                    # Custom OpenStack traits
  aggregates: []                      # Host aggregates
  allowedProjects: []                 # Allowed OpenStack projects
  highAvailability: true              # HA handling enabled
  createCertManagerCertificate: false # Create Cert-Manager certificate
  installCertificate: true            # Certificate installation via agent
  maintenance: ""                     # manual|auto|ha|termination
status:
  libVirtVersion: "8.0.0"
  operatingSystem:
    version: "1.0.0"
    variantID: "garden-linux"
    kernelRelease: "6.1.0"
    hardwareVendor: "Dell"
    hardwareModel: "PowerEdge R750"
  capabilities:
    cpuArch: "x86_64"
    memory: "512Gi"
    cpus: "128"
  domainCapabilities:
    arch: "x86_64"
    hypervisorType: "kvm"             # "kvm" or "cloud-hypervisor"
    supportedDevices: ["video", "network"]
    supportedCpuModes: ["host-passthrough"]
    supportedFeatures: ["sev", "sgx"]
  instances:
    - id: "abc123"
      name: "vm-001"
      active: true
  numInstances: 5
  hypervisorId: "hv-001"
  serviceId: "svc-001"
  conditions:
    - type: Ready
      status: "True"
      reason: Ready
    - type: Onboarding
      status: "False"

Condition Types:

Type	Description
`Onboarding`	Node is undergoing initial configuration and OpenStack registration
`Offboarded`	Node has completed the offboarding process and is no longer active
`Ready`	All onboarding steps and tests passed; node can accept workloads
`Terminating`	Node is being terminated and resources are being cleaned up
`Tainted`	Node has been flagged with a problem preventing normal scheduling
`TraitsUpdated`	OpenStack traits have been synchronized from spec to Nova
`AggregatesUpdated`	Host aggregate assignments have been synchronized from spec to Nova

Maintenance Modes:

Mode	Description
`manual`	Manual maintenance mode by external user
`auto`	Automatic maintenance mode
`ha`	High availability maintenance mode
`termination`	Internal mode during termination

For maintenance mode details, see Hypervisor Lifecycle -- Maintenance Mode. For HA-triggered maintenance, see High Availability.

Eviction CRD (`hypervisor.c5c3.io/v1`)

Represents an eviction request for a hypervisor. For the full eviction flow, see Hypervisor Lifecycle -- Eviction Process.

yaml

apiVersion: hypervisor.c5c3.io/v1
kind: Eviction
metadata:
  name: eviction-001
spec:
  hypervisor: "hypervisor-001"    # Name of hypervisor to evacuate
  reason: "Planned maintenance"   # Reason for eviction
status:
  hypervisorServiceId: "svc-001"
  outstandingRamMb: 16384
  outstandingInstances:
    - "vm-001"
    - "vm-002"
  conditions:
    - type: Evicting
      status: "True"
      reason: Running

Eviction Condition Types:

Type	Description
`MigratingInstance`	Migration status of a server
`PreflightChecksSucceeded`	Preflight checks successful
`HypervisorReEnabled`	Hypervisor re-enabled
`HypervisorDisabled`	Hypervisor disabled
`Evicting`	Eviction status

Migration CRD (`hypervisor.c5c3.io/v1alpha1`)

Represents an ongoing VM migration with detailed metrics.

yaml

apiVersion: hypervisor.c5c3.io/v1alpha1
kind: Migration
metadata:
  name: migration-vm-001
spec: {}
status:
  origin: "hypervisor-001"
  destination: "hypervisor-002"
  type: "live"
  started: "2024-01-15T10:00:00Z"
  operation: "running"

  # Timing metrics
  timeElapsed: "45s"
  timeRemaining: "30s"
  setupTime: "5s"
  downtime: "0s"

  # Data transfer
  dataTotal: "32Gi"
  dataProcessed: "20Gi"
  dataRemaining: "12Gi"

  # Memory metrics
  memTotal: "16Gi"
  memProcessed: "10Gi"
  memRemaining: "6Gi"
  memBps: "1Gi"
  memDirtyRate: "100Mi"
  memIteration: 3
  memPageSize: "4Ki"
  memNormal: 2621440
  memConstant: 1048576
  memPostcopyRequests: 0

  # Disk metrics
  diskTotal: "16Gi"
  diskProcessed: "10Gi"
  diskRemaining: "6Gi"
  diskBps: "500Mi"

  # Additional information
  autoConvergeThrottle: "0"
  errMsg: ""

RemoteCluster CRD (`ceph.c5c3.io/v1alpha1`)

Defines access to a remote Kubernetes cluster for external arbiter deployment. For the storage cluster integration, see Storage Architecture.

yaml

apiVersion: ceph.c5c3.io/v1alpha1
kind: RemoteCluster
metadata:
  name: arbiter-site
spec:
  # Namespace in remote cluster for arbiter deployment
  namespace: external-arbiter
  # Reference to secret with kubeconfig
  accesskeyRef:
    name: arbiter-kubeconfig
    key: "kubeconfig.yaml"
  # Interval for health checks
  checkInterval: 1m
  # Request timeout for remote client
  timeout: 10s
status:
  state: Ready  # Init|Progressing|Error|Ready|Deleting
  message: "Cluster reachable and permissions verified"
  conditions:
    - type: SecretAvailable
      status: "True"
    - type: ConfigValid
      status: "True"
    - type: ClusterReachable
      status: "True"
    - type: HasEnoughPermissions
      status: "True"

RemoteArbiter CRD (`ceph.c5c3.io/v1alpha1`)

Defines a Ceph Monitor (arbiter) to be deployed in a RemoteCluster.

yaml

apiVersion: ceph.c5c3.io/v1alpha1
kind: RemoteArbiter
metadata:
  name: stretched-cluster-arbiter
spec:
  # Reference to RemoteCluster or inline spec
  remoteCluster:
    name: arbiter-site  # Name of RemoteCluster in same namespace
  # Reference to Rook-managed CephCluster
  cephCluster:
    name: my-cluster
    namespace: rook-ceph
  # Prefix for monitor ID (e.g., "ext-a", "ext-b")
  monIdPrefix: "ext-"
  # Interval for health checks
  checkInterval: 1m
  # Optional: Service configuration for arbiter exposure
  service:
    type: NodePort
    nodeIp: 10.10.0.1
  # Optional: Pod configuration
  deployment:
    nodeSelector:
      node-role: arbiter
status:
  state: Ready  # Init|Progressing|Error|Ready|Deleting
  monId: "ext-a"  # Reserved monitor ID
  message: "Arbiter deployed and joined quorum"
  conditions:
    - type: RemoteClusterReady
      status: "True"
    - type: CephClusterReady
      status: "True"
    - type: ArbiterDeploymentReady
      status: "True"

Use Case for Stretched Cluster:

text

┌─────────────────────┐     ┌─────────────────────┐     ┌─────────────────────┐
│    Datacenter A     │     │    Datacenter B     │     │   Arbiter Site      │
│    (Storage Cluster)│     │    (Storage Cluster)│     │   (Arbiter Cluster) │
│                     │     │                     │     │                     │
│  MON + OSDs         │     │  MON + OSDs         │     │  MON only           │
│  Ext. Arbiter Op ───┼─────┼─────────────────────┼─────┼─▶ (Tiebreaker)      │
│                     │     │                     │     │                     │
└─────────────────────┘     └─────────────────────┘     └─────────────────────┘

OVSNode CRD (`ovs.c5c3.io/v1alpha1`)

Represents the OVS status of a hypervisor node. Automatically created and updated by the OVS Agent. For the OVS bridge layout and OVN architecture, see Network Architecture.

yaml

apiVersion: ovs.c5c3.io/v1alpha1
kind: OVSNode
metadata:
  name: hypervisor-node-01
  namespace: ovn-system
spec:
  nodeRef:
    name: hypervisor-node-01
status:
  ovsVersion: "3.4.1"
  dpdkEnabled: true
  dpdkVersion: "23.11.1"
  bridges:
    - name: br-int
      ports: 156
      flows: 2847
      status: Active
    - name: br-ex
      ports: 2
      flows: 45
      status: Active
  interfaces:
    bonds:
      - name: bond0
        mode: balance-tcp
        status: Active
        members: 2
  ovnController:
    connected: true
    chassisId: "a1b2c3d4-..."
  conditions:
    - type: Ready
      status: "True"
    - type: OVSDBConnected
      status: "True"
    - type: OVNControllerConnected
      status: "True"

K-ORC Keystone CRDs (`openstack.k-orc.cloud/v1alpha1`)

K-ORC (Kubernetes OpenStack Resource Controller) provides CRDs for declarative management of Keystone resources. These CRDs are essential for the bootstrap process --- without them, OpenStack services cannot register in the service catalog or authenticate.

Common Fields:

All K-ORC CRDs share the following fields:

Field	Description
`spec.cloudCredentialsRef.cloudName`	Cloud name from `clouds.yaml`
`spec.cloudCredentialsRef.secretName`	Kubernetes Secret containing `clouds.yaml`
`spec.managementPolicy`	`managed` (full lifecycle) or `unmanaged` (read-only import)

Service CRD

Registers an OpenStack service in the Keystone service catalog.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Service
metadata:
  name: nova-service
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: nova
    type: compute
    description: "OpenStack Compute Service"

Endpoint CRD

Registers a service endpoint (public or internal) in the Keystone service catalog.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Endpoint
metadata:
  name: nova-public
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    serviceRef: nova-service
    interface: public
    url: "https://compute.example.com"
    region: RegionOne

---
apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Endpoint
metadata:
  name: nova-internal
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    serviceRef: nova-service
    interface: internal
    url: "http://nova-api.openstack.svc:8774"
    region: RegionOne

User CRD

Creates a real Keystone service user for a workload pod. In CobaltCore (issue #30) the c5c3-operator creates one User per pod (svc-<service>-<ordinal>), with the password supplied as an input via passwordRef — the operator generates the password; if passwordRef is omitted the user is created without a password. These are the credential workloads actually use; there are no per-service Application Credentials for workloads.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: User
metadata:
  name: svc-nova-api-0                 # one per pod / replica slot
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml      # the admin App Cred (K-ORC's only credential)
  managementPolicy: managed
  resource:
    name: svc-nova-api-0
    domainRef: default
    passwordRef:                       # password is an INPUT, generated by the c5c3-operator
      name: svc-nova-api-0-password
      key: password

The matching Role/grant CRs place every per-pod user in one stable service project (D5). On pod-identity deletion a finalizer makes K-ORC delete the Keystone user, revoking its tokens and grants. See Credential Lifecycle — Per-pod service users.

ApplicationCredential CRD

Creates the single admin Application Credential — K-ORC's only credential and the only App Cred anywhere in the design. It is project-scoped and restricted (cannot mint further app creds or trusts), bootstrapped from the admin password, and never rendered into a workload pod. A project-scoped admin App Cred carrying role:admin satisfies every K-ORC operation — including catalog (service/endpoint) writes — even under enforce_scope=True (verified; see the admin App Cred section).

Self-service invariant. Keystone only lets a user create an Application Credential for themselves — POST /v3/users/{user_id}/application_credentials is hard-rejected unless the calling token's user is {user_id} (ForbiddenAction; the create_application_credential policy is RULE_OWNER, not an admin rule, so there is no admin override). resource.userRef must therefore be the same user that cloudCredentialsRef authenticates as — here both are admin. The credential is additionally project-scoped and may carry only roles the user already holds in that project (no privilege escalation, no system scope). The same rule constrains the brownfield Credential Bridge.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: ApplicationCredential
metadata:
  name: admin-app-credential
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: admin-bootstrap-clouds-yaml   # admin password, used only to mint/rotate
  managementPolicy: managed
  resource:
    name: k-orc-admin
    userRef: admin                            # the bootstrap admin user
    unrestricted: false                       # restricted (default): no further app creds/trusts
    roles:
      - admin
    accessRules: []                           # optional: narrow to identity/catalog endpoints
    secretRef:
      name: k-orc-app-credential              # → PushSecret → OpenBao → ESO → orc-system

Rotation is restricted + password-driven re-mint via the CredentialRotation CRD (the admin App Cred is its only target). Workload service users rotate by pod recreation, not via this CRD.

Domain CRD

Manages Keystone identity domains.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Domain
metadata:
  name: default-domain
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: Default
    description: "Default domain"
    enabled: true

Project CRD

Manages Keystone projects within a domain.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Project
metadata:
  name: service-project
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: service
    domainRef: default-domain
    description: "Service project for OpenStack services"
    enabled: true
    tags: ["infrastructure", "service"]

Role CRD

Manages RBAC roles in Keystone.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Role
metadata:
  name: admin-role
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: admin
    description: "Admin role"

Group CRD

Manages user groups in Keystone.

yaml

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Group
metadata:
  name: service-admins
  namespace: openstack
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: k-orc-clouds-yaml
  managementPolicy: managed
  resource:
    name: service-admins
    domainRef: default-domain
    description: "Group for service administrators"

ControlPlane CRD (`c5c3.io/v1alpha1`)

The ControlPlane CRD is the top-level API for an entire OpenStack deployment. Users or GitOps apply a single CR, and the c5c3-operator handles everything downstream --- infrastructure provisioning, service CR creation, and phased rollout.

yaml

apiVersion: c5c3.io/v1alpha1
kind: ControlPlane
metadata:
  name: production
  namespace: openstack
spec:
  openStackRelease: "2025.2"         # Target OpenStack release
  region: RegionOne
  infrastructure:
    database:
      replicas: 3                    # MariaDB Galera cluster size
      storageClass: fast-ssd
      storageSize: 50Gi
    messaging:
      replicas: 3                    # RabbitMQ cluster size
    cache:
      replicas: 3                    # Memcached replicas
  services:
    keystone:
      enabled: true
      replicas: 3
      fernet:
        maxActiveKeys: 3
        rotationInterval: 24h
    nova:
      enabled: true
      replicas:
        api: 3
        scheduler: 2
        conductor: 2
      serviceUser:                         # per-pod real Keystone user (issue #30, D3)
        project: service                   # stable service project shared by all pods (D5)
        roles: [service, admin]
        mode: PerReplica                   # PerReplica (default) | Ephemeral | None
        maxAge: 720h                        # rolling recreation for long-lived pods (D4)
      policyOverrides:                     # Per-service policy overrides
        rules:
          "compute:create": "role:member"
          "compute:delete": "role:admin"
    neutron:
      enabled: true
      replicas: 3
      serviceUser:
        project: service
        roles: [service, admin]
        mode: PerReplica
    glance:
      enabled: true
      replicas: 3
    cinder:
      enabled: true
      replicas:
        api: 3
        scheduler: 2
        volume: 2
    placement:
      enabled: true
      replicas: 3
  global:
    tls:
      enabled: true
      issuerRef:
        name: letsencrypt-prod
        kind: ClusterIssuer
    policyOverrides:                       # Global policy rules for all services
      rules:
        "admin_required": "role:admin"
  korc:
    cloudCredentialsRef:
      cloudName: openstack
      secretName: k-orc-clouds-yaml        # the admin App Cred (K-ORC's only credential)
    adminCredential:                       # K-ORC's single admin Application Credential
      passwordSecretRef:                   # admin password (root of trust), used only to mint/rotate
        name: keystone-admin-credentials
        key: password
      applicationCredential:
        restricted: true                   # cannot mint further app creds/trusts
        rotation:
          mode: PasswordDriven             # restricted + password-driven re-mint (D2)
          intervalDays: 90
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AllServicesReady
    - type: InfrastructureReady
      status: "True"
    - type: KeystoneReady
      status: "True"
  updatePhase: Complete              # Idle|Validating|UpdatingInfra|UpdatingKeystone|UpdatingServices|Verifying|Complete|RollingBack
  services:
    keystone:
      ready: true
      version: "28.0.0"
    nova:
      ready: true
      version: "32.1.0"

Update Phases:

Phase	Description
`Idle`	No update in progress
`Validating`	Pre-flight checks (CRD schema, dependency resolution)
`UpdatingInfra`	Infrastructure CRs being updated (MariaDB, RabbitMQ, Memcached)
`UpdatingKeystone`	Keystone CR updated and rolling out
`UpdatingServices`	Remaining service CRs updated and rolling out
`Verifying`	Post-update health checks
`Complete`	Update finished successfully
`RollingBack`	Reverting to previous state due to failure

For the full orchestration flow and Go type definitions, see C5C3 Operator.

SecretAggregate CRD (`c5c3.io/v1alpha1`)

Merges multiple Kubernetes Secrets into a single aggregated Secret. This is useful when a service needs credentials from multiple sources in a single mount.

yaml

apiVersion: c5c3.io/v1alpha1
kind: SecretAggregate
metadata:
  name: nova-all-credentials
  namespace: openstack
spec:
  sources:
    - secretRef:
        name: nova-db-credentials
    - secretRef:
        name: nova-rabbitmq-credentials
    - secretRef:
        name: ceph-client-nova
      keys:                              # Select specific keys (empty = all)
        - key
  target:
    name: nova-aggregated-credentials
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AllSourcesAvailable
    - type: SourcesResolved
      status: "True"

The c5c3-operator watches source Secrets and re-aggregates whenever any source changes. For the Go type definitions, see C5C3 Operator -- SecretAggregate.

CredentialRotation CRD (`c5c3.io/v1alpha1`)

Automates rotation of the single admin Application Credential — K-ORC's only credential. It is not used for workload service users: those rotate by pod recreation (issue #30, D4), not by an in-place credential swap. The only supported target is adminApplicationCredential.

yaml

apiVersion: c5c3.io/v1alpha1
kind: CredentialRotation
metadata:
  name: admin-app-credential-rotation
  namespace: openstack
spec:
  target: adminApplicationCredential     # the only supported target
  schedule:
    intervalDays: 90                     # Rotate every 90 days
    preRotationDays: 7                   # Mint successor 7 days before expiry
  gracePeriodDays: 1                     # Overlap period with both credentials valid
status:
  lastRotation: "2026-03-01T00:00:00Z"
  nextRotation: "2026-05-23T00:00:00Z"
  currentCredential:
    id: "abc123..."
    restricted: true
  conditions:
    - type: Ready
      status: "True"
      reason: CredentialValid
    - type: RotationScheduled
      status: "True"

Rotation Lifecycle (restricted + password-driven re-mint, D2):

text

Day 0:  Restricted admin App Cred active (minted from the admin password)

Day 83: Pre-rotation (7 days before expiry)
       ├── operator re-mints a fresh RESTRICTED admin App Cred from the admin password
       ├── written to K8s Secret → PushSecret → OpenBao → ESO → k-orc-clouds-yaml
       └── old App Cred still valid (grace window)

Day 84: Grace period ends
       └── old admin App Cred deleted from Keystone

For the full rotation flow and Go type definitions, see C5C3 Operator -- CredentialRotation. For the secret management architecture, see Secret Management.

CRDs ​

Hypervisor CRD (hypervisor.c5c3.io/v1) ​

Eviction CRD (hypervisor.c5c3.io/v1) ​

Migration CRD (hypervisor.c5c3.io/v1alpha1) ​

RemoteCluster CRD (ceph.c5c3.io/v1alpha1) ​

RemoteArbiter CRD (ceph.c5c3.io/v1alpha1) ​

OVSNode CRD (ovs.c5c3.io/v1alpha1) ​

K-ORC Keystone CRDs (openstack.k-orc.cloud/v1alpha1) ​

Service CRD ​

Endpoint CRD ​

User CRD ​

ApplicationCredential CRD ​

Domain CRD ​

Project CRD ​

Role CRD ​

Group CRD ​

ControlPlane CRD (c5c3.io/v1alpha1) ​

SecretAggregate CRD (c5c3.io/v1alpha1) ​

CredentialRotation CRD (c5c3.io/v1alpha1) ​

CRDs

Hypervisor CRD (`hypervisor.c5c3.io/v1`)

Eviction CRD (`hypervisor.c5c3.io/v1`)

Migration CRD (`hypervisor.c5c3.io/v1alpha1`)

RemoteCluster CRD (`ceph.c5c3.io/v1alpha1`)

RemoteArbiter CRD (`ceph.c5c3.io/v1alpha1`)

OVSNode CRD (`ovs.c5c3.io/v1alpha1`)

K-ORC Keystone CRDs (`openstack.k-orc.cloud/v1alpha1`)

Service CRD

Endpoint CRD

User CRD

ApplicationCredential CRD

Domain CRD

Project CRD

Role CRD

Group CRD

ControlPlane CRD (`c5c3.io/v1alpha1`)

SecretAggregate CRD (`c5c3.io/v1alpha1`)

CredentialRotation CRD (`c5c3.io/v1alpha1`)