NovaCompute CRD

novacomputes.nova.openstack.c5c3.io/v1alpha1, kind NovaCompute. The CRD is generated from operators/nova/api/v1alpha1/novacompute_types.go; the defaulting and validating webhooks live in operators/nova/api/v1alpha1/novacompute_webhook.go.

One CR runs nova-compute on one node pool of a compute cluster. It names a Nova in its own namespace, selects the pool's nodes with a required spec.nodeSelector, and projects a DaemonSet running ghcr.io/c5c3/nova-compute onto them. Each node registers a compute service under its Node name and joins the Nova's single cell. Everything below the selector (the libvirt settings, the rollout pace, the overrides) applies to every node the selector matches, so nodes with different settings get a CR of their own.

The CR also looks after what the node's life in Nova needs. It keeps the host aggregates openstack-hypervisor-operator onboards a hypervisor into and the custom Placement traits the pool declares for its hosts, and a node that leaves the pool is drained before its pod goes: its compute service is disabled, the pod stays until Nova reports no instance on the host, and the compute service is deleted after the pod is gone.

kubectl get novacomputes prints Ready (.status.conditions[?(@.type=='Ready')].status), Desired (.status.desiredNumberScheduled), Ready pods (.status.numberReady), and Age.

Spec

Field	Type	Required	Default	Description
novaRef	NovaRef	yes	none	The Nova this pool joins. The pods mount its compute contract and its API registers their services. Immutable, enforced by a CEL transition rule and by the webhook
nodeSelector	map[string]string (MinProperties=1)	yes	none	The nodes of the pool. At least one label is required: an empty selector matches every node in the cluster. It may change, and a node that stops matching is drained (see The drain)
tolerations	[]corev1.Toleration	no	none	Lets the pods run on tainted nodes. A toleration of kvm.cloud.sap/offboarding:NoExecute without tolerationSeconds is rejected (see Webhook rules)
image	*commonv1.ImageSpec	no	ghcr.io/c5c3/nova-compute:<Nova status.installedRelease>	The nova-compute image. When nil the tag is the release the referenced Nova has installed, not the one it is moving to: the control plane upgrades first, and the pool follows once the schemas have moved. An unset image names no pullPolicy of its own, so its containers take the operator default (--default-image-pull-policy) or the rule: Always for the tag (see pull policy resolution)
libvirt	NovaComputeLibvirtSpec	no	{}	The [libvirt] options the pool renders
updateStrategy	NovaComputeUpdateStrategy	no	{}	Paces the DaemonSet rollout
resources	*corev1.ResourceRequirements	no	150m CPU request; 768Mi memory request and limit	Requests and limits of the nova-compute container, applied to both init containers too. The operator resolves them per resource when it renders the pod and never writes them into the field: a CPU the block names neither as request nor as limit gets a 150m request and no limit, and a memory it names neither way gets 768Mi as both request and limit. A resource the block names is used as written, and anything else it sets is kept; see Resources. Before this default a nil block rendered none and a set block was used whole, so a pool whose block names no CPU or no memory changes its pod template on the operator upgrade
verticalAutoscaling	*VerticalAutoscalingSpec	no	nil	Opts the pool DaemonSet ({name}-nova-compute) for as long as the operator renders it into a VerticalPodAutoscaler that controls the requests of its containers; see VerticalAutoscalingSpec. On a cluster without the VPA, VPAReady turns False with reason VPANotInstalled.
extraConfig	map[string]map[string]string	no	none	INI sections merged over the rendered compute-pool.conf. It is the per-pool override surface. The keys the pod takes from its environment or its mounts, and the keys that select the live-migration transport, are rejected at admission (see NovaComputeOwnedConfigKeys)
traits	[]string (set, at most 64, items ^CUSTOM_[A-Z0-9_]+$ and at most 255 characters)	no	none	The custom Placement traits every Pending and Active node of the pool carries. The operator creates a missing trait in Placement and writes each node's list into its Hypervisor, from where openstack-hypervisor-operator applies it; see The traits. Standard traits stay with the virt driver
hostTraits	[]NovaComputeHostTraits (list-map keyed by nodeName, at most 1000)	no	none	Adjusts the traits of single nodes. A node without an entry carries traits
targetClusterRef	*commonv1.TargetClusterRefSpec	no	nil (the local cluster)	The registered target cluster the DaemonSet and its ConfigMaps are created on. The CR, its status and its finalizers stay on the management cluster. Immutable. See Target Clusters

NovaRef

Field	Type	Required	Default	Description
name	string (MinLength=1)	yes	none	The Nova's name. The reference is namespace-local: the pods mount the compute contract Secret in the CR's namespace

NovaComputeLibvirtSpec

Field	Type	Required	Default	Description
virtType	string (Enum kvm, qemu)	no	kvm	[libvirt] virt_type. qemu runs guests without hardware acceleration
cpuMode	string (Enum host-model, host-passthrough, custom, none)	no	none	[libvirt] cpu_mode. When empty the key is not rendered and Nova's default applies
cpuModels	[]string (items ^[A-Za-z0-9_.-]+$)	no	none	[libvirt] cpu_models, rendered comma-joined. Required with cpuMode: custom and refused otherwise
imagesType	string (Enum default, qcow2, raw, flat)	no	none	[libvirt] images_type. When empty the key is not rendered. The rbd, lvm and ploop backends are not offered: rbd needs a Ceph credential contract the NovaCompute does not carry yet, and the image carries no lvm2

Changing the libvirt settings of a pool with servers

A change of virtType, cpuMode, cpuModels or imagesType renders a new {name}-config-<hash> ConfigMap, and the pod template names it. Under RollingUpdate the pool's pods restart, maxUnavailable nodes at a time. Under OnDelete a pod keeps the old file until it is deleted.

nova-compute reads compute-pool.conf when it starts. A server that runs keeps the CPU of its domain: the restart of nova-compute leaves the domain alone. A hard reboot (openstack server reboot --hard) builds the domain anew from the configuration of the nova-compute process on the server's node and replaces its live and its persistent definition. It applies the new CPU only once that node's pod has restarted. A hard reboot handled by the old pod keeps the old CPU and has to be repeated.

The pod has no probe, so it counts as ready as soon as its container starts. On the lab nova-compute took requests 24 to 28 seconds after its pod started, and a hard reboot sent in that window is lost. At start-up nova-compute clears the reboot's task state, Nova records the reboot action as Error (openstack server event list <server>), and the server stays ACTIVE with its old domain. openstack server reboot --hard --wait reports success all the same.

Before the hard reboot, wait for three things:

• ConfigReady is True and its observedGeneration equals metadata.generation: the operator has rendered the current spec, and the condition's message names the ConfigMap. status.observedGeneration does not show this. Every pass sets it, including one that stopped before the ConfigMap or the DaemonSet step.
• The pod on the server's node mounts that ConfigMap in its pool-config volume. DaemonSetReady, which step 2 below waits for, does not prove it: the operator reads the DaemonSet back from its cache right after it applies the template, and a read that still shows the previous rollout reports True for a moment. Otherwise DaemonSetReady stays False under OnDelete until the pods that run the old template are deleted, and kubectl rollout status answers rollout status is only available for RollingUpdate strategy type.
• The compute service of the server's node has reported since the pod's nova-compute container started. nova-compute reports to Nova once it takes requests, every report_interval (10 seconds by default). Read Updated At while the container runs, and wait until it has changed twice while the container's startedAt stays the same. The first change can come from something else: a report the old process sent before it died that nova-conductor applied late, a report the old process sent while it shut down after kubectl delete pod --force had the DaemonSet start the new pod, or another write to the service row such as openstack compute service set. Do not compare Updated At with startedAt: nova-conductor stamps Updated At with its own clock, startedAt comes from the node's, and a skew between them can make the old process's last report look later than the new container's start.

# 1. the operator has rendered the current spec, and the ConfigMap it names;
#    kubectl wait skips a condition whose observedGeneration is older than
#    metadata.generation
kubectl wait novacompute/<name> -n <namespace> --for=condition=ConfigReady --timeout=5m
kubectl get novacompute <name> -n <namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="ConfigReady")].message}{"\n"}'

# 2. every pod runs the current template (rollout status: RollingUpdate only)
kubectl rollout status daemonset/<name>-nova-compute -n <namespace> --timeout=15m
kubectl wait novacompute/<name> -n <namespace> --for=condition=DaemonSetReady --timeout=15m

# 3. the pod on the server's node: its name, the ConfigMap it mounts, the start
#    of its nova-compute container, and the CPU keys it read
kubectl get pod -n <namespace> -o name \
  -l app.kubernetes.io/instance=<name>,app.kubernetes.io/component=nova-compute \
  --field-selector spec.nodeName=<node>
kubectl get -n <namespace> <pod> \
  -o jsonpath='{.spec.volumes[?(@.name=="pool-config")].configMap.name}{"\n"}'
kubectl get -n <namespace> <pod> \
  -o jsonpath='{.status.containerStatuses[?(@.name=="nova-compute")].state.running.startedAt}{"\n"}'
kubectl exec -n <namespace> <pod> -c nova-compute -- \
  grep -E '^cpu_(mode|models)' /etc/nova/compute-pool.conf.d/compute-pool.conf

# 4. the last report of the node's compute service: read it while the container
#    runs, and again until it has changed twice
openstack compute service list --service nova-compute --host <node> -c 'Updated At' -f value

<node> is the server's host (openstack server show <server> -c OS-EXT-SRV-ATTR:host -f value). With spec.targetClusterRef set, the rollout status line and the four pod lines run against the target cluster, the others against the management cluster.

The lab moved its pool to cpuMode: custom because live migration between its nodes needed a named model; Lab hypervisors records that and the reproduction behind this section.

NovaComputeUpdateStrategy

Field	Type	Required	Default	Description
type	string (Enum RollingUpdate, OnDelete)	no	RollingUpdate	The rollout mode. OnDelete hands the pace to whoever deletes the pods
maxUnavailable	*intstr.IntOrString	no	1 (operator-resolved)	How many nodes may restart nova-compute at once. RollingUpdate only

The set of nodes a pool holds is part of the pod template (see The pod), so every change of it rolls the pool's pods under RollingUpdate: a drain that starts, a node that is released, a conflict that appears. Running instances survive a restart, because their domains live in the node's libvirt, but a live migration in flight through a restarting nova-compute fails. A pool that must not restart nova-compute on its own takes type: OnDelete.

NovaComputeHostTraits

Field	Type	Required	Default	Description
nodeName	string (1 to 253 characters)	yes	none	The node, which is also the name of its Hypervisor. It may name a node the pool does not hold yet; TraitsApplied names such entries
traits	[]string (the bounds of spec.traits)	no	none	The node's custom traits. Required unless mode is Replace
mode	string (Enum Append, Replace)	no	Append	Append gives the node spec.traits and these traits. Replace gives it these traits alone, and an entry without traits clears the node

Defaulting and validation

The mutating webhook leaves the object untouched. Every default is a +kubebuilder:default the API server applies from the schema, or a value the operator resolves at reconcile time: the image, the maxUnavailable of 1, and the CPU request and the memory of the three containers (see Resources). The webhook stays registered so a default that has to be materialized later can be added without changing the deployed webhook configuration.

Schema-layer rules

These hold even when the webhook is down.

Message	Where it comes from
targetClusterRef is immutable	Two CEL transition rules on NovaComputeSpec, one for adding or removing the ref and one for renaming it
novaRef is immutable	A third transition rule. A pool moved to another Nova leaves its compute services registered with the first one
cpuModels is required when cpuMode is custom and must be empty otherwise	A CEL rule on NovaComputeLibvirtSpec
exactly one of image.tag or image.digest must be set	Inherited from commonv1.ImageSpec on spec.image
traits must not be empty unless mode is Replace	A CEL rule on NovaComputeHostTraits: an Append entry without traits would change nothing

Beside them the schema carries MinLength=1 on spec.novaRef.name, MinProperties=1 on spec.nodeSelector, the three spec.libvirt enums, the item pattern on spec.libvirt.cpuModels, and the RollingUpdate/OnDelete enum on spec.updateStrategy.type. spec.traits and every spec.hostTraits[].traits take at most 64 distinct items, each matching ^CUSTOM_[A-Z0-9_]+$ and at most 255 characters long, which is Placement's own schema for a custom trait, so admission refuses what Placement would. A second spec.hostTraits entry for one nodeName answers Duplicate value, and the list takes at most 1000 entries. None of these rules has a webhook twin.

Webhook rules

The validating webhook accumulates every violation into one admission response.

Message	Trigger
name must be at most 63 characters: it is the app.kubernetes.io/instance label value of every child	metadata.name longer than MaxNovaComputeNameLength. Create only: the name is immutable, and the webhook also sees the finalizer removal that completes a deletion
novaRef.name must be set (the Nova this node pool joins)	spec.novaRef.name is empty
novaRef is immutable	An update renames spec.novaRef.name
nodeSelector must carry at least one label	spec.nodeSelector is empty or absent
the messages of IsQualifiedName and IsValidLabelValue	A selector key or value the API server would refuse in a label selector, which would fail the Node list on every pass
tolerates kvm.cloud.sap/offboarding:NoExecute; openstack-hypervisor-operator deletes the compute service only after every agent pod on the node is gone, and a nova-compute that stays re-registers it	A toleration without tolerationSeconds that tolerates the taint, matched the way openstack-hypervisor-operator matches it. operator: Exists with an empty key tolerates every taint and is rejected too. A toleration with tolerationSeconds is admitted: the pod leaves on its own, and the operator counts it as evictable
maxUnavailable applies to RollingUpdate only	maxUnavailable together with type: OnDelete
maxUnavailable must be an integer or a percentage such as "25%"	A malformed value
maxUnavailable must resolve to at least 1 for RollingUpdate	A value that scales to zero against 100
cpuModels is required when cpuMode is custom and must be empty otherwise	The webhook twin of the CEL rule
<key> is managed via <source> and must not be set in extraConfig (<impact>)	A rejected key of the owned-key registry in spec.extraConfig
extraConfig ... must not contain a newline or carriage return	A control character in a section, key or value of spec.extraConfig
no such option in the nova <release> option catalog	An option the catalog of the referenced Nova's spec.openStackRelease does not carry
repository must be set, exactly one of image.tag or image.digest must be set	spec.image is set and incomplete
target cluster name must be set	spec.targetClusterRef is present with an empty name
targetClusterRef is immutable (...)	An update adds, drops or renames the ref

The option-catalog check needs the release of the Nova, so the webhook reads the Nova spec.novaRef names. A Nova that does not exist yet (a pool is commonly applied beside it) skips the check with the single warning extraConfig catalog check skipped: Nova <ns>/<name> not found. An empty spec.extraConfig reads nothing and warns nothing. On update the check runs again only when spec.extraConfig changed.

An update to a CR that is being deleted and leaves the spec unchanged is admitted without validation. That is the drain finalizer's removal, and an unchanged spec admitted earlier can fail today's rules, such as an owned key a later operator rejects; rejecting the removal would hold the CR in Terminating with nothing left to edit.

Owned keys

NovaComputeOwnedConfigKeys in operators/nova/api/v1alpha1/config_ownership.go lists the keys of compute-pool.conf the operator owns.

Section	Key	Treatment	Source
DEFAULT	host	rejected	the node name, OS_DEFAULT__HOST from spec.nodeName
DEFAULT	my_ip	rejected	OS_DEFAULT__MY_IP from status.hostIP
DEFAULT	transport_url	rejected	OS_DEFAULT__TRANSPORT_URL from the compute contract
DEFAULT	state_path	rejected	the /var/lib/nova host mount
oslo_concurrency	lock_path	rejected	the /var/lib/nova host mount
libvirt	connection_uri	rejected	the /run/libvirt host mount
libvirt	live_migration_scheme	rejected	operator-computed
libvirt	live_migration_with_native_tls	rejected	operator-computed
libvirt	live_migration_uri	rejected	operator-computed
libvirt	live_migration_tunnelled	rejected	operator-computed
libvirt	live_migration_inbound_addr	rejected	OS_LIBVIRT__LIVE_MIGRATION_INBOUND_ADDR from status.hostIP
os_vif_ovs	ovsdb_connection	rejected	the /run/openvswitch host mount
vnc	server_proxyclient_address	rejected	OS_VNC__SERVER_PROXYCLIENT_ADDRESS from status.hostIP
keystone_authtoken, service_user, placement, neutron, cinder	password	rejected	OS_<SECTION>__PASSWORD from the compute contract
libvirt	virt_type, cpu_mode, cpu_models, images_type	reported	spec.libvirt.*

A reported key is honored and surfaced through the ExtraConfigHealthy condition and a Warning event. [DEFAULT] compute_driver and [vnc] server_listen are rendered as defaults and are not owned: the e2e suite runs the fake driver through spec.extraConfig, and the listen address is a per-pool choice. The default server_listen = $my_ip binds every instance's console to the node address the console proxy dials. The pod shares the node's network and Nova sets no console password, so 0.0.0.0 would open every console on every interface of the node.

Status

Field	Type	Description
conditions	[]metav1.Condition	List-map keyed by type; see Conditions
observedGeneration	int64	The .metadata.generation the controller last reconciled
installedImage	string	The image the running DaemonSet was projected from, recorded once every node runs a ready pod
desiredNumberScheduled	int32	How many nodes the DaemonSet should run on, mirrored from it
numberReady	int32	How many of those nodes run a ready pod
nodes	[]NovaComputeNodeStatus	The per-node ledger of the pool, list-map keyed by name
createdTraits	[]string	The custom traits this pool created in Placement (the create answered 201) and has not deleted yet; see The traits

NovaComputeNodeStatus

A node that stops being selected keeps its entry until its compute service is deleted, so the record of a drain outlives the selection.

Field	Type	Description
name	string	The node's name, which is also the host of its compute service
phase	string	One of the node phases
zone	string	The node's topology.kubernetes.io/zone label, read from the Node and not from Nova: the availability zone of the aggregate the pool ensures for it. Nova places the host in that zone only while the host is in an aggregate of the zone; AggregatesReady reports NodesOutsideZoneAggregate otherwise
serviceID	string	The UUID of the nova-compute service registered under the node name
serviceStatus	string	enabled or disabled, as Nova reports it
serviceState	string	up or down, as Nova reports it
disabledReason	string	The reason Nova records for a disabled service
instances	*int32	How many servers Nova counts on a Draining node
conflictsWith	string	The NovaCompute that holds a node in Conflict
traits	[]string	While the pool manages the node's traits: the CUSTOM_ traits openstack-hypervisor-operator last reported on the node's resource provider (the CUSTOM_ subset of its Hypervisor's status.traits), and the list the pool wrote into spec.customTraits for it to apply. No pool deletes a trait it records

Node phases

Phase	Meaning
Pending	Selected; no compute service is registered under the node name yet, or its host is not mapped into the cell yet
Active	Selected; the service is registered and the pool has seen its host mapped
Draining	The node left the pool. Its pod stays, its service is disabled, and Nova still counts instances on it
Releasing	No instance is left, or another pool took the node over. The pod is released, and the service is deleted once the pod is gone
Conflict	Selected, but another NovaCompute of the same Nova on the same cluster holds the node. No pod of this CR runs there

The figure draws the five phases with what triggers each change and who drives it. The rules below decide the cases the arrows leave out.

A Pending node with a registered service turns Active once Nova lists its host as a hypervisor, which Nova does only for a host with a host mapping. Until then the pool runs the host discovery (see Reaching Nova). An Active node is not checked again.

A node is held by a CR when it appears in that CR's status.nodes in any phase but Conflict. The rules, in order:

1. A node the selector matches and this CR holds stays; a Draining or Releasing entry goes back to Active (or Pending), and its service stays disabled. Only when an older pool that is not being deleted holds it Pending or Active too, which a pass that read the other pool's status before its write landed can leave behind, does this CR yield: Conflict, with the service left to the older pool. Held by another pool: Conflict. Neither holds it but an older pool selects it (by creationTimestamp, then name): Conflict. Otherwise this CR takes it as Pending.
2. A held node that is no longer selected, is gone from the cluster, or belongs to a CR being deleted: Releasing without a disable when a pool that is not being deleted selects it (a handover). Otherwise Draining, except that an entry already Releasing stays Releasing.
3. A Conflict entry that is no longer selected is dropped.

Pools of different Novas, or on different clusters, never conflict.

Conditions

Eight steps each own one condition type. The aggregate Ready is True only when all eight are. ExtraConfigHealthy reports on the overlay the owner writes and stays out of the aggregate. For the pipeline see Reconciler Architecture.

Type	Status	Reason	Meaning
NovaReady	True	NovaResolved	The Nova exists, has installed a release, has published its contract, and its service-user password is readable
NovaReady	False	NovaNotFound	No Nova of that name in the namespace
NovaReady	False	WaitingForInstalledRelease	The Nova has not installed a release yet
NovaReady	False	ComputeConfigNotPublished	The Nova has not published status.computeConfigSecretRef yet
NovaReady	False	TargetClusterUnavailable	The pool's target cluster, or the Nova's, does not resolve
NovaReady	False	WaitingForServiceUserSecret	The Nova's service-user Secret, or its key, is absent or empty
NovaReady	False	NovaError	Reading the Nova or its Secret failed
NodesReady	True	NodesResolved	The message counts the selected nodes and the ones leaving
NodesReady	False	NoMatchingNodes	Nothing is selected and nothing is held. Not a wait: the later steps still run
NodesReady	False	NodeConflict	A selected node is held by another pool; the message lists node (held by <pool>). A Warning event NodeConflict records each new conflict once
NodesReady	False	NodesForbidden	The Node list answered 403, which is the nova chart installed with rbac.namespaceScoped=true and a pool on the local cluster
NodesReady	False	NodeListError	Listing or reading Nodes failed otherwise
ConfigReady	True	ConfigRendered	compute-pool.conf is rendered into its ConfigMap; the message names it
ConfigReady	False	WaitingForComputeConfig	The contract Secret is not in the CR's namespace on the pool's cluster. The ControlPlane mirrors it for a ControlPlane-managed Nova; otherwise copy it (Connect a compute cluster)
ConfigReady	False	ComputeConfigIncomplete	nova-compute.conf, transport_url or password is missing or empty; the message names them
ConfigReady	False	ConfigError	Reading the Secret, or writing or pruning the ConfigMaps, failed
DaemonSetReady	True	DaemonSetReady	Every node the DaemonSet selects runs a ready pod, or the pool holds no node and the DaemonSet was removed
DaemonSetReady	False	DaemonSetProgressing	A rollout is in flight. Not a wait: the aggregates and the drain still run
DaemonSetReady	False	DaemonSetError	The DaemonSet could not be applied, read or deleted
AggregatesReady	True	AggregatesEnsured	The aggregates of the pool's zones and tenant_filter_tests are in place
AggregatesReady	True	NodesOutsideZoneAggregate	The aggregates are in place, but an Active node is in no aggregate of its zone, so Nova does not place it in that availability zone. The message names each node and its zone, which is also the name of the pool's aggregate; past 20 nodes it names the first 20 and counts the rest. Reported after NodesWithoutZone and AggregateZoneMismatch. See The aggregates
AggregatesReady	False	NodesWithoutZone	A selected node carries no topology.kubernetes.io/zone; openstack-hypervisor-operator cannot onboard it. The other zones are still ensured
AggregatesReady	False	AggregateZoneMismatch	An aggregate of that name exists with another availability zone; the message names both
AggregatesReady	False	WaitingForAggregate	Another creator made the aggregate between the list and the create; the next pass reads it
AggregatesReady	False	ComputeAPIError	A Keystone or Nova call failed; the message names the call and the HTTP status. Retried after 30 seconds
AggregatesReady	False	NovaComputeListError	Listing the other NovaComputes of the Nova failed, on a pass that did not run the Nodes step
ServicesReady	True	ServicesUp	Every node's compute service is registered and up
ServicesReady	True	Draining	Nodes are Draining or Releasing; the message counts the instances per node
ServicesReady	False	WaitingForServices	A selected node has no compute service yet. Reported before WaitingForHostMapping
ServicesReady	False	WaitingForHostMapping	A selected node's service is registered, but Nova has not mapped its host into the cell yet. The message names the hosts and the state of the host discovery Job; the pool looks again after 10 seconds
ServicesReady	False	HostDiscoveryError	Reading the Nova API Deployment, or reading, creating or deleting the host discovery Job failed, or a Job of that name on a target cluster does not belong to the Nova. Logged and retried at the pool's poll, after 30 seconds at most
ServicesReady	False	ServicesDown	Nova reports an Active node's service down
ServicesReady	False	ComputeAPIError	A Keystone or Nova call failed, or a cell did not answer the service list. Retried after 30 seconds at most: a failed hypervisor list keeps a Releasing node's 10-second poll
ServicesReady	False	PodListError	Listing the pods on a Releasing node failed
TraitsReady	True	NoTraitsDeclared	No node desires a trait, no node records one and the pool holds no created trait. The step makes no call
TraitsReady	True	TraitsApplied	Every managed node's Hypervisor reports its desired list. The message counts the nodes, names the created traits that wait for their last carrier before deletion, and names the hostTraits entries that match no node the pool holds
TraitsReady	True	WaitingForHypervisorOperator	A Hypervisor carries the desired list in spec.customTraits, and openstack-hypervisor-operator has not applied it yet. Retried after 30 seconds. See The traits
TraitsReady	False	NodesWithCustomTraitsAnnotation	A managed node carries the annotation nova.openstack.cloud.sap/custom-traits, from which the hypervisor operator overwrites the pool's list. The pool leaves the node alone; past 20 nodes the message names the first 20 and counts the rest
TraitsReady	False	NodesWithoutHypervisor	A managed node has no Hypervisor. The other nodes are still written
TraitsReady	False	HypervisorLifecycleDisabled	A Hypervisor has spec.lifecycleEnabled: false, so the hypervisor operator applies no trait to it
TraitsReady	False	HypervisorTraitsFailed	A Hypervisor reports TraitsUpdated=False; the message carries the hypervisor operator's message of the first such node
TraitsReady	False	HypervisorOperatorNotInstalled	The pool's cluster serves no hypervisors.kvm.cloud.sap
TraitsReady	False	HypervisorsForbidden	Reading or patching a Hypervisor answered 403. A placed pool needs hypervisorTraits: true in the target-cluster-access chart; a local one gets the grant from the nova-operator chart's ClusterRole, which rbac.namespaceScoped=true does not install
TraitsReady	False	HypervisorError	Reading or patching a Hypervisor failed otherwise; returned as an error
TraitsReady	False	PlacementEndpointNotFound	spec.endpoints.placement.override of the Nova is empty, and the token's catalog lists no internal Placement endpoint in the Nova's spec.region
TraitsReady	False	PlacementAPIError	Creating or deleting a trait failed; the message names the trait. A failed create writes no Hypervisor in that pass
TraitsReady	False	ComputeAPIError	The Keystone token request failed
TraitsReady	False	NovaComputeListError	Listing the other NovaComputes of the Nova failed, on a pass that did not run the Nodes step
TraitsReady	False	TargetClusterUnavailable	The pool's target cluster does not resolve
Ready	True	AllReady	All eight sub-conditions are True
Ready	False	NotAllReady	At least one is not
VPAReady	True	VPAReady	The VerticalPodAutoscaler of every opted-in workload is applied; the message names them. See VerticalAutoscalingSpec
VPAReady	True	VPANotRequired	No workload opts in (spec.verticalAutoscaling unset); a VPA the CR created before is deleted
VPAReady	False	VPANotInstalled	A workload opts in, but the cluster the children land on does not serve autoscaling.k8s.io/v1 VerticalPodAutoscaler. Nothing is created, and the other conditions still converge
VPAReady	False	CapabilityProbeFailed	The target cluster the CR names could not be probed for the kind
VPAReady	False	VPAError	Listing, applying or deleting a VPA failed; the message carries the error

Sub-Resource Naming Convention

Resource	Name	Notes
DaemonSet	{name}-nova-compute	Labels app.kubernetes.io/name: novacompute, app.kubernetes.io/instance: {name}, app.kubernetes.io/component: nova-compute
ConfigMap	{name}-config-<hash>	The immutable compute-pool.conf; the three newest before the mounted one are kept

The compute contract Secret is not a child: the Nova publishes it, or the ControlPlane mirrors it. Neither is the host discovery Job {nova}-discover-hosts, which belongs to the Nova (see Reaching Nova). On a target cluster the children carry the ownership labels instead of an owner reference, and the teardown sweeps them.

Node contract

The figure shows one node that all three node-level resources select: the four pods, the host paths they share, and the order their init containers enforce. The sections below cover the part a NovaCompute owns.

The pod

The DaemonSet has no node selector. The node set is a required node affinity: one term selects the pool (a key In [value] per selector pair, minus a metadata.name NotIn per node in Conflict), and one term metadata.name In [node] per Draining node keeps its pod while its instances leave. A deleting pool that holds no Draining node has no term, and its DaemonSet is deleted.

The pod runs with hostNetwork: true and the seccomp profile RuntimeDefault. nova-compute runs privileged as uid 0 with nova-compute --config-file /etc/nova/compute-config/nova-compute.conf --config-dir /etc/nova/compute-pool.conf.d. Root is what a stock host needs: the libvirt socket is root:libvirt 0660 with a group ID that differs from host to host, and a hostPath directory comes up root:root. There is no probe; the service state Nova reports is the health signal.

Volume	Mount path	Details
the contract Secret	/etc/nova/compute-config	read-only; the fragment's ssl_ca_file names ca.crt here
the pool ConfigMap	/etc/nova/compute-pool.conf.d	read-only
hostPath /run/libvirt	same	DirectoryOrCreate; the libvirt socket
hostPath /var/lib/nova	same	DirectoryOrCreate, mountPropagation: Bidirectional, so the NFS exports nova mounts below it (in /var/lib/nova/mnt) reach the host's QEMU. The node's compute_id lives here, so a restarted pod keeps its identity. The create-instances-dir init container mounts it as well, without propagation
hostPath /run/openvswitch	same	DirectoryOrCreate; the node's Open vSwitch database
hostPath /dev	same
hostPath /sys/fs/cgroup	same	read-only
hostPath /lib/modules	same	read-only, for os-brick's modprobe
hostPath /etc/iscsi, /etc/nvme, /etc/multipath	same	DirectoryOrCreate
hostPath /etc/multipath.conf	same	FileOrCreate
emptyDir	/tmp

A Cinder NFS volume needs more of the node than the pod brings. The kernel modules nfs and nfsv4 have to be loaded, because os-brick mounts the share from inside the pod through the node's kernel. The mount has to reach wherever libvirtd runs: a libvirtd on the host sees it through the Bidirectional propagation above, and a libvirtd in a pod needs the same propagation on its own /var/lib/nova mount. libvirt also has to leave the volume file's owner alone (dynamic_ownership = 0 in qemu.conf): Cinder's NFS backends keep their files 42424:42424 with mode 660 (see Rendered backend section), and a file handed to QEMU's user shuts Cinder out of the volume. On the metal-stack lab, whose Cinder runs on Ceph and takes an NFS backend only in Part 3 of its quick start, the Lab NFS stack of WITH_NFS=true loads the modules, and the Lab hypervisors carry the propagation and the setting.

The wait-for-chassis init container runs the image's python3 under the restricted profile and waits until the node's OVN chassis has written external_ids:system-id into the local Open vSwitch database. It speaks the OVSDB JSON-RPC protocol itself, because the image ships no ovsdb-client, so a pool's nodes need an OVNChassis.

The create-instances-dir init container runs first, as uid 0 and gid 0. It is not privileged, and every capability is dropped but DAC_OVERRIDE. It runs mkdir -p -m 0755 /var/lib/nova/instances, nova's default instances_path: the host mount hides the directory the image ships, and the kubelet creates /var/lib/nova and nothing below it. An existing directory keeps its owner and mode. When mkdir fails, the pod stays in its init phase, nova-compute does not start, and the pool reports DaemonSetReady=False with reason DaemonSetProgressing. A file at the path fails it with File exists, a read-only host filesystem with Read-only file system.

Resources

nova-compute and both init containers, create-instances-dir and wait-for-chassis, carry one block that the operator resolves from spec.resources when it renders the pod. The rule works per resource. A CPU the block names neither as request nor as limit gets a 150m request and no limit. A memory it names neither way gets 768Mi as both request and limit. A resource the block names is used as written, a zero quantity included, and anything else it sets (ephemeral storage, hugepages, claims) is kept. The default is never written into the CR. The init containers add nothing to the pod's footprint, because a pod's effective request is the larger of its largest init container and the sum of its containers.

The memory figure holds nova-compute, its privsep helpers and a live migration:

• A = 395 MiB, the largest anonymous memory of the nova-compute container on the two busy lab computes during the six-hour soak 20261007T180919Z of #1274 (see Lab soak run): nova-compute at 209 to 229 MB and three privsep-helper processes at 69 to 78 MB each. The lab VPA run of 2026-10-05 agrees. Its memory target of 455Mi is 396 MiB plus the recommender's 15 % (Recorded lab run).
• B = 120 MiB, an allowance for a live-migration burst. No burst was measured. 120 MiB is the headroom #1312 found left below the 512Mi limit a namespace LimitRange set on the lab.
• F = (A + B) × 1.25 = 644 MiB, rounded up to a multiple of 128 MiB, which gives 768Mi.

The page cache of the hostPath /var/lib/nova/instances, about 105 MiB on the soak, is charged to the container as well, and the kernel reclaims it before it OOM-kills the container. The soak's highest working set, 504 MiB, read 98 % of 512Mi and reads 66 % of 768Mi.

The CPU request is the lab VPA target under KVM, 143m (the run's 90th percentile plus 15 %), rounded up to a multiple of 50m. The block sets no CPU limit, so a burst above the request is throttled only by a CPU limit a namespace LimitRange fills in.

A namespace LimitRange still fills in a default for a limit the block does not name. A memory the block names only as a request gets the LimitRange's default memory as its limit, and its default CPU becomes the CPU limit unless the block names one. A filled-in limit below the request rejects the pods. The LimitRange's max and min apply as well.

kubectl top reports the working set, which counts the page cache of /var/lib/nova/instances. The anonymous share is rssBytes of the nova-compute container in the kubelet's stats summary:

kubectl get --raw /api/v1/nodes/<node>/proxy/stats/summary \
  | jq '.pods[]
      | select(.podRef.namespace == "<namespace>"
          and (.podRef.name | startswith("<pool>-nova-compute-")))
      | .containers[] | select(.name == "nova-compute")
      | .memory | {workingSetBytes, rssBytes}'

A container at its limit shows OOMKilled as the last state of nova-compute. The kubelet restarts it, and the operations it was running on instances are interrupted. The remedy is a larger memory in spec.resources, set as request and as limit.

A LimitRange whose max memory lies below 768Mi, whose default CPU lies below the CPU request (150m by default) or whose default memory lies below a memory the block names only as a request rejects the pods, and so does a ResourceQuota without room: the DaemonSet records FailedCreate events, and the pool reports DaemonSetReady=False with reason DaemonSetProgressing and the message Waiting for the nova-compute DaemonSet: <ready> of <desired> nodes run a ready pod. A node without 768Mi of unrequested allocatable memory keeps its pod Pending, with a FailedScheduling event naming Insufficient memory.

Upgrading the operator changes the pod template of every pool whose block names no CPU or no memory. A RollingUpdate pool replaces its pods maxUnavailable nodes at a time (default 1), and an OnDelete pool replaces each pod when it is deleted. A pool that names both CPU and memory keeps its template.

With spec.verticalAutoscaling set, the VPA controls the requests only, and the 768Mi limit caps its memory recommendation.

The namespace

A pod carrying the privileged profile needs a namespace that admits it. On a compute cluster that is a namespace in the target-cluster-access chart's privilegedNamespaces, the list the OVN chassis uses as well (see Target Clusters). A local pool runs in its namespace on the management cluster. Label that namespace with pod-security.kubernetes.io/enforce deliberately: one without the label admits the privileged pod, so whoever may create a NovaCompute there gets root on the nodes it selects (see Who may name a cluster). openstack-hypervisor-operator waits for the agent pods of its --agent-namespaces to leave a node it offboards, so the pool's namespace belongs in that list too, and no pod of the pool may tolerate kvm.cloud.sap/offboarding:NoExecute indefinitely.

Every node needs the topology.kubernetes.io/zone label. The pool creates the aggregate named after the zone. openstack-hypervisor-operator keeps the host in it after onboarding only when the Node carries the annotation nova.openstack.cloud.sap/aggregates (see The aggregates).

Reaching Nova

The operator calls Keystone and the Nova API itself: it lists and disables the compute services, reads the hypervisor list, counts the servers on a host and keeps the aggregates. It authenticates as the Nova's spec.serviceUser at microversion 2.53, and lists the services at 2.69: below it Nova leaves out a cell that does not answer, and the servers list skips that cell as well, so its draining hosts would read as empty. A service Nova reports UNKNOWN fails the pass with ComputeAPIError instead. The disable, the delete, the aggregates and the host filter of the server count are admin-only under Nova's default policies, so that user needs the admin role. The ControlPlane grants it; a standalone Nova whose service user lacks it reports ComputeAPIError with HTTP 403. The operator reaches the local Nova at its Service URL and a placed one through the target cluster's service proxy, so a NetworkPolicy in front of Keystone or Nova has to admit the nova-operator.

A pool that declares traits also calls Placement: PUT and DELETE /traits/{name} at Placement microversion 1.6, with the same token. The address is the one the Nova's own [placement] section resolves: the Nova's spec.endpoints.placement.override when set, else the internal endpoint the token's catalog lists for the placement service in the Nova's spec.region. For a placed Nova the call goes through the service proxy, which reaches only a <service>.<namespace>.svc host. The ControlPlane registers its catalog rows in that shape. Placement's placement:traits:update and placement:traits:delete policies admit the admin role the service user already has. A NetworkPolicy in front of Placement has to admit the nova-operator as well.

Nova 32.0.0 and 33.0.0 leave the compute node of a host without a host mapping out of GET /os-hypervisors/detail, so a registered host missing from that list is not mapped into its cell yet. The pool reads the list, page by page at microversion 2.88, whose entries carry no cpu_info or resource counters, only while a Pending node has a registered service. A failed read keeps those nodes Pending, and the pool's other nodes still move. For such a host it runs nova-manage cell_v2 discover_hosts --verbose in the Job <nova>-discover-hosts, which the operator creates in the Nova's namespace on the Nova's cluster as a child of the Nova: with an owner reference locally, with the Nova's ownership labels on a target cluster. Its pod is the one the archive CronJob runs nova-manage in, with the image and the config of the Nova API Deployment. That Deployment rolls to spec.image only once the Nova has migrated its schemas, so during an upgrade the Job runs the release the schemas are at. No Job is created while that Deployment mounts no config. Every pool of the Nova shares the Job. While a host stays unmapped, a finished Job is replaced 30 seconds after it ended, a failed one after the HostDiscoveryFailed event; the last Job is removed 300 seconds after it finished. The [scheduler] discover_hosts_in_cells_interval periodic of Host discovery still maps a compute without a pool.

The aggregates

For each zone of its Pending and Active nodes the pool ensures an aggregate named after the zone, carrying it as its availability zone, and, while it is not being deleted, the zone-less tenant_filter_tests. The pool never adds a host to an aggregate. openstack-hypervisor-operator (hvo) decides which hosts the aggregates hold. The statements about hvo are as read at a2baf3f, the commit images/openstack-hypervisor-operator/source.json pins:

1. While hvo onboards a host, it keeps the host in the zone's aggregate and tenant_filter_tests and in no other. This covers the Onboarding reasons Initial and Testing, and Handover until TraitsUpdated is True. The zone is the Node's topology.kubernetes.io/zone label. hvo creates neither aggregate. While one is missing, the Hypervisor's AggregatesUpdated condition reads False with reason Failed and the message aggregates not found: [<names>], and onboarding stays at Initial. skipTests skips only the smoke test and leaves this step in place.
2. From Handover with TraitsUpdated=True on, the host belongs to the aggregates in Hypervisor.spec.aggregates and to no other. hvo adds the host to the missing ones first. It then removes the host from every aggregate outside the list: tenant_filter_tests, the zone's unless the list names it, and any aggregate an admin added the host to by hand. hvo applies the list only when its names differ from status.aggregates.
3. hvo fills spec.aggregates from the Node. On every Node reconcile it copies the annotation nova.openstack.cloud.sap/aggregates, a comma-separated list whose blank entries it drops, into the field. Whenever the annotation is present, even with an empty value, hvo appends the zone label. Without the annotation the field keeps its value, which is [] on every Hypervisor hvo creates, so the host leaves both aggregates at Handover. The empty value nova.openstack.cloud.sap/aggregates= yields [<zone>]. A value that names the zone lists it twice, so the value names only the aggregates beyond the zone's. Every name in it must exist; a missing one fails the apply with aggregates not found, as in item 1.
4. Once a terminating Hypervisor is evicted (Evicting=False), the list is empty and the host leaves every aggregate. Offboarding waits until the host is in none.

The pool reports an Active node that is in no aggregate of its zone as AggregatesReady=True under NodesOutsideZoneAggregate, with the node and its zone in the message. It leaves Ready alone, because a pool may run without hvo. Membership in any aggregate that carries the zone counts, since Nova derives a host's zone from every aggregate the host is in. An openstack aggregate add host by hand lasts only until hvo next applies its list, so extra aggregates go into the annotation's value.

An aggregate the pool creates carries the metadata c5c3.io:nova=<namespace>/<nova>; one it created but could not mark is deleted again and recreated on the next pass. One that exists already is used as it is. A marked aggregate that no pool of the Nova needs is deleted once no host is left in it, so the last pool of a Nova removes tenant_filter_tests on its way out. One that carries metadata beyond the marker and its availability zone, such as the filter_tenant_id of a tenant isolation, is kept instead, with a Warning event AggregateKept naming the keys: deleting it would drop them, and the aggregate the zone gets back would come without them. An unmarked aggregate is never deleted.

The traits

A pool declares custom Placement traits for its hosts. Nova then schedules a flavor asking for trait:CUSTOM_GPU=required onto them and keeps one asking for trait:CUSTOM_GPU=forbidden off them. Each Pending and Active node gets one desired list:

The node's hostTraits entry	Desired list
none	spec.traits
mode: Append (the default)	spec.traits and the entry's traits
mode: Replace	the entry's traits alone; without traits, none

The list is deduplicated and sorted ascending in byte order. An entry naming a node the pool does not hold changes nothing, and TraitsApplied names it, so a typo shows without failing the pool. Only CUSTOM_ traits are managed.

The pool never writes the traits of a resource provider itself. It writes each node's list into spec.customTraits of the node's Hypervisor (kvm.cloud.sap/v1, cluster-scoped, named after the Node), and openstack-hypervisor-operator (hvo) applies it. The statements about hvo are as read at a2baf3f:

1. hvo creates one Hypervisor per Node labelled nova.openstack.cloud.sap/virt-driver. Its traits controller acts only while spec.lifecycleEnabled is true, spec.maintenance is not termination, and onboarding has reached Handover or finished. A node that joins a pool carries the pool's traits from Handover on.
2. It compares spec.customTraits in order with the CUSTOM_ subset of status.traits. It adds the missing traits and removes the extra ones in one PUT /resource_providers/{uuid}/traits, writes the provider's sorted trait list to status.traits, and sets TraitsUpdated: True with reason Succeeded, or False with reason Failed and Placement's error. The pool writes the list in the order hvo stores, so an applied list compares equal and hvo calls Placement no more. A custom trait the list does not name is removed, which is why no second writer of provider traits can coexist with hvo.
3. hvo creates no trait, and Placement refuses an unknown one with a 400 that holds onboarding at Handover. The pool therefore creates every trait a node desires and does not carry yet before it writes the list, and writes no list in a pass whose create failed.
4. On every Node reconcile hvo overwrites spec.customTraits from the Node annotation nova.openstack.cloud.sap/custom-traits while the annotation exists, whatever its value. The pool neither reads nor writes the Hypervisor of a node carrying it and reports TraitsReady=False with reason NodesWithCustomTraitsAnnotation; removing the annotation hands the node to the pool. The pool never writes the annotation, since the target-cluster-access chart grants no patch on Nodes.

While hvo applies a list, TraitsReady stays True under WaitingForHypervisorOperator, as AggregatesReady does for the aggregate membership, so Ready does not flap while hvo onboards a node. The pool looks again every 30 seconds: nothing watches Hypervisors, and the Node watch misses annotation edits. Draining and Releasing nodes are not written. The drain empties the host, and deleting its compute service removes the resource provider with every trait on it.

The pool creates a trait with PUT /traits/{name} and records it in status.createdTraits when Placement answers 201, with a Normal event TraitCreated. A trait that existed already (204) is not recorded, and the pool never deletes it. A recorded trait is deleted (TraitDeleted) once no NovaCompute of the Nova that is not being deleted declares it in spec.traits or spec.hostTraits, and no NovaCompute of the Nova records it in status.nodes[].traits. A node's record lists the CUSTOM_ traits of its Hypervisor's status.traits and the list the pool wrote, so a trait waits while hvo has yet to apply or remove it. When Placement answers 409, a resource provider no pool of the Nova manages carries the trait: the pool drops it from createdTraits with the event TraitKept and leaves it in Placement.

A deleting pool deletes the traits it created before its drain finalizer goes, so a Placement call that keeps failing holds the deletion. A trait another pool of the Nova still declares is dropped with TraitKept instead and stays in Placement, also after that pool drops it, because only its creator records it. Such a trait matches no host, and openstack trait delete removes it.

A placed pool patches the Hypervisors on its compute cluster, which needs hypervisorTraits: true in the target-cluster-access chart (see Target Clusters). Without it the pool reports HypervisorsForbidden. A local pool gets get and patch on hypervisors.kvm.cloud.sap from the nova-operator chart's ClusterRole. A pool on a cluster without hvo, such as the kind clusters of CI, reports HypervisorOperatorNotInstalled once it declares a trait.

Before a downgrade to an operator release without these fields, remove traits and hostTraits from every pool and wait for TraitsReady=True/NoTraitsDeclared. The older operator does not know the fields, and once the CRD is rolled back they are pruned the next time each CR is written. The last list stays on each Hypervisor, where hvo keeps applying it.

Live migration

Every pool live-migrates over libvirt TLS, and QEMU encrypts the guest's memory and disk stream as well. compute-pool.conf carries [libvirt] live_migration_scheme = tls and live_migration_with_native_tls = true whatever spec.libvirt holds. The pod sets [libvirt] live_migration_inbound_addr to the node's address through OS_LIBVIRT__LIVE_MIGRATION_INBOUND_ADDR, from status.hostIP. The destination hands that address to the source, which dials qemu+tls://<node IP>/system and sends the QEMU stream to the same address, so no host has to resolve another host's Node name.

spec.extraConfig cannot set live_migration_scheme, live_migration_with_native_tls, live_migration_uri, live_migration_tunnelled or live_migration_inbound_addr (see Owned keys). The other live_migration_* options (bandwidth, downtime, completion timeout, post-copy, auto-converge, and live_migration_parallel_connections on 2026.1) stay open to it and are checked against the option catalog.

The pod mounts no certificate. Nova migrates peer to peer: nova-compute talks only to its local libvirtd over /run/libvirt, and that libvirtd opens the connection to the destination with the host's own client certificate.

The host image and its owner provide what each host owes, and the pool configures none of it:

• libvirtd listening for TLS on port 16514.
• The CA, the certificate and the key at libvirt's default paths, which kvm-node-agent writes from the Secret tls-libvirt-<node>: the CA at /etc/pki/CA/cacert.pem and /etc/pki/qemu/ca-cert.pem, the certificate at /etc/pki/libvirt/servercert.pem and /etc/pki/qemu/server-cert.pem, and the key at /etc/pki/libvirt/private/serverkey.pem and /etc/pki/qemu/server-key.pem. The client certificate and key in both directories link to the server ones. The kvm-node-agent image of this repository writes the keys with mode 0600. QEMU opens its key itself, so a host whose QEMU does not run as root sets PKI_KEY_GROUP on the agent to the numeric group QEMU runs in, which gives /etc/pki/qemu/server-key.pem that group and mode 0640. The agent container then needs the CHOWN capability or that group among its groups; without either, the agent fails every update and keeps the previous files. All key files hold the same key, so that group can read the key the node authenticates with to the libvirtd of its peers as well.
• TCP 16514 and QEMU's migration ports, 49152 to 49215 by default, open between every pair of nodes that can migrate to each other and closed to every other source.

Two controllers issue that certificate, and they compete. openstack-hypervisor-operator creates the cert-manager Certificate libvirt-<node>, with the Secret tls-libvirt-<node>, for every Node labelled nova.openstack.cloud.sap/virt-driver. kvm-node-agent creates a Certificate of the same name in its own namespace while the node's Hypervisor.spec.createCertManagerCertificate is true, and it installs any Secret named tls-libvirt-<node>, in any namespace. With both in one namespace, the two controllers overwrite each other's spec on every pass and cert-manager reissues the certificate each time. With two namespaces, the agent installs whichever Secret changed last, and the host alternates between two CAs.

That Secret is the host's trust root for migration. Whoever can write a Secret named tls-libvirt-<node>, in any namespace of the compute cluster, picks the CA the node's libvirtd trusts, and a client certificate from that CA gets full access to it: root on the hypervisor. As long as kvm-node-agent installs these Secrets from every namespace, the deployment confines them to the certificate namespace, for example with a ValidatingAdmissionPolicy that denies a tls-libvirt-* Secret anywhere else.

A compute cluster runs with openstack-hypervisor-operator's certificate. It lives 47 days, carries the node's InternalIP and ExternalIP addresses as IP SANs, and comes from the Issuer that the operator's --certificate-issuer-name names in its --certificate-namespace. The deployment provides that Issuer. Hypervisor.spec.createCertManagerCertificate stays false, its default, on every Hypervisor. The operator issues its certificate for every hypervisor node unconditionally. The agent's certificate is opt-in, and upstream marks its 8-hour lifetime as a testing value. The pool reads no kvm.cloud.sap object and does not enforce this choice.

One CA spans a migration domain. Each host trusts only /etc/pki/CA/cacert.pem, the ca.crt of its own Secret, and Nova may pick any compute of the Nova as the destination, including one in another pool or on another compute cluster. Every node that can be a migration peer has to chain to the same CA and reach the others on the ports above. Node sets that cannot meet this are kept apart by availability zone.

The CA also decides who else gets in. libvirtd checks no client subject by default, so any client certificate that chains to /etc/pki/CA/cacert.pem gets full access to the node's libvirtd on port 16514. The Issuer behind --certificate-issuer-name therefore holds a CA dedicated to libvirt migration, which signs nothing else and is shared with no other Issuer. In --certificate-namespace, only openstack-hypervisor-operator and cert-manager create Certificates and CertificateRequests, because anyone who can create one there obtains such a client certificate. tls_allowed_dn_list in libvirtd.conf can narrow the accepted clients further, to the subject the node certificates carry (O=nova, CN=<node>).

Reading one of these Secrets grants the same access as writing one. The key in tls-libvirt-<node> is also the node's client key, so whoever can read one of these Secrets, or the CA Secret behind the Issuer, can connect to every libvirtd in the migration domain. tls_allowed_dn_list does not stop that, because the stolen certificate carries a node subject. In --certificate-namespace, only cert-manager, openstack-hypervisor-operator and kvm-node-agent read Secrets. kvm-node-agent reads them in every namespace, so a compromised node or agent token reaches every migration peer of that node. The CobaltCore operators read Secrets in every namespace the target-cluster-access chart's namespaces lists, so --certificate-namespace stays out of that list. A local pool cannot keep that boundary. It needs the nova chart installed cluster-scoped (see NodesForbidden under Conditions), and every operator chart left at its default rbac.namespaceScoped=false reads Secrets in every namespace of the management cluster, so each of these operators can read every node key and the CA there.

To check a node, read the TLSCertificateInstalled condition of its Hypervisor on the compute cluster. kvm-node-agent sets it once it has written the Secret into the host's PKI. Then move an ACTIVE server off the node:

kubectl get hypervisor <node> \
  -o jsonpath='{.status.conditions[?(@.type=="TLSCertificateInstalled")].status}'
openstack server migrate --live-migration <server>

The drain

Leaving the pool is the drain, and Drain a Compute Node walks it on the devstack and on a compute cluster under openstack-hypervisor-operator. The drain starts when a node stops matching the selector, when its Node is deleted, or for every node when the CR is deleted:

1. The node goes Draining. The pool disables its compute service once, with the reason c5c3.io: leaving NovaCompute <namespace>/<name>, and keeps its pod through the held affinity term.
2. The pool never migrates an instance. On a compute cluster, openstack-hypervisor-operator's Eviction, started through Hypervisor.spec.maintenance, empties the host; without it the owner does. The Eviction live-migrates an ACTIVE server, which needs Live migration. The hvo image this repository builds leaves the choice of block migration to Nova.
3. When Nova counts no server on the host, the node goes Releasing and its pod is released.
4. Once the pod is gone the pool deletes the compute service. Nova removes the host from every aggregate, deletes its resource providers and destroys its host mapping. A delete Nova refuses because instances came back returns the node to Draining.

On a compute cluster openstack-hypervisor-operator (hvo) empties the host before the pool label comes off. The figure draws that order, and its numbers are the steps below. The statements about hvo are as read at a2baf3f, the commit images/openstack-hypervisor-operator/source.json pins.

1. A person sets spec.maintenance: manual on the node's Hypervisor.
2. hvo disables the compute service, with the reason Hypervisor CRD: spec.maintenance=manual.
3. hvo creates an Eviction named after the node. It migrates every server away, and hvo sets status.evicted to true. Up to here clearing spec.maintenance makes hvo enable the service and delete the Eviction.
4. The person removes the pool label from the Node. The node goes Draining. From here on spec.maintenance stays set: the pool never enables a service, and hvo would enable it under a node that is leaving.
5. The pool counts the servers on the host and finds none. The service is disabled already, so the pool disables nothing and hvo's reason stays.
6. The node goes Releasing. The pool releases its pod and waits until the pod is gone, polling every 10 seconds.
7. The pool deletes the compute service. Nova drops the host mapping, the resource providers and the aggregate membership with it. This is the point of no return: a node labelled again afterwards starts as a new Pending entry and registers a new service.

Without hvo the order starts at step 4: the pool disables the service itself, with the reason c5c3.io: leaving NovaCompute <namespace>/<name>, and whoever owns the servers moves them.

The pool never enables a service. A node selected again mid-drain goes back to Active with its service still disabled, and so does a node another pool takes over (Releasing without a disable): the new pool finds the service as the old one left it. openstack compute service set --enable <host> nova-compute re-enables it by hand. A node deleted while it still holds instances stays Draining until the owner acts.

The CR carries the finalizer nova.openstack.c5c3.io/compute-drain until every node is released and the aggregates are cleaned up, so an unreachable Nova API holds a deletion. The escape is removing that finalizer, and only that one, by hand, which leaves the compute services and the marked aggregates in Nova:

i="$(kubectl get novacompute <name> -n <namespace> -o json \
  | jq '.metadata.finalizers | index("nova.openstack.c5c3.io/compute-drain")')"
kubectl patch novacompute <name> -n <namespace> --type=json -p "[
  {\"op\":\"test\",\"path\":\"/metadata/finalizers/$i\",\"value\":\"nova.openstack.c5c3.io/compute-drain\"},
  {\"op\":\"remove\",\"path\":\"/metadata/finalizers/$i\"}]"

A placed pool also carries openstack.c5c3.io/remote-children, which stays: with it the operator still deletes the pool's children on the target cluster and reaps the mirrors below before it releases the CR.

When the Nova is gone, or the target cluster was abandoned, the teardown skips the Nova side. The last pool of a Nova on a cluster also deletes two Secrets in its namespace, each only when it carries the label nova.openstack.c5c3.io/compute-config-mirror: "true", the mark of the ControlPlane's mirror: the compute contract <nova>-compute-config, and <nova>-hypervisor-operator-auth, the credentials the ControlPlane copies there for openstack-hypervisor-operator. Each deleted Secret gets a ComputeConfigMirrorReaped event of its own, and an absent one is skipped. A pool being deleted that still drains a node counts as one left: its draining pod mounts the contract. One whose drain finalizer was removed, or whose Nova is gone, drains nothing and does not count.

Example

apiVersion: nova.openstack.c5c3.io/v1alpha1
kind: NovaCompute
metadata:
  name: pool-a
  namespace: openstack
spec:
  novaRef:
    name: nova-pool
  nodeSelector:
    openstack.c5c3.io/nova-compute-pool: a
  traits:
    - CUSTOM_C5C3_E2E_POOL
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      memory: 768Mi
  extraConfig:
    DEFAULT:
      compute_driver: fake.FakeDriverWithoutFakeNodes
      vif_plugging_is_fatal: "false"
      vif_plugging_timeout: "0"

This is the pool the compute-node-pool suite applies (tests/e2e/nova/compute-node-pool/12-novacompute-pool-a.yaml), with the traits its traits step adds later against a stand-in Hypervisor CRD. The extraConfig switches to the fake driver, because kind has no KVM; a pool on real hypervisors leaves compute_driver at its default and sets spec.libvirt instead.