Skip to content

Sizing Calibration ​

A CobaltCore CR that names no resources gets its CPU and memory from render-time defaults, and a ControlPlane on the Minimal sizing profile gets fixed figures for its service pods and backing services. This page describes how those figures are measured with a VerticalPodAutoscaler (VPA) recommender in CI, how a script derives them from the measurement, and which run the figures in the code come from. The lab measurement reads the same recommendations on the metal-stack lab under servers on KVM hypervisors and derives no figure.

What is measured ​

A labelled CI run installs the VPA recommender on the kind cluster of three jobs and creates one VPA per Deployment, StatefulSet and DaemonSet in the openstack namespace. Every VPA has updateMode: "Off", so the recommender watches the pods and recommends requests without changing any pod. The run needs neither the VPA updater nor the admission controller, and the recommender overlay disables both. It lowers the recommender's per-pod floors from 25 millicores and 250 MiB to 1 millicore and 1 MiB, so a small container reports its own use and not its share of the floor.

The recommender accepts a VPA only when its target is the topmost well-known or scalable controller of its pods. The MariaDB CRD serves a scale subresource without a pod selector, so a VPA on the database StatefulSet is rejected as ConfigUnsupported, and a VPA on the MariaDB finds no pods. WITH_VPA=true therefore removes the scale subresource from the MariaDB CRD of the kind cluster, which makes the StatefulSet the topmost controller. Nothing in the kind stack scales a MariaDB through it.

A measured workload must not opt into verticalAutoscaling. The operator's VPA would be a second VPA on the same pods, which upstream leaves undefined, and a VPA in any other mode than "Off" changes the requests the measurement derives its figures from. None of the three jobs' ControlPlanes sets the block, and the built-in profiles set it nowhere.

The three jobs cover the process counts the profiles and the defaults run:

JobWorkloadsWhat it contributes
e2e-controlplaneThe Minimal ControlPlane of the full-chain suite, its MariaDB, Memcached and dedicated OpenBao, and the shared kind brokerService rows at one process, the Minimal CPU request, the backing-service figures, the budget projection
e2e-controlplane-ssoThe federated Minimal ControlPlaneThe federation-proxy sidecar
tempest (12 legs)Every Tempest-covered service at its default process count, and at four processes where the fixture raises it, under Tempest API loadService rows at two and four processes, the render-time CPU request

The derivation produces these figures:

KeyConstant
memoryBasememoryBase in internal/common/types/resources.go
defaultMemoryPerProcessdefaultMemoryPerProcess in internal/common/types/resources.go
glanceMemoryPerProcessglanceMemoryPerProcess in operators/glance/internal/controller/reconcile_deployment.go
defaultCPURequestdefaultCPURequest in internal/common/types/workload.go
minimalServiceCPURequestminimalServiceCPURequest in operators/c5c3/api/v1alpha1/sizing_profiles.go
minimalDatabase, minimalCache, minimalMessaging, minimalSecretStorethe memoryBound figures of the Minimal database, cache, messaging and secret store in sizing_profiles.go
sidecarCPURequest, sidecarMemorysidecarCPURequest and sidecarMemory in internal/common/types/resources.go

Some figures are not measured. Job and CronJob pods live for seconds and the recommender samples once a minute, so the Job memory stays pinned at 368Mi and the Job CPU follows defaultCPURequest. No measured run sets threads above one, so memoryPerExtraThread stays 32Mi. The OVN Raft memory floor of 256Mi stays, because the Raft working set grows with the logical port count, which a CI data set does not represent. The Glance cache-maintenance sidecar runs only in the Glance e2e-operator leg, which the label does not switch on, so the shared sidecar figure only rises. Neither the memory nor the CPU request of the Neutron metadata agent is taken from this run: none of its legs makes the agent provision a network, so its rows show an idle agent. Each figure has a measurement of its own (see Metadata agent memory and Metadata agent CPU). The operators' manager pods, the OVN chassis and NovaCompute DaemonSets and the platform stack keep their own figures. The chassis, NovaCompute and metadata agent readings under real servers come from the lab measurement, which sets them against those figures.

Running a measurement ​

The ci:measure-sizing label runs the measurement on a pull request. Neither ci:full nor a tag push implies it. CI Workflow lists the steps it adds to each job.

  1. Add the label to the pull request. A push to the pull request while the run is in flight cancels it.

  2. Wait until e2e-controlplane, e2e-controlplane-sso and the twelve tempest legs have finished. Each uploads a sizing-* artifact, also when a suite fails. A leg whose Collect the sizing measurement step failed recorded no recommendation; run it again:

    bash
    gh run rerun <run-id> --failed
  3. Download the artifacts:

    bash
    gh run download <run-id> --pattern 'sizing-*' --dir _output/sizing-run
  4. Copy the TOTAL row of the node budget table (Link 6z) from the log of the same run's e2e-controlplane job:

    bash
    gh run view <run-id> --job <job-id> --log | grep -E '[[:space:]]TOTAL[[:space:]]'
  5. Derive the figures, with the TOTAL row's CPU in millicores and memory in MiB:

    bash
    python3 hack/derive-sizing-figures.py --budget-total <cpu>m,<memory>Mi _output/sizing-run

The script prints one key=value line per figure, the projected node budget, and a Markdown section with every row it used. It exits 1 when the input is incomplete or the Minimal figures miss the node budget, and 2 on a usage or parse error.

Each artifact holds the snapshots of hack/ci-vpa-recommendations.sh, its watch.log, and the report it wrote. recommendations.tsv opens with a comment line naming the run, the attempt, the commit, the job, the Tempest leg and the recommender image, and has one row per workload container:

ColumnContent
namespace, kind, workload, replicasThe workload; replicas is - for a DaemonSet
owner_kind, owner_nameThe workload's controller owner, such as Keystone or MariaDB
app, componentThe pod template's app.kubernetes.io/name and app.kubernetes.io/component labels
containerThe container the recommendation is for
processes, threadsThe values after --processes and --threads (or --n-threads=) in the container's command and args, - when absent
cpu_target_m, memory_target_miThe recommender's target, in millicores and MiB, rounded up
cpu_upper_m, memory_upper_miThe recommender's upper bound
cpu_request_m, memory_request_mi, memory_limit_miWhat the container requested when the snapshot was taken, - when unset
snapshotThe time of the snapshot

Derivation rules ​

hack/derive-sizing-figures.py applies these rules in order. Rounding up to a step s means rounding up to the next multiple of s, and a job is the job= value of a file's comment line.

  1. Classification. A row is a formula row when its owner is a Keystone, Horizon, Placement, Barbican, Neutron, Cinder, Nova or NeutronMetadataAgent, or an OVNCentral with the component northd or sb-relay. A row owned by a Glance is a Glance row. The containers federation-proxy and cache-maintenance are sidecar rows, whatever their owner. A row owned by a MariaDB, Memcached, RabbitmqCluster or OpenBaoCluster is a backing row when it comes from e2e-controlplane; the Tempest legs' database and broker are ignored. Every other row is printed as ignored and used for nothing, except that rule 5 reads the OVN nb and sb rows. Two kinds of formula row have a fixed memory figure. A Cinder row with the component backup gets backupMemory (2Gi), because a backup's footprint follows its chunk size. A NeutronMetadataAgent row with the component metadata-agent gets metadataAgentMemory, because the agent's footprint follows the networks on its node (see Metadata agent memory). The CPU of such a row counts in rules 5 and 6 like any formula row's; its memory counts in neither rule 3 nor rule 9.

  2. Process and thread counts. A formula or Glance row takes its process count p and thread count t from its processes and threads columns. The Nova scheduler and conductor and an eventlet Glance take their worker count from configuration instead: one process in the two ControlPlane jobs, where Minimal sets one worker, and two in tempest, the operators' default. Every other row counts one process of one thread. Its memory target is normalized to T = target − p × (t − 1) × 32Mi.

  3. Shared memory formula. For every process count, M(p) is the largest T of the formula rows without a fixed memory figure. defaultMemoryPerProcess is the steepest slope between two measured process counts, at least 32Mi, rounded up to 16Mi. memoryBase is the largest M(p) − p × defaultMemoryPerProcess, at least 0, rounded up to 16Mi. Together they give memoryBase + p × defaultMemoryPerProcess ≥ M(p) at every measured process count. Rows at fewer than two process counts stop the derivation.

  4. Glance per-process figure. glanceMemoryPerProcess is the largest (T − memoryBase) / p of the Glance rows, at least 32Mi, rounded up to 16Mi.

  5. Render-time CPU request. From the formula and Glance rows of the tempest legs and their OVN nb and sb rows, the largest CPU target per owner_kind/component/container key counts once. defaultCPURequest is the median of those values, the upper of the two middle values for an even count, rounded up to 10m and at least 10m. That is what a VPA would request for a service under API load, and it is also the CPU half of the OVN Raft request floor.

  6. Minimal service CPU. The same median over the formula and Glance rows of the two ControlPlane jobs, without the NeutronMetadataAgent and OVNCentral rows the ControlPlane does not size, rounded up to 5m and at least 10m, is minimalServiceCPURequest.

  7. Minimal backing services. Per workload of a backing kind the main container is the one with the largest memory target; a sidecar beside it keeps its own figures. The CPU figure is the largest CPU target of the main containers, rounded up to 5m and at least 10m. The memory figure is their largest memory target rounded up to 16Mi, and never below a figure the service's own configuration depends on:

    Backing serviceMemory floorWhy
    Database (MariaDB)1GiIts working set grows with the connection count, which one CI run does not drive to the ceiling
    Cache (Memcached)96MiThe operator's floor of maxMemoryMB 64 plus 32Mi
    Messaging (RabbitmqCluster)1GiThe Cluster Operator alarms the broker at 0.6 of the limit minus a fifth, so its use reflects the limit it ran under. Its idle footprint is about 236 MiB: 512Mi (alarm at 245.8 MiB) alarmed without a backlog (#1298), and 1Gi (491.5 MiB) keeps the footprint below half the alarm threshold. In the full chain this is the shared kind broker
    Secret store (OpenBaoCluster)64Mi
  8. Sidecar. sidecarCPURequest is the largest federation-proxy CPU target rounded up to 5m and at least 25m; sidecarMemory is its largest memory target rounded up to 16Mi and at least 256Mi. Neither figure falls, because the Glance cache sidecar shares them and no measured run carries it.

  9. Budget projection. Starting from the Link 6z total, the script adds the change in requests, times the pod count, of the e2e-controlplane containers the Minimal figures reach: the service rows of rule 6 take minimalServiceCPURequest, and their memory follows the refitted formula when the formula rendered it (the limit equals the request and the memory is not fixed). The main containers of the database, cache and secret store take their rule 7 figures. Every other container keeps its request, including the shared kind broker and the compute leg Link 6z leaves out. While the CPU exceeds 4000m the script lowers minimalServiceCPURequest by 5m; at 10m still over, it exits 1. Memory above 16384Mi exits 1 as well, since a memory request equals its limit and never goes below a measured target. Either miss means the 4 vCPU / 16 GiB target is not reachable with measured figures, and the figures stay as they are until that is decided.

Recorded run ​

The figures in the code come from this run:

ItemValue
Run36339033005, attempt 1, on 2026-09-27
Measured commit4ebf79968dc3b9dbec072199f72ee6858f4a864e, the merge of pull request #1123 with main
Recommenderregistry.k8s.io/autoscaling/vpa-recommender:1.8.0
Link 6z total3480m CPU and 14754Mi of memory
Derivationpython3 hack/derive-sizing-figures.py --budget-total 3480m,14754Mi _output/sizing-run
Projectionprojected: 2885m 15474Mi of 4000m 16384Mi

The two keystone Tempest legs of the run failed on three RBAC tests (#1127); the suite ran to its end, so their recommendations are complete. An earlier run of the same pull request, 36307184647, recorded no MariaDB row, because the MariaDB CRD still served its scale subresource, and is not used.

Derivation output ​

The derivation printed this Markdown for the recorded run. It predates the 1Gi broker floor (#1298): on the same inputs the derivation now prints 815m,1Gi for minimalMessaging and 815m, 1Gi for the RabbitmqCluster row.

Derived figures ​

ConstantValue
memoryBase16Mi
defaultMemoryPerProcess352Mi
glanceMemoryPerProcess1Gi
defaultCPURequest70m
minimalServiceCPURequest15m
minimalDatabase65m,1Gi
minimalCache15m,96Mi
minimalMessaging815m,512Mi
minimalSecretStore35m,64Mi
sidecarCPURequest25m
sidecarMemory256Mi

Input: 196 rows from 14 job runs. T is the memory target less 32Mi per extra thread of each process.

Memory formula ​

pM(p), the largest T (MiB)16Mi + p × 352Mi
1363368
2363720
410521424

The steepest slope between two process counts is 344.5Mi, which rounds up to 352Mi.

These containers have a fixed memory figure and stay out of the fit: Cinder/backup/backup, NeutronMetadataAgent/metadata-agent/metadata-agent.

Glance ​

JobLegWorkloadptT (MiB)(T − memoryBase) / p
e2e-controlplane-controlplane-keystone-glance11284268
tempestcinder-2025.2glance-cinder-tempest-2025-2211484734
tempestcinder-2026.1glance-cinder-tempest-2026-12120631023.5
tempestglance-2025.2glance-tempest-2025-22115670
tempestnova-2025.2glance-nova-tempest-2025-221423203.5
tempestnova-2026.1glance-nova-tempest-2026-121363173.5

Render-time CPU request ​

Key (owner_kind/component/container)Largest CPU target (m)
Barbican/api/barbican-api11
Cinder/api/cinder-api182
Cinder/backup/backup1101
Cinder/scheduler/scheduler78
Cinder/volume-nfs1/volume-nfs1296
Glance/api/glance-api627
Keystone/api/keystone1938
Keystone/api/trust-flush0
Neutron/api/neutron-api1836
Neutron/ovn-maintenance-worker/ovn-maintenance-worker35
Neutron/periodic-workers/periodic-workers23
NeutronMetadataAgent/metadata-agent/metadata-agent23
Nova/api/nova-api247
Nova/conductor/conductor203
Nova/metadata/nova-metadata11
Nova/novncproxy/novncproxy11
Nova/scheduler/scheduler63
OVNCentral/nb/ovsdb49
OVNCentral/northd/northd35
OVNCentral/sb/ovsdb63
Placement/api/placement-api49

Median of 21 keys: 63m, so defaultCPURequest is 70m.

Minimal service CPU request ​

Key (owner_kind/component/container)Largest CPU target (m)
Barbican/api/barbican-api11
Cinder/api/cinder-api11
Cinder/backup/backup716
Cinder/scheduler/scheduler35
Cinder/volume-nfs1/volume-nfs149
Glance/api/glance-api23
Horizon/api/horizon271
Keystone/api/keystone296
Neutron/api/neutron-api11
Neutron/ovn-maintenance-worker/ovn-maintenance-worker11
Neutron/periodic-workers/periodic-workers11
Nova/api/nova-api11
Nova/conductor/conductor864
Nova/metadata/nova-metadata11
Nova/novncproxy/novncproxy11
Nova/scheduler/scheduler35
Placement/api/placement-api11

Median of 17 keys: 11m, which rounds up to 15m.

Minimal backing services ​

KindWorkloadContainerCPU target (m)Memory target (MiB)Figure
MariaDBopenstack-dbmariadb6348965m, 1Gi
Memcachedopenstack-memcachedmemcached112315m, 96Mi
RabbitmqClustershared-rabbitmq-serverrabbitmq813175815m, 512Mi
OpenBaoClustercontrolplane-keystone-barbican-baoopenbao356135m, 64Mi
OpenBaoClusteropenbao-instanceopenbao354835m, 64Mi

Sidecar ​

JobLegWorkloadCPU target (m)Memory target (MiB)
e2e-controlplane-sso-controlplane-sso-keystone1175

Budget projection ​

The Link 6z total of the measured e2e-controlplane node was 3480m CPU and 14754Mi of memory. These e2e-controlplane containers take the Minimal figures:

WorkloadContainerPodsCPU request (m)Memory request (MiB)
controlplane-keystone-barbicanbarbican-api150 → 15368 → 368
controlplane-keystone-barbican-baoopenbao1- → 35- → 64
controlplane-keystone-cindercinder-api150 → 15368 → 368
controlplane-keystone-cinder-backupbackup150 → 152048 → 2048
controlplane-keystone-cinder-schedulerscheduler150 → 15368 → 368
controlplane-keystone-cinder-volume-nfs1volume-nfs1150 → 15368 → 368
controlplane-keystone-glanceglance-api150 → 15624 → 1040
controlplane-keystone-horizonhorizon150 → 15512 → 720
controlplane-keystone-keystonekeystone150 → 15368 → 368
controlplane-keystone-neutronneutron-api150 → 15368 → 368
controlplane-keystone-neutron-ovn-maintenance-workerovn-maintenance-worker150 → 15368 → 368
controlplane-keystone-neutron-periodic-workersperiodic-workers150 → 15368 → 368
controlplane-keystone-novanova-api150 → 15368 → 368
controlplane-keystone-nova-conductorconductor150 → 15368 → 368
controlplane-keystone-nova-metadatanova-metadata150 → 15368 → 368
controlplane-keystone-nova-novncproxynovncproxy150 → 15368 → 368
controlplane-keystone-nova-schedulerscheduler150 → 15368 → 368
controlplane-keystone-placementplacement-api150 → 15368 → 368
openbao-instanceopenbao1- → 35- → 64
openstack-dbmariadb1100 → 651024 → 1024
openstack-memcachedmemcached150 → 15128 → 96
minimalServiceCPURequestProjected CPUProjected memory
15m2885m15474Mi

The node budget is 4000m CPU and 16384Mi of memory. The shared kind broker's memory request rose from 512Mi to 1Gi after this measurement (#1298), so the measured total and the projection are 512Mi low until the next measurement.

Input rows ​

joblegnamespaceowner_kindcomponentworkloadcontainerclassptcpu_target_mmemory_target_miT
e2e-controlplane-openstack--controlplane-keystone-fake-computenova-computeignored--11175-
e2e-controlplane-openstack--nfs-servernfs-serverignored--1175-
e2e-controlplane-openstackBarbicanapicontrolplane-keystone-barbicanbarbican-apiformula1111138138
e2e-controlplane-openstackCinderapicontrolplane-keystone-cindercinder-apiformula1111175175
e2e-controlplane-openstackCinderbackupcontrolplane-keystone-cinder-backupbackupformula, fixed memory11716260260
e2e-controlplane-openstackCinderschedulercontrolplane-keystone-cinder-schedulerschedulerformula1135156156
e2e-controlplane-openstackCindervolume-nfs1controlplane-keystone-cinder-volume-nfs1volume-nfs1formula1149309309
e2e-controlplane-openstackGlanceapicontrolplane-keystone-glanceglance-apiglance1123284284
e2e-controlplane-openstackHorizonapicontrolplane-keystone-horizonhorizonformula2111260260
e2e-controlplane-openstackKeystoneapicontrolplane-keystone-keystonekeystoneformula11126156156
e2e-controlplane-openstackMariaDB-openstack-dbmariadbbacking--63489-
e2e-controlplane-openstackMemcached-openstack-memcachedmemcachedbacking--1123-
e2e-controlplane-openstackNeutronapicontrolplane-keystone-neutronneutron-apiformula1111237237
e2e-controlplane-openstackNeutronovn-maintenance-workercontrolplane-keystone-neutron-ovn-maintenance-workerovn-maintenance-workerformula1111309309
e2e-controlplane-openstackNeutronperiodic-workerscontrolplane-keystone-neutron-periodic-workersperiodic-workersformula1111335335
e2e-controlplane-openstackNeutronMetadataAgentmetadata-agentcontrolplane-keystone-agent-metadata-agentmetadata-agentformula, fixed memory1111175175
e2e-controlplane-openstackNovaapicontrolplane-keystone-novanova-apiformula1111215215
e2e-controlplane-openstackNovaconductorcontrolplane-keystone-nova-conductorconductorformula11864284284
e2e-controlplane-openstackNovametadatacontrolplane-keystone-nova-metadatanova-metadataformula1111156156
e2e-controlplane-openstackNovanovncproxycontrolplane-keystone-nova-novncproxynovncproxyformula1111138138
e2e-controlplane-openstackNovaschedulercontrolplane-keystone-nova-schedulerschedulerformula1135156156
e2e-controlplane-openstackOVNCentralnbcontrolplane-keystone-ovn-nbovsdbignored--2311-
e2e-controlplane-openstackOVNCentralnorthdcontrolplane-keystone-ovn-northdnorthdformula11231111
e2e-controlplane-openstackOVNCentralsbcontrolplane-keystone-ovn-sbovsdbignored--2323-
e2e-controlplane-openstackOVNChassisovn-controllercontrolplane-keystone-chassis-ovn-controllerovn-controllerignored--2311-
e2e-controlplane-openstackOVNChassisovscontrolplane-keystone-chassis-ovsovs-vswitchdignored--2311-
e2e-controlplane-openstackOVNChassisovscontrolplane-keystone-chassis-ovsovsdb-serverignored--2311-
e2e-controlplane-openstackOpenBaoCluster-controlplane-keystone-barbican-baoopenbaobacking--3561-
e2e-controlplane-openstackOpenBaoCluster-openbao-instanceopenbaobacking--3548-
e2e-controlplane-openstackPlacementapicontrolplane-keystone-placementplacement-apiformula1111105105
e2e-controlplane-openstackRabbitmqClusterrabbitmqshared-rabbitmq-serverrabbitmqbacking--813175-
e2e-controlplane-sso-openstack--keycloakkeycloakignored--1737684-
e2e-controlplane-sso-openstack--openldapopenldapignored--11777-
e2e-controlplane-sso-openstackHorizonapicontrolplane-sso-horizonhorizonformula21271260260
e2e-controlplane-sso-openstackKeystoneapicontrolplane-sso-keystonefederation-proxysidecar--1175-
e2e-controlplane-sso-openstackKeystoneapicontrolplane-sso-keystonekeystoneformula11296175175
e2e-controlplane-sso-openstackMariaDB-openstack-dbmariadbignored--78215-
e2e-controlplane-sso-openstackMemcached-openstack-memcachedmemcachedignored--1123-
e2e-controlplane-sso-openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestbarbican-2025.2openstackBarbicanapibarbican-tempest-2025-2barbican-apiformula4111489489
tempestbarbican-2025.2openstackKeystoneapikeystone-barbican-tempest-2025-2keystoneformula21126284284
tempestbarbican-2025.2openstackMariaDB-openstack-dbmariadbignored--49237-
tempestbarbican-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestbarbican-2025.2openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestbarbican-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestbarbican-2026.1openstackKeystoneapikeystone-barbican-tempest-2026-1keystoneformula2193284284
tempestbarbican-2026.1openstackMariaDB-openstack-dbmariadbignored--63237-
tempestbarbican-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestbarbican-2026.1openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestbarbican-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestcinder-2025.2openstack--nfs-servernfs-serverignored--1175-
tempestcinder-2025.2openstack--nova-cinder-tempest-2025-2-fake-computenova-computeignored--23175-
tempestcinder-2025.2openstackCinderapicinder-tempest-2025-2cinder-apiformula41182684684
tempestcinder-2025.2openstackCinderbackupcinder-tempest-2025-2-backupbackupformula, fixed memory11977335335
tempestcinder-2025.2openstackCinderschedulercinder-tempest-2025-2-schedulerschedulerformula1178156156
tempestcinder-2025.2openstackCindervolume-nfs1cinder-tempest-2025-2-volume-nfs1volume-nfs1formula11182363363
tempestcinder-2025.2openstackGlanceapiglance-cinder-tempest-2025-2glance-apiglance2114314841484
tempestcinder-2025.2openstackKeystoneapikeystone-cinder-tempest-2025-2keystoneformula21163284284
tempestcinder-2025.2openstackMariaDB-openstack-dbmariadbignored--182600-
tempestcinder-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestcinder-2025.2openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestcinder-2025.2openstackNeutronapineutron-cinder-tempest-2025-2neutron-apiformula41379826826
tempestcinder-2025.2openstackNeutronovn-maintenance-workerneutron-cinder-tempest-2025-2-ovn-maintenance-workerovn-maintenance-workerformula1111309309
tempestcinder-2025.2openstackNeutronperiodic-workersneutron-cinder-tempest-2025-2-periodic-workersperiodic-workersformula1111335335
tempestcinder-2025.2openstackNovaapinova-cinder-tempest-2025-2nova-apiformula41126729729
tempestcinder-2025.2openstackNovaconductornova-cinder-tempest-2025-2-conductorconductorformula21182284284
tempestcinder-2025.2openstackNovametadatanova-cinder-tempest-2025-2-metadatanova-metadataformula2111309309
tempestcinder-2025.2openstackNovanovncproxynova-cinder-tempest-2025-2-novncproxynovncproxyformula1111138138
tempestcinder-2025.2openstackNovaschedulernova-cinder-tempest-2025-2-schedulerschedulerformula2149237237
tempestcinder-2025.2openstackOVNCentralnbovn-cinder-tempest-2025-2-nbovsdbignored--2311-
tempestcinder-2025.2openstackOVNCentralnorthdovn-cinder-tempest-2025-2-northdnorthdformula11231111
tempestcinder-2025.2openstackOVNCentralsbovn-cinder-tempest-2025-2-sbovsdbignored--2311-
tempestcinder-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestcinder-2025.2openstackPlacementapiplacement-cinder-tempest-2025-2placement-apiformula2123195195
tempestcinder-2025.2openstackRabbitmqClusterrabbitmqshared-rabbitmq-serverrabbitmqignored--23175-
tempestcinder-2026.1openstack--nfs-servernfs-serverignored--1175-
tempestcinder-2026.1openstack--nova-cinder-tempest-2026-1-fake-computenova-computeignored--23175-
tempestcinder-2026.1openstackCinderapicinder-tempest-2026-1cinder-apiformula41182641641
tempestcinder-2026.1openstackCinderbackupcinder-tempest-2026-1-backupbackupformula, fixed memory111101309309
tempestcinder-2026.1openstackCinderschedulercinder-tempest-2026-1-schedulerschedulerformula1163156156
tempestcinder-2026.1openstackCindervolume-nfs1cinder-tempest-2026-1-volume-nfs1volume-nfs1formula11296335335
tempestcinder-2026.1openstackGlanceapiglance-cinder-tempest-2026-1glance-apiglance2162720632063
tempestcinder-2026.1openstackKeystoneapikeystone-cinder-tempest-2026-1keystoneformula21271284284
tempestcinder-2026.1openstackKeystoneapikeystone-cinder-tempest-2026-1trust-flushformula11011
tempestcinder-2026.1openstackMariaDB-openstack-dbmariadbignored--182600-
tempestcinder-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestcinder-2026.1openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestcinder-2026.1openstackNeutronapineutron-cinder-tempest-2026-1neutron-apiformula41296777777
tempestcinder-2026.1openstackNeutronovn-maintenance-workerneutron-cinder-tempest-2026-1-ovn-maintenance-workerovn-maintenance-workerformula1111309309
tempestcinder-2026.1openstackNeutronperiodic-workersneutron-cinder-tempest-2026-1-periodic-workersperiodic-workersformula1123260260
tempestcinder-2026.1openstackNovaapinova-cinder-tempest-2026-1nova-apiformula4149684684
tempestcinder-2026.1openstackNovaconductornova-cinder-tempest-2026-1-conductorconductorformula2193284284
tempestcinder-2026.1openstackNovametadatanova-cinder-tempest-2026-1-metadatanova-metadataformula2111309309
tempestcinder-2026.1openstackNovanovncproxynova-cinder-tempest-2026-1-novncproxynovncproxyformula1111138138
tempestcinder-2026.1openstackNovaschedulernova-cinder-tempest-2026-1-schedulerschedulerformula2135260260
tempestcinder-2026.1openstackOVNCentralnbovn-cinder-tempest-2026-1-nbovsdbignored--2323-
tempestcinder-2026.1openstackOVNCentralnorthdovn-cinder-tempest-2026-1-northdnorthdformula11231111
tempestcinder-2026.1openstackOVNCentralsbovn-cinder-tempest-2026-1-sbovsdbignored--2311-
tempestcinder-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--3561-
tempestcinder-2026.1openstackPlacementapiplacement-cinder-tempest-2026-1placement-apiformula2123195195
tempestcinder-2026.1openstackRabbitmqClusterrabbitmqshared-rabbitmq-serverrabbitmqignored--23195-
tempestglance-2025.2openstackGlanceapiglance-tempest-2025-2glance-apiglance2123156156
tempestglance-2025.2openstackKeystoneapikeystone-glance-tempest-2025-2keystoneformula21143284284
tempestglance-2025.2openstackMariaDB-openstack-dbmariadbignored--63215-
tempestglance-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestglance-2025.2openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestglance-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestglance-2026.1openstackKeystoneapikeystone-glance-tempest-2026-1keystoneformula21126284284
tempestglance-2026.1openstackMariaDB-openstack-dbmariadbignored--93309-
tempestglance-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestglance-2026.1openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestglance-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestkeystone-2025.2openstackKeystoneapikeystone-tempest-2025-2keystoneformula211938335335
tempestkeystone-2025.2openstackKeystoneapikeystone-tempest-2025-2trust-flushformula11011
tempestkeystone-2025.2openstackMariaDB-openstack-dbmariadbignored--442215-
tempestkeystone-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestkeystone-2025.2openstackMemcached-openstack-memcachedmemcachedignored--4961-
tempestkeystone-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestkeystone-2026.1openstackKeystoneapikeystone-tempest-2026-1keystoneformula211938335335
tempestkeystone-2026.1openstackKeystoneapikeystone-tempest-2026-1trust-flushformula11011
tempestkeystone-2026.1openstackMariaDB-openstack-dbmariadbignored--442215-
tempestkeystone-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestkeystone-2026.1openstackMemcached-openstack-memcachedmemcachedignored--4961-
tempestkeystone-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestneutron-2025.2openstackKeystoneapikeystone-neutron-tempest-2025-2keystoneformula21511284284
tempestneutron-2025.2openstackMariaDB-openstack-dbmariadbignored--627561-
tempestneutron-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestneutron-2025.2openstackMemcached-openstack-memcachedmemcachedignored--1123-
tempestneutron-2025.2openstackNeutronapineutron-tempest-2025-2neutron-apiformula411836991991
tempestneutron-2025.2openstackNeutronovn-maintenance-workerneutron-tempest-2025-2-ovn-maintenance-workerovn-maintenance-workerformula1123335335
tempestneutron-2025.2openstackNeutronperiodic-workersneutron-tempest-2025-2-periodic-workersperiodic-workersformula1111335335
tempestneutron-2025.2openstackOVNCentralnbovn-neutron-tempest-2025-2-nbovsdbignored--3511-
tempestneutron-2025.2openstackOVNCentralnorthdovn-neutron-tempest-2025-2-northdnorthdformula11351111
tempestneutron-2025.2openstackOVNCentralsbovn-neutron-tempest-2025-2-sbovsdbignored--3523-
tempestneutron-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--3548-
tempestneutron-2026.1openstackKeystoneapikeystone-neutron-tempest-2026-1keystoneformula21671284284
tempestneutron-2026.1openstackMariaDB-openstack-dbmariadbignored--548600-
tempestneutron-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestneutron-2026.1openstackMemcached-openstack-memcachedmemcachedignored--2323-
tempestneutron-2026.1openstackNeutronapineutron-tempest-2026-1neutron-apiformula411737991991
tempestneutron-2026.1openstackNeutronovn-maintenance-workerneutron-tempest-2026-1-ovn-maintenance-workerovn-maintenance-workerformula1135309309
tempestneutron-2026.1openstackNeutronperiodic-workersneutron-tempest-2026-1-periodic-workersperiodic-workersformula1111260260
tempestneutron-2026.1openstackOVNCentralnbovn-neutron-tempest-2026-1-nbovsdbignored--4911-
tempestneutron-2026.1openstackOVNCentralnorthdovn-neutron-tempest-2026-1-northdnorthdformula11351111
tempestneutron-2026.1openstackOVNCentralsbovn-neutron-tempest-2026-1-sbovsdbignored--6323-
tempestneutron-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestnova-2025.2openstack--nova-tempest-2025-2-fake-computenova-computeignored--78175-
tempestnova-2025.2openstackGlanceapiglance-nova-tempest-2025-2glance-apiglance2135423423
tempestnova-2025.2openstackKeystoneapikeystone-nova-tempest-2025-2keystoneformula21350309309
tempestnova-2025.2openstackKeystoneapikeystone-nova-tempest-2025-2trust-flushformula11011
tempestnova-2025.2openstackMariaDB-openstack-dbmariadbignored--410684-
tempestnova-2025.2openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestnova-2025.2openstackMemcached-openstack-memcachedmemcachedignored--2323-
tempestnova-2025.2openstackNeutronapineutron-nova-tempest-2025-2neutron-apiformula4171610521052
tempestnova-2025.2openstackNeutronovn-maintenance-workerneutron-nova-tempest-2025-2-ovn-maintenance-workerovn-maintenance-workerformula1123309309
tempestnova-2025.2openstackNeutronperiodic-workersneutron-nova-tempest-2025-2-periodic-workersperiodic-workersformula1111335335
tempestnova-2025.2openstackNeutronMetadataAgentmetadata-agentneutron-nova-tempest-2025-2-agent-metadata-agentmetadata-agentformula, fixed memory1123175175
tempestnova-2025.2openstackNovaapinova-tempest-2025-2nova-apiformula41143826826
tempestnova-2025.2openstackNovaconductornova-tempest-2025-2-conductorconductorformula21203309309
tempestnova-2025.2openstackNovametadatanova-tempest-2025-2-metadatanova-metadataformula2111309309
tempestnova-2025.2openstackNovanovncproxynova-tempest-2025-2-novncproxynovncproxyformula1111138138
tempestnova-2025.2openstackNovaschedulernova-tempest-2025-2-schedulerschedulerformula2163260260
tempestnova-2025.2openstackOVNCentralnbovn-nova-tempest-2025-2-nbovsdbignored--3523-
tempestnova-2025.2openstackOVNCentralnorthdovn-nova-tempest-2025-2-northdnorthdformula11351111
tempestnova-2025.2openstackOVNCentralsbovn-nova-tempest-2025-2-sbovsdbignored--3523-
tempestnova-2025.2openstackOVNChassisovn-controllerovn-nova-tempest-2025-2-chassis-ovn-controllerovn-controllerignored--2311-
tempestnova-2025.2openstackOVNChassisovsovn-nova-tempest-2025-2-chassis-ovsovs-vswitchdignored--2323-
tempestnova-2025.2openstackOVNChassisovsovn-nova-tempest-2025-2-chassis-ovsovsdb-serverignored--2311-
tempestnova-2025.2openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestnova-2025.2openstackPlacementapiplacement-nova-tempest-2025-2placement-apiformula2135195195
tempestnova-2025.2openstackRabbitmqClusterrabbitmqshared-rabbitmq-serverrabbitmqignored--35195-
tempestnova-2026.1openstack--nova-tempest-2026-1-fake-computenova-computeignored--93175-
tempestnova-2026.1openstackGlanceapiglance-nova-tempest-2026-1glance-apiglance2111363363
tempestnova-2026.1openstackKeystoneapikeystone-nova-tempest-2026-1keystoneformula21350309309
tempestnova-2026.1openstackKeystoneapikeystone-nova-tempest-2026-1trust-flushformula11011
tempestnova-2026.1openstackMariaDB-openstack-dbmariadbignored--323729-
tempestnova-2026.1openstackMariaDB-openstack-db-metricsexporterignored--1111-
tempestnova-2026.1openstackMemcached-openstack-memcachedmemcachedignored--2323-
tempestnova-2026.1openstackNeutronapineutron-nova-tempest-2026-1neutron-apiformula41627991991
tempestnova-2026.1openstackNeutronovn-maintenance-workerneutron-nova-tempest-2026-1-ovn-maintenance-workerovn-maintenance-workerformula1123309309
tempestnova-2026.1openstackNeutronperiodic-workersneutron-nova-tempest-2026-1-periodic-workersperiodic-workersformula1111260260
tempestnova-2026.1openstackNeutronMetadataAgentmetadata-agentneutron-nova-tempest-2026-1-agent-metadata-agentmetadata-agentformula, fixed memory1111156156
tempestnova-2026.1openstackNovaapinova-tempest-2026-1nova-apiformula41247826826
tempestnova-2026.1openstackNovaconductornova-tempest-2026-1-conductorconductorformula21203309309
tempestnova-2026.1openstackNovametadatanova-tempest-2026-1-metadatanova-metadataformula2111309309
tempestnova-2026.1openstackNovanovncproxynova-tempest-2026-1-novncproxynovncproxyformula1111138138
tempestnova-2026.1openstackNovaschedulernova-tempest-2026-1-schedulerschedulerformula2163363363
tempestnova-2026.1openstackOVNCentralnbovn-nova-tempest-2026-1-nbovsdbignored--3511-
tempestnova-2026.1openstackOVNCentralnorthdovn-nova-tempest-2026-1-northdnorthdformula11231111
tempestnova-2026.1openstackOVNCentralsbovn-nova-tempest-2026-1-sbovsdbignored--3523-
tempestnova-2026.1openstackOVNChassisovn-controllerovn-nova-tempest-2026-1-chassis-ovn-controllerovn-controllerignored--2311-
tempestnova-2026.1openstackOVNChassisovsovn-nova-tempest-2026-1-chassis-ovsovs-vswitchdignored--2311-
tempestnova-2026.1openstackOVNChassisovsovn-nova-tempest-2026-1-chassis-ovsovsdb-serverignored--2311-
tempestnova-2026.1openstackOpenBaoCluster-openbao-instanceopenbaoignored--4948-
tempestnova-2026.1openstackPlacementapiplacement-nova-tempest-2026-1placement-apiformula2149195195
tempestnova-2026.1openstackRabbitmqClusterrabbitmqshared-rabbitmq-serverrabbitmqignored--49195-

Old and new values ​

ConstantBefore the runFrom the run
memoryBase224Mi16Mi
defaultMemoryPerProcess144Mi352Mi
glanceMemoryPerProcess400Mi1Gi
defaultCPURequest100m70m
minimalServiceCPURequest50m15m
minimalDatabase100m, 1Gi65m, 1Gi
minimalCache50m, 128Mi15m, 96Mi
minimalMessaging100m, 512Mi815m, 512Mi
minimalSecretStore50m, 256Mi35m, 64Mi
sidecarCPURequest25m25m
sidecarMemory256Mi256Mi

The old figures were estimates: the memory formula from the working sets of CI job 107440936441, glanceMemoryPerProcess from the Glance API with the S3 store driver under image traffic, defaultCPURequest from when the default CPU limit was dropped, and the Minimal figures chosen to pass the node budget gate.

The formula still renders 368Mi at one process, so a single-process service keeps its memory. At two processes it renders 720Mi instead of 512Mi, and at four 1424Mi instead of 800Mi. A Glance API renders 1040Mi at one process and 2064Mi at two.

Metadata agent memory ​

The memory of the NeutronMetadataAgent containers is metadataAgentMemory in operators/neutron/internal/controller/reconcile_daemonset.go. It is sized for 32 networks on the agent's node and comes from the metadata-agent suite (tests/e2e/neutron/metadata-agent), which runs in the neutron leg of e2e-operator.

What the suite measures ​

The agent container holds the agent, its privsep daemons and one haproxy per network with a port bound on the node. Steps 9 to 11 of the suite write networks into the Northbound database with ovn-nbctl, bind one port of each on the kind node, and read the container's cgroup in four phases:

ReleaseNetworksHow the agent gets there
2026.10No port is bound on the node
2026.11The port of one network is bound
2026.132The ports of 31 more networks are bound
2026.232A restart into 2026.2, whose first sync provisions all 32 networks

A phase polls every 5 seconds until the agent runs one haproxy per network, and for 30 seconds more. Its reading is the largest working set of those polls: memory.current less the inactive_file of memory.stat, rounded up to MiB. The proxies are idle, because no instance asks them for metadata. The phase prints one line:

text
MEMORY-READING release=<release> networks=<n> working_set_mi=<MiB> peak_mi=<MiB> oom_kill=<count> haproxy=<count> limit_mi=<MiB>

It fails on a container restart, on an OOM kill in the cgroup and on a working set above 90% of the container's memory limit. A default that no longer holds 32 networks therefore fails the leg.

After each phase with 32 networks the suite prints one MEMORY-PROCESSES line per process name in the container, with the process count and the sum of VmRSS. VmRSS counts a shared page once per process, so the sums can exceed the working set.

From the readings to the figure ​

W1 is the working set of 2026.1 with one network, W32a that of 2026.1 with 32 networks and W32b that of 2026.2 with 32 networks, all in MiB.

FigureRuleMeaning
Sceil((W32a − W1) / 31), at least 1The memory one more network adds
Lmax(0, 550 − W1)What the lab reading with one network lies above the runner's
F(max(W32a, W32b) + L) × 1.25, rounded up to a multiple of 64 and at least 768metadataAgentMemory in MiB

The 550 is the working set of an agent with one network on a hypervisor of the metal-stack lab (the run of #1142, 2026-10-01). A hypervisor may run larger privsep daemons than a CI runner, so the lab reading enters as an offset. The factor 1.25 is the headroom for metadata requests, which the idle proxies of the suite do not serve. The 768 is the memory the lab agent ran with.

S sizes a node beyond the default: an agent whose node binds n networks, n above 32, needs about F + (n − 32) × S × 1.25 MiB, rounded up to a multiple of 64. The factor gives each further proxy the headroom F gives the first 32.

Metadata agent CPU ​

The CPU request of the NeutronMetadataAgent containers is metadataAgentCPU in operators/neutron/internal/controller/reconcile_daemonset.go, 230m. It comes from a session on the metal-stack lab and from no CI job: no CI leg boots a server through the agent, so its CI rows read an idle agent, 11 to 23 m. The lab measurement read a target of 126m against the 70m the container requested then (#1259), on one pass of the quick start with one server per node. The session below loaded one node on purpose.

What the session measured ​

The agent idles and works in bursts. When the first port of a network is bound on its node, it creates the network's namespace and starts the network's haproxy, and the booting server then asks it for metadata. The session read the container's cumulative CPU time from the kubelet (usageCoreNanoSeconds of /api/v1/nodes/<node>/proxy/stats/summary) every 10 seconds, which runs nothing in the measured container. A reading is the CPU time between two samples at least 60 seconds apart, in millicores, and the readings of a phase do not overlap.

PhaseLoad on the nodeLengthReadings (m)CPU time
A0The agent runs, no server300 s1.0 to 1.20.3 s
A1One server boots on one new network379 s150, then 1.0 to 1.111.4 s
A231 servers boot at once, each on a new network of its own296 s198, 63, 1.3, 1.116.4 s
A3The 32 servers run on their 32 networks600 s1.1 to 1.40.7 s
A4The 32 servers are hard-rebooted within 29 seconds282 s357, 1.5, 1.3, 1.223.8 s

Every server ran the cirros-kvm image on flavor 1 and was placed on the node with --availability-zone. In A2 the agent logged Provisioning metadata for network 31 times, the console log of each of the 31 servers showed the metadata fetch of cirros-init, and the container ran 32 haproxy processes at the end of the phase. In A4 it logged 34 provisionings. The largest reading over 10 seconds was 342m in A1, 222m in A2 and 770m in A4. The agents of the two other nodes, with one server each, read at most 5m from A2 to A4.

From the reading to the request ​

C is 1.15 times the 90th percentile of the readings of A2, A3 and A4, rounded up to a multiple of 10m, and at least 130m. Those phases hold 16 readings, whose 90th percentile is the second largest, 198m: 1.15 × 198 is 228, so C is 230m. The percentile and the 15 % are what a VPA recommender applies to a container's CPU samples, and the 130m is the 126m of the earlier lab run, rounded up.

The request covers the minute in which 31 servers booted on the node, and the single boot of A1. It does not cover the minute of A4. The agent has no CPU limit, so a burst above the request is not throttled: on a node whose CPUs are all busy the agent gets the share its request buys, and the burst takes longer.

The shoot's own recommender watched the DaemonSet's three pods for 56 minutes, from before A0 until the load had ended, and gave a CPU target of 11m, the lowest value it gives. The bursts fill few of its one-minute samples, so its 90th percentile is the idle agent. A burst weighs more in a shorter watch: the earlier run's lasted 22 minutes.

Recorded session ​

The session of 2026-10-05 ran on shoot newforge from commit d1f114f7, on Kubernetes v1.35.6, with three workers of 16 CPUs each and the agent image ghcr.io/c5c3/neutron:2025.2. It ran Part 1, Steps 2 to 7 and Part 2, Steps 1 to 5 of the Quick Start (metal-stack), loaded the node 7znw9, and tore the stack down again. Three things were done by hand. make deploy-infra left the glance-operator release and the k-orc Kustomization suspended, and both were resumed with kubectl patch. The c5c3-operator pod, which had exited when the Glance CRD arrived late, was deleted once out of its restart backoff. The first attempt at A4 passed 32 names to openstack server reboot, which takes one, and issued no reboot; the 53 seconds it lasted are in no reading.

With 32 networks bound on the node, the container's working set peaked at 730 MiB, against the 2Gi of Metadata agent memory; it was 551 MiB before the first server. The files of the session are in the comment on #1259.

Lab measurement ​

Every row above comes from pods that never ran a virtual machine. In the three jobs compute is Nova's fake driver: no chassis carries a server's traffic, and no libvirt runs. The lab measurement runs hack/ci-vpa-recommendations.sh by hand on the metal-stack lab while Part 2 of the Quick Start (metal-stack) boots, migrates and evicts servers on KVM hypervisors. It measures the namespaces openstack and hypervisor-system with the recommender the Gardener shoot runs, and sets the readings of the compute workloads against their defaults. It derives no figure: hack/derive-sizing-figures.py accepts the three CI jobs only, and a reading that contradicts a default becomes an issue of its own, as #1173 was for the metadata agent.

What differs from CI ​

TopicCI on kindThe lab
Recommenderdeploy/kind/vpa: VPA 1.8.0, floors of 1 millicore and 1 MB per podThe shoot's. Its image and flags run in the seed and cannot be read; report writes recommender=-. Gardener's source at gardener/gardener@92a252c0 passes floors of 10 millicores and 10 MB
Sampling--memory-saver=false: every pod from its startThe same source passes --memory-saver=true: a pod is sampled once a VPA selects it, at most 60 seconds after the workload appeared
Updater and admission controllernot installedBoth run. A VPA in mode Off changes no pod; the run checks it
MariaDB scale subresourceremoved by WITH_VPA=true make deploy-infraremoved by hack/ci-vpa-recommendations.sh prepare
ComputeNova's fake driverKVM on every worker, with one server each
Loadthe e2e suites and TempestPart 2 of the Quick Start (metal-stack), once
Reportjob= names a CI job; input of the derivationjob=lab; hack/derive-sizing-figures.py refuses it, and no constant is derived from it

The Gardener figures come from computeRecommenderArgs in pkg/component/autoscaling/vpa/recommender.go. The margin of 15 %, the CPU target at the 90th percentile and the one-minute interval it passes are defaults a shoot spec can override (pkg/apis/core/v1beta1/types_shoot.go), and neither the seed's Gardener version nor the shoot spec can be read from the shoot. They are what to expect; the record states what the run showed.

Running the lab measurement ​

The measurement runs beside one pass of the Quick Start (metal-stack). It starts on a bare shoot, in bash in the root of the clone, with KUBECONFIG set as in Part 1, Step 1. Run the whole session in one bash shell: the third block uses the helpers, out and the watch of the first. Each block names the quick start step it follows, and the quick start's own commands run there unchanged.

After Part 1, Step 3 and before Step 4, define the helpers, remove the MariaDB scale subresource and start the watch:

bash
out=_output/sizing-lab; mkdir -p "$out"
flat() { tr '\n' ' ' | sed 's/ *$//'; }
pod_state() { # <namespace>...: namespace, pod, UID, restarts, the admission controller's vpaUpdates annotation, requests
  local ns
  for ns in "$@"; do
    kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" '.items[] | select(.metadata.deletionTimestamp == null)
      | [$ns, .metadata.name, .metadata.uid, ([.status.containerStatuses[]?.restartCount] | add // 0),
         (.metadata.annotations.vpaUpdates // "-"),
         ([.spec.containers[] | "\(.name)=\(.resources.requests.cpu // "-")/\(.resources.requests.memory // "-")"] | join(","))] | @tsv'
  done | sort
}
no_recommendation() { # <namespace>...: the measurement's VPAs without a recommendation, with their conditions
  local ns
  for ns in "$@"; do
    kubectl get vpa -n "$ns" -l app.kubernetes.io/managed-by=ci-vpa-recommendations -o json |
      jq -r --arg ns "$ns" '.items[] | select((.status.recommendation.containerRecommendations // []) | length == 0)
        | [$ns, .metadata.name, ([.status.conditions[]? | "\(.type)=\(.status)"] | join(","))] | @tsv'
  done
}
libvirtd_scope() { # per libvirt pod: node, then memory.current, memory.peak (bytes) and the CPU time of libvirtd's host scope
  local pod
  for pod in $(kubectl get pod -n openstack -l app.kubernetes.io/name=libvirt -o name); do
    printf '%s %s\n' "$(kubectl get -n openstack "$pod" -o jsonpath='{.spec.nodeName}' 2>&1 | flat)" \
      "$(kubectl exec -n openstack "$pod" -c libvirtd -- sh -c 'cd /sys/fs/cgroup/system.slice/cobaltcore-libvirtd.scope && cat memory.current memory.peak && grep usage_usec cpu.stat' 2>&1 | flat)"
  done
}
floors() { # <recommendations.tsv>: per workload its CPU bound, memory bound, containers and name
  awk -F'\t' 'NR > 2 { k = $1 "/" $2 "/" $3; n[k]++
      if (!(k in c) || $12 + 0 < c[k]) c[k] = $12 + 0
      if (!(k in m) || $13 + 0 < m[k]) m[k] = $13 + 0 }
    END { for (k in n) print n[k] * (c[k] + 1), n[k] * m[k], n[k], k }' "$1" | sort -n
}
floor_bounds() { # <recommendations.tsv>: the smallest bound of each resource
  floors "$1" | awk 'NR == 1 || $1 < c { c = $1 } NR == 1 || $2 < m { m = $2 }
    END { if (NR) print "floor bounds: cpu below " c "m, memory at most " m "Mi" }'
}
verdicts() { # <recommendations.tsv>: the rows of the compute workloads with a verdict per resource
  awk -F'\t' '
    function verdict(target, request) { return request == "-" ? "no default" : (target + 0 <= request + 0 ? "confirmed" : "contradicted") }
    NR > 2 && $3 ~ /^(lab-nova-compute|lab-chassis-ovn-controller|lab-chassis-ovs|lab-metadata-agent-metadata-agent|libvirt)$/ {
      printf "%s %s cpu %sm request %s: %s; memory %sMi request %s: %s\n", $3, $9, $12, $16, verdict($12, $16), $13, $17, verdict($13, $17) }' "$1"
}
over_request() { # <recommendations.tsv>: every row whose memory target exceeds its memory request
  awk -F'\t' 'NR > 2 && $17 != "-" && $13 + 0 > $17 + 0 { print $1, $3, $9, "memory target " $13 "Mi above request " $17 "Mi" }' "$1"
}

hack/ci-vpa-recommendations.sh prepare 2>&1 | tee "$out/prepare.log"
nohup hack/ci-vpa-recommendations.sh watch "$out" openstack hypervisor-system >"$out/watch.log" 2>&1 &

prepare runs once the deploy has installed the MariaDB CRD, before anything creates a database. The watch starts before Step 4 applies the ControlPlane, as CI starts it before its suites, so each workload gets its VPA within 60 seconds of its creation. nohup and & keep it running after the command that started it. openstack holds the ControlPlane's services and the five compute DaemonSets, hypervisor-system kvm-node-agent and the migration port reservation, two more workloads that run only on a hypervisor.

The last four helpers read recommendations.tsv by column, below its comment line and its header (see the column table): 3 workload, 9 container, 12 cpu_target_m, 13 memory_target_mi, 16 cpu_request_m and 17 memory_request_mi. verdicts picks the five DaemonSets that the CRs in deploy/lab/metal-stack/hypervisor/compute.yaml and the lab's libvirt manifest render:

WorkloadRendered from
lab-nova-computethe NovaCompute lab, through novaComputeDaemonSetName in operators/nova/internal/controller/reconcile_novacompute_daemonset.go
lab-chassis-ovn-controller, lab-chassis-ovsthe OVNChassis lab-chassis, through chassisControllerName and chassisOVSName in operators/ovn/internal/controller/
lab-metadata-agent-metadata-agentthe NeutronMetadataAgent lab-metadata-agent, through agentDaemonSetName in operators/neutron/internal/controller/reconcile_daemonset.go
libvirtdeploy/lab/metal-stack/hypervisor/libvirt-daemonset.yaml

A VPA reads a pod's cgroup. On the lab, libvirtd runs in the host scope cobaltcore-libvirtd.scope (the systemd-run line of deploy/lab/metal-stack/hypervisor/libvirt-configmap.yaml) and QEMU in libvirt's own /machine cgroups, both outside the libvirt pod's cgroup, so the libvirt row covers bash and the probes. libvirtd_scope reads what no VPA gives: memory.current, memory.peak and the CPU time of libvirtd's scope, through the container's mount of the host's /sys/fs/cgroup.

After the checks of Part 1, Step 7, record every pod:

bash
pod_state openstack hypervisor-system >"$out/pods.before"

Then run Part 2. After its last step, Part 2, Step 11, while the servers still run, end the measurement:

bash
sleep 300
libvirtd_scope | tee "$out/libvirtd-scope.log"
kill "$(cat "$out/watch.pid")"
hack/ci-vpa-recommendations.sh prepare 2>&1 | tee -a "$out/prepare.log"
hack/ci-vpa-recommendations.sh snapshot "$out" openstack hypervisor-system
GITHUB_SHA="$(git rev-parse HEAD)" GITHUB_JOB=lab \
  MEASURE_LEG="$(kubectl get configmap shoot-info -n kube-system -o jsonpath='{.data.shootName}')" \
  hack/ci-vpa-recommendations.sh report "$out" >/dev/null
pod_state openstack hypervisor-system >"$out/pods.after"
{ diff "$out/pods.before" "$out/pods.after" | grep '^<' || true; } | tee "$out/pods.changed"
awk -F'\t' '$5 != "-"' "$out/pods.after" | tee "$out/pods.rewritten"
no_recommendation openstack hypervisor-system | tee "$out/no-recommendation.log"
floors "$out/recommendations.tsv" | tee "$out/floors.log"
floor_bounds "$out/recommendations.tsv" | tee -a "$out/floors.log"
verdicts "$out/recommendations.tsv" | tee "$out/verdicts.log"
over_request "$out/recommendations.tsv" | tee "$out/over-request.log"
kubectl delete vpa -n openstack -l app.kubernetes.io/managed-by=ci-vpa-recommendations --wait --timeout=120s
kubectl delete vpa -n hypervisor-system -l app.kubernetes.io/managed-by=ci-vpa-recommendations --wait --timeout=120s
kubectl get vpa -A -l app.kubernetes.io/managed-by=ci-vpa-recommendations

The sleep gives the recommender five more samples with every server running. The watch is stopped, and one last snapshot follows it. MEASURE_LEG carries the shoot's name into the comment line of recommendations.tsv, which then reads run=- attempt=- sha=<commit> job=lab leg=<shoot> recommender=-.

  • A line of prepare.log that starts with ci-vpa-recommendations: is a failed prepare, whose exit code the pipe into tee hides. The second prepare line reads MariaDB CRD serves no scale subresource unless a chart upgrade of mariadb-operator-crds restored the subresource during the run.
  • pods.changed lists the lines of pods.before that are gone or differ, in UID, restart count or requests.
  • pods.rewritten lists the pods the admission controller rewrote. A VPA in mode Off rewrites none.
  • no-recommendation.log lists the measurement's VPAs that hold no recommendation at the end, with their conditions.
  • floors.log, verdicts.log and over-request.log are what the rules of Reading the lab report read.
  • The last command prints No resources found.

The subresource stays removed until the next chart upgrade of mariadb-operator-crds or until make teardown-infra deletes the CRD: k8s.mariadb.com is among the STACK_CRD_GROUPS of hack/teardown-infra.sh. The run's files stay in _output/sizing-lab. Git ignores _output/, and the teardown does not touch it.

Reading the lab report ​

The floor rule. The recommender raises a container's recommendation to its pod's floor divided by the pod's containers. On either recommender, a container with usage samples reads at least 11m and 11Mi: the lowest histogram bucket (10 millicores, 10 MB) plus the 15 % margin. A floor therefore shows only where its share exceeds that. For a workload with n containers, the floor lies below n × (smallest CPU target + 1) millicores and does not exceed n × smallest memory target MiB. The + 1 covers the recommender's rounding to whole millicores, and the report rounds memory up. floors prints these bounds per workload and floor_bounds the smallest of each. When it prints a CPU bound of at most 12m and a memory bound of at most 11Mi, no lab row is a floor share, and a lab row compares with the CI row of the same container at any value. Otherwise, with B the bound floor_bounds printed for the resource, a row of a workload with n containers whose target is at most B / n is marked "at the floor" in the record and gets no verdict for that resource.

The verdict rule. Per container of the five compute workloads and per resource, verdicts prints no default when the workload renders no request, confirmed when the target is at most the request, and contradicted when it is above. The target is what the recommender would set as the request: the 90th percentile of CPU and the peak of memory, each plus 15 %. The upper bound is not used, because after hours instead of days of samples it is a multiple of the target. These are the defaults the verdicts read:

WorkloadDefaultSource
lab-nova-computeCPU request 150m, memory 768Mi as request and limit; none at the recorded runnovaComputeCPU, novaComputeMemory and effectiveNovaComputeResources, operators/nova/internal/controller/reconcile_novacompute_daemonset.go
lab-chassis-ovn-controller, lab-chassis-ovsnonechassisResources, operators/ovn/internal/controller/reconcile_ovs.go
lab-metadata-agent-metadata-agentCPU request 230m, memory 2Gi as request and limit; the CPU request was 70m at the recorded runmetadataAgentCPU, metadataAgentMemory and effectiveAgentResources, operators/neutron/internal/controller/reconcile_daemonset.go
libvirt, container libvirtdCPU request 100m, memory request 256Mi, limit 512Mi, for bash and the probesthe libvirtd container's resources, deploy/lab/metal-stack/hypervisor/libvirt-daemonset.yaml

Every other row is judged on memory alone: over_request lists the rows whose memory target exceeds the request. For a service row this contradicts memoryBase + p × defaultMemoryPerProcess at its process count, for a backing-service row its Minimal figure.

Recorded lab run ​

The run of 2026-10-05 measured on shoot newforge from commit d34fde24 (d34fde24334c2809ba2225654c206af9f217eee9), on Kubernetes v1.35.6, with the operators from the charts published from main (deploy/flux-system/releases/c5c3-operator.yaml). The shoot has three workers, whose == cpu line of the node probe reads Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz. Below, a node is named by the last part of its name, shoot--df33f0b4c1--newforge-group-0-85bcf-<suffix>. At 15:27:32Z the shoot's namespaces were default, firewall, kube-node-lease, kube-public, kube-system and metallb-system and no other. Its platform facts were those Lab autoscaling gives: the VPA CRD admitted Off, Initial, Recreate, InPlaceOrRecreate and Auto, kube-system held six VPAs in mode InPlaceOrRecreate, the resource metrics API was available and pods/resize was served. Of those VPAs, vpn-shoot read the target 11m and 11500000 and the CPU lower bound 10m, the lowest values the recommender gives (see the floor rule). Part 2, Step 5 booted one server on each worker, lab-0 on 7znw9, lab-1 on hmw67 and lab-2 on rt6kn, each bound to one network, lab-net. Step 8 moved lab-0 to rt6kn, and Step 9 evicted rt6kn, so from 15:48Z on 7znw9 ran lab-0 and lab-2, hmw67 ran lab-1, and rt6kn ran none. The watch's first pass ran at 15:33:06Z, before the ControlPlane was applied, and the last snapshot, the time every row carries, was taken at 15:55:08Z. The comment line of recommendations.tsv reads run=- attempt=- sha=d34fde24334c2809ba2225654c206af9f217eee9 job=lab leg=newforge recommender=-. Part 1, Steps 1 to 7 ran, without the git clone of Step 1 and without the optional checks Step 7 links, and Part 2, Steps 1 to 10 ran. The ControlPlane was Ready at 15:38:28Z, at the end of Part 1, Step 6. Every block of the quick start and of this page exited 0 on its first attempt. A script typed the console commands of Part 2, Steps 6 to 8 through tmux send-keys. Nothing was done by hand on the cluster: the kubeconfig had a current context, so no context was selected, and the two reads outside the pages, the server list after Step 10 and kubectl top after the third block, changed nothing. The files of the run are in the comment on #1224.

WorkloadContainerCPU target (m)Memory target (MiB)CPU requestMemory requestCI readingVerdict (CPU; memory)
lab-chassis-ovn-controllerovn-controller1111nonenone23 m, 11 MiBno default; no default
lab-chassis-ovsovs-vswitchd2323nonenone23 m, 11 to 23 MiBno default; no default
lab-chassis-ovsovsdb-server1111nonenone23 m, 11 MiBno default; no default
lab-metadata-agent-metadata-agentmetadata-agent12664170m2Gi11 to 23 m, 156 to 175 MiBcontradicted; confirmed
lab-nova-computenova-compute143455nonenone11 to 93 m, 175 MiB, on the fake driverno default; no default
libvirtlibvirtd1111100m256Minoneconfirmed; confirmed

lab-nova-compute rendered no request at the recorded run, so its targets of 143m and 455Mi under KVM stood without a default; on the fake driver CI read up to 93m and 175Mi. #1312 has since set the pool's default from these readings and the six-hour soak, 150m and 768Mi (see Resources). The two chassis DaemonSets render no request, and their three containers read 11 to 23 m and 11 to 23 MiB, at or below what CI read. The metadata agent's CPU request of 70m is contradicted by its target of 126m (#1259), and Metadata agent CPU has raised it to 230m since. Its memory of 2Gi is confirmed: the target was 641Mi, which is 557 MiB without the 15 % margin, beside the 550 MiB of the lab reading that L rests on (see From the readings to the figure). The libvirt pod's 100m and 256Mi are confirmed for bash and the probes, whose targets are the lowest values the recommender gives.

WorkloadContainersSmallest CPU targetSmallest memory targetCPU boundMemory bound
openstack/DaemonSet/lab-chassis-ovs211m11Mibelow 24mat most 22Mi

floors.log ends in floor bounds: cpu below 12m, memory at most 11Mi. The bounds are at most 12m and 11Mi, so no lab row is a floor share, and every lab row compares with the CI row of the same container at its value.

Nodememory.currentmemory.peakCPU timeServers on the node at the read
hmw6719 MiB51 MiB7.7 slab-1
rt6kn18 MiB48 MiB9.1 snone; lab-2 until Step 9, and lab-0 from Step 8 to Step 9
7znw919 MiB47 MiB9.0 slab-0 and lab-2 since Step 9; lab-0 until Step 8

The table gives cobaltcore-libvirtd.scope at the start of the third block, in MiB rounded up and in seconds. The scope holds libvirtd alone, which runs in the host's system.slice; no pod request covers it, and QEMU is in neither figure.

The checks of the run:

  • prepare.log holds MariaDB CRD scale subresource removed, printed at 15:33:06Z before Part 1, Step 4, and MariaDB CRD serves no scale subresource from the third block. The report has the row openstack-db, mariadb.
  • pods.changed is empty: each of the 38 pods of pods.before, read at 15:39Z, kept its UID, its restart count and its requests, the pods of the finished Jobs among them.
  • pods.rewritten is empty.
  • no-recommendation.log is empty: each of the 35 VPAs of the measurement held a recommendation, and the deletes left none (No resources found).

over-request.log holds three rows:

text
hypervisor-system migration-port-reservation hold memory target 11Mi above request 4Mi
openstack nfs-client-modules hold memory target 11Mi above request 8Mi
openstack nfs-server nfs-server memory target 75Mi above request 64Mi

The nfs-server row contradicts the 64Mi request of deploy/kind/nfs/nfs-server.yaml (#1260). That server was the kernel's nfsd. NFS-Ganesha replaced it on the day of the run, and its request of 640Mi comes from a reading of its own (see Lab NFS stack). The two hold rows read 11Mi, the lowest value the recommender gives (see the floor rule): their containers used less than 10 MB, and kubectl top read 0Mi for each of them at 15:56:10Z. They contradict neither 4Mi nor 8Mi, and no issue is filed for them. No service or backing-service row exceeds its memory request.

The table below is the report of the run, recommendations.md as the third block wrote it:

Sizing measurement: run=- attempt=- sha=d34fde24334c2809ba2225654c206af9f217eee9 job=lab leg=newforge recommender=-

namespacekindworkloadreplicasowner_kindowner_nameappcomponentcontainerprocessesthreadscpu_target_mmemory_target_micpu_upper_mmemory_upper_micpu_request_mmemory_request_mimemory_limit_misnapshot
hypervisor-systemDaemonSetkvm-node-agent-controller-manager---kvm-node-agent-manager--11351131355810641282026-10-05T15:55:08Z
hypervisor-systemDaemonSetmigration-port-reservation---migration-port-reservation-hold--11111138113614162026-10-05T15:55:08Z
openstackDeploymentcontrolplane-barbican1Barbicancontrolplane-barbicanbarbicanapibarbican-api111113894311835153683682026-10-05T15:55:08Z
openstackStatefulSetcontrolplane-barbican-bao1OpenBaoClustercontrolplane-barbican-baoopenbao-openbao--236128294900---2026-10-05T15:55:08Z
openstackDeploymentcontrolplane-cinder1Cindercontrolplane-cindercinderapicinder-api11231751161414179153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-cinder-backup1Cindercontrolplane-cindercinderbackupbackup--163215238231730015204820482026-10-05T15:55:08Z
openstackDeploymentcontrolplane-cinder-scheduler1Cindercontrolplane-cindercinderschedulerscheduler--231561171112761153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-cinder-volume-nfs11Cindercontrolplane-cindercindervolume-nfs1volume-nfs1--492601648821074153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-glance1Glancecontrolplane-glanceglanceapiglance-api--23284116252304015104010402026-10-05T15:55:08Z
openstackDeploymentcontrolplane-horizon1Horizoncontrolplane-horizonhorizonapihorizon2111260515021211157207202026-10-05T15:55:08Z
openstackDeploymentcontrolplane-keystone1Keystonecontrolplane-keystonekeystoneapikeystone111261561108612080153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-neutron1Neutroncontrolplane-neutronneutronapineutron-api11932602837521036153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-neutron-ovn-maintenance-worker1Neutroncontrolplane-neutronneutronovn-maintenance-workerovn-maintenance-worker--1136393330783153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-neutron-periodic-workers1Neutroncontrolplane-neutronneutronperiodic-workersperiodic-workers--1133594928902153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-nova1Novacontrolplane-novanovaapinova-api11232151484819581153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-nova-conductor1Novacontrolplane-novanovaconductorconductor--491561293314092153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-nova-metadata1Novacontrolplane-novanovametadatanova-metadata11111561314814327153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-nova-novncproxy1Novacontrolplane-novanovanovncproxynovncproxy--111381312112658153683682026-10-05T15:55:08Z
openstackDeploymentcontrolplane-nova-scheduler1Novacontrolplane-novanovaschedulerscheduler--231381302612566153683682026-10-05T15:55:08Z
openstackStatefulSetcontrolplane-ovn-nb1OVNCentralcontrolplane-ovnovncentralnbovsdb--111180580470256-2026-10-05T15:55:08Z
openstackDeploymentcontrolplane-ovn-northd1OVNCentralcontrolplane-ovnovncentralnorthdnorthd-11111839838703683682026-10-05T15:55:08Z
openstackStatefulSetcontrolplane-ovn-sb1OVNCentralcontrolplane-ovnovncentralsbovsdb--2311169580970256-2026-10-05T15:55:08Z
openstackDeploymentcontrolplane-placement1Placementcontrolplane-placementplacementapiplacement-api111110575268476153683682026-10-05T15:55:08Z
openstackDeploymenthypervisor-operator-controller-manager1--openstack-hypervisor-operator-manager--11352405361710641282026-10-05T15:55:08Z
openstackDaemonSetlab-chassis-ovn-controller-OVNChassislab-chassisovnchassisovn-controllerovn-controller--111112241222---2026-10-05T15:55:08Z
openstackDaemonSetlab-chassis-ovs-OVNChassislab-chassisovnchassisovsovs-vswitchd--232323662314---2026-10-05T15:55:08Z
openstackDaemonSetlab-chassis-ovs-OVNChassislab-chassisovnchassisovsovsdb-server--111111311129---2026-10-05T15:55:08Z
openstackDaemonSetlab-metadata-agent-metadata-agent-NeutronMetadataAgentlab-metadata-agentneutronmetadataagentmetadata-agentmetadata-agent--126641272607069270204820482026-10-05T15:55:08Z
openstackDaemonSetlab-nova-compute-NovaComputelabnovacomputenova-computenova-compute--1434552559247080---2026-10-05T15:55:08Z
openstackDaemonSetlibvirt---libvirt-libvirtd--1111112311211002565122026-10-05T15:55:08Z
openstackDaemonSetnfs-client-modules---nfs-client-modules-hold--111176276118322026-10-05T15:55:08Z
openstackDeploymentnfs-server1--nfs-server-nfs-server--1175775526050642562026-10-05T15:55:08Z
openstackStatefulSetopenbao-instance1OpenBaoClusteropenbao-instanceopenbao-openbao--236115954203---2026-10-05T15:55:08Z
openstackStatefulSetopenstack-db1MariaDBopenstack-dbmariadb-mariadb--6348946563607465102410242026-10-05T15:55:08Z
openstackDeploymentopenstack-memcached1Memcachedopenstack-memcachedmemcached-memcached--113580825431596962026-10-05T15:55:08Z
openstackStatefulSetopenstack-rabbitmq-server1RabbitmqClusteropenstack-rabbitmqopenstack-rabbitmqrabbitmqrabbitmq--231951685142408155125122026-10-05T15:55:08Z