Dependency Management

Two related processes that touch the same parts of the repository: bumping the Go toolchain across the workspace, and keeping third-party libraries, container base images, and GitHub Actions current. Renovate runs continuously and opens grouped PRs; this guide is the human-side rulebook for what Renovate does on its own, what needs a reviewer, and what is better handled in a dedicated PR.

The authoritative configuration lives in renovate.json at the repository root.

Renovate at a glance

Renovate is configured via config:recommended plus a small set of custom regex managers and package rules. The high-level split is:

What Renovate does on its own	What needs a human
Opens PRs for every dependency it understands (Go, Docker, GitHub Actions, npm).	Merging native-manager PRs — Go/Docker/Actions/npm bumps are opened but not auto-merged.
Auto-merges the custom-regex–managed pins and the `gateway-helm` and `headlamp` chart floors (patch/minor) after a 3-day cooldown, see below.	Reviewing major-version bumps.
Maintains Docker image digests (`@sha256:…`) alongside floating tags.	Reviewing anything touching coupled stacks (k8s).
Pins GitHub Actions to commit SHAs (annotated with a `# vX.Y` tag).	Updating the Go workspace / `go.mod` directives; triaging CVEs.

Renovate combines the config:recommended native managers (Go modules, GitHub Actions, npm), the native flux manager over the Flux manifests, and the custom regex managers (jq '.customManagers | length' renovate.json counts them) that track pins those native managers cannot see. Auto-merge (patch/minor, 3-day cooldown) is wired onto the custom-regex managers and, among the native managers, only onto the two floor-tracked Flux charts, gateway-helm and headlamp (see Flux HelmRelease chart versions). Every other native-manager PR waits for a human merge (config:recommended does not auto-merge, and no top-level automerge is set).

Renovate performs the merge itself: platformAutomerge is set to false at the top level, so an automergeable PR never goes to GitHub's own auto-merge. Renovate merges it once every check on the PR head has succeeded, because ignoreTests is left unset. Note for operators: main currently defines no required status checks, and GitHub's auto-merge waits for required checks alone. Tightening branch protection is a repository-settings change and cannot be made from files in this repository.

The custom managers cover:

OpenStack release refs — git tags in releases/*/source-refs.yaml and PyPI pins in releases/*/test-refs.yaml.
Test tooling in hack/ — Chainsaw, Flux CLI, kind, and kubectl versions in hack/install-test-deps.sh, plus FLUX_OPERATOR_VERSION and ENVOY_GATEWAY_VERSION (the Envoy Gateway CRD pin, grouped with the gateway-helm chart) in hack/deploy-infra.sh.
kind deploy components — flux-web.yaml under deploy/kind/base/.
K-ORC Flux source — the ref.commit of the K-ORC GitRepository in deploy/flux-system/sources/k-orc.yaml (git-refs on upstream main, digest updates). This closes a drift gap: without it the Flux-applied K-ORC CRDs could fall behind the k-orc/openstack-resource-controller Go module the operator compiles against. A commit bump is not complete on its own: the commit-<short sha> image tag and digest in deploy/flux-system/releases/k-orc.yaml and the Go pseudo-version in operators/c5c3/go.mod move in the same change, and the hack/ci-deploy-korc.sh drift guard fails CI until the image is re-pinned.
Go build tooling in Makefile / .github/workflows/*.yaml — gofumpt, controller-gen, golangci-lint, and yq, plus the envtest Kubernetes minor (ENVTEST_K8S_VERSION), which follows the envtest-vX.Y.Z releases of kubernetes-sigs/controller-tools that setup-envtest downloads its assets from.
renovate-config-validator pin — the RENOVATE_VALIDATOR_VERSION constant in tests/unit/renovate/, the Renovate release test-shell downloads and executes to validate renovate.json.
OVN image pin — the version and rev keys of images/ovn/source.json (github-tags on ovn-org/ovn, regex versioning because the 26.03 line carries a leading zero and a v prefix). One matchStrings entry spans both adjacent keys, so the tag and the commit it names move in a single PR; make nix-lock then writes the hashes and the OVS commit of the ovs gitlink (see Derived keys of the image pins). Digest updates are disabled, as for noVNC. A second manager tracks the same upstream tag in defaultOVNVersion, the constant in operators/ovn/internal/controller/image.go that the ovn-operator resolves for a CR leaving spec.image unset. It carries the bare version, so its versioning regex expects no v and an extractVersionTemplate strips the one the tag has. Both pins are grouped under OVN LTS patch releases, so they move in a single PR; TestDefaultOVNVersionMatchesSourcePin fails when they diverge.
openstack-hypervisor-operator image pin — the rev key of images/openstack-hypervisor-operator/source.json (git-refs on upstream main of cobaltcore-dev/openstack-hypervisor-operator, digest updates, like the K-ORC source); make nix-lock then writes its hash and vendorHash. The rule runs on the weekly schedule (before 6am on monday) behind the 3-day cooldown and is not automerged. Every upstream main commit is a new digest, so without the schedule each one would open a PR. A move also needs a human: a downstream patch may no longer apply, and from #1142 on the lab's chart reference has to move with the pin. A failed patch phase in build-hvo means that patch has to be re-cut; see Re-cutting the openstack-hypervisor-operator patch.
kvm-node-agent image pin — the rev key of images/kvm-node-agent/source.json (git-refs on upstream main of cobaltcore-dev/kvm-node-agent, digest updates), with the same weekly schedule, 3-day cooldown, make nix-lock step and no automerge as the hvo pin. The lab's chart reference, ref.tag and ref.digest of the kvm-node-agent OCIRepository in deploy/lab/metal-stack/hypervisor/sources.yaml, has no manager of its own and moves with the pin in the same PR; tests/unit/deploy/metal_stack_hypervisor_test.sh fails while its short SHA differs from the pin. A failed patch phase in build-kna means the patch has to be re-cut; see Re-cutting the kvm-node-agent patch.
noVNC console assets — the version and rev keys of images/nova/novnc.json (github-tags on novnc/noVNC, regex versioning because the tags carry a v prefix). One matchStrings entry spans both adjacent keys, so the tag and the commit it names move in a single PR; make nix-lock then writes the hash of the new commit (see Nix locks). Majors are disabled; minors and patches wait the 3-day cooldown and are not automerged, because the console page is user-facing and no e2e suite loads it before #1018. Digest updates are disabled: a tag moved upstream to another commit is not a release, and the pin stays on the reviewed commit.
mod_auth_openidc of the federation proxy — the version and rev keys of images/keystone-federation-proxy/source.json (github-tags on OpenIDC/mod_auth_openidc, regex versioning because the tags carry a v prefix and a fourth component). Renovate reads the fourth component as a build number, so every 2.4.x.y release is a patch. One matchStrings entry spans both adjacent keys, so the tag and the commit it names move in a single PR; make nix-lock then writes the hash of the new commit (see Derived keys of the image pins). Majors are disabled; minors and patches wait the 3-day cooldown in the group mod_auth_openidc releases and are not automerged. The module authenticates every federated login in front of Keystone, and a reviewer reads upstream's ChangeLog: 2.4.20, for one, changed the key derivation of the session cookies. Digest updates are disabled, as for noVNC. The PR rebuilds the image through the proxy filter of the build-images workflow and runs the keystone e2e-operator job through the image_proxy filter of the CI workflow.
Lab libvirt image — the eight ghcr.io/c5c3/libvirt:<tag>@sha256:<digest> image lines of the metal-stack lab in deploy/lab/metal-stack/hypervisor/libvirt-daemonset.yaml, probe/nfs-module-load.yaml, nfs/client-modules-daemonset.yaml and chaos-mesh/modules-daemonset.yaml (docker datasource, deb versioning). The tag is a keeper tag. hack/ci-tag-libvirt-keeper.sh creates <libvirt-version>-r<N>, such as 12.2.0-r57, once on main for the Nix-built image; N counts the commits under images/libvirt, nix/pkgs/libvirt.nix, nix/lib/image-defaults.nix and flake.lock, so a nixpkgs-images re-lock that rebuilds QEMU while libvirt keeps its version still gets a new tag. The <libvirt-package-version>-r<N> keepers of the Docker-built image, such as 10.0.0-2ubuntu8.19-r2, stay in GHCR as the lab's rollback target. deb reads 12.2.0-r57 as upstream 12.2.0 and revision r57 and compares both; docker versioning would read -r57 as a compatibility suffix and never offer another. deb also parses latest and a commit SHA and sorts both above 12.x, so allowedVersions admits the Nix keeper shape alone, which also keeps out the Docker-built keepers. Every update type, majors included, moves all eight lines in one PR (metal-stack lab libvirt image): a pin on a Docker-built keeper moves to the newest Nix keeper as a major (10 to 12), and a libvirt major that a nixpkgs-images re-lock brings is one too. The lab never pins a branch build, because cleanup-images.yaml removes a branch's <sha> tag after 24 hours, so a pull request that changes the image leaves the pins alone and Renovate proposes the keeper main mints after the merge. The PR has no cooldown, since this repository's own main builds the image from a reviewed commit. It is not automerged: the image runs privileged as root on the lab's nodes, and the reviewed bump, rolled node by node as libvirt 12 on the lab describes, is what the pin is for.
Lab Ceph image — the quay.io/ceph/ceph:<tag>@sha256:<digest> image lines of the lab Ceph in deploy/lab/metal-stack/ceph/cluster/cluster.yaml, the CephCluster, and toolbox.yaml, the toolbox that runs the same image (docker datasource, docker versioning). Majors are disabled. Minor, patch and digest updates move both lines in one PR (ceph image) after the 3-day cooldown and are not automerged: a Ceph minor is a question of the versions the pinned Rook operator supports. tests/unit/renovate/ceph_image_custommanager_test.sh pins the manager and both rules.
Lab soak runner image — the docker.io/alpine/k8s:<tag>@sha256:<digest> image line of the dizzy soak's runner in deploy/lab/metal-stack/dizzy-soak/job.yaml (docker datasource, docker versioning). The tag is the kubectl release, and it stays on the lab's Kubernetes minor, so major and minor updates are disabled. Patch and digest updates are automerged after the 3-day cooldown in the group dizzy soak runner image; tests/unit/renovate/dizzy_soak_runner_image_custommanager_test.sh pins the manager and both rules. The dizzy image of the same Job carries no tag of its own: hack/dizzy-soak.sh start takes it from the one DIZZY_VERSION pin in hack/dizzy.sh.

Major updates are disabled for all custom-regex managers — these touch deploy-time CRDs, the OpenStack release matrix, and build tooling where a major bump always needs human-driven coordination. The renovate-config-validator pin is one exception: it has to follow the major the hosted bot runs, or the validator silently stops checking renovate.json against the schema that bot enforces. That pin is never automerged either — the validator run is the only check that exercises the new release, so a bump must not be able to approve itself into a test-shell job that runs on main and on v* tag pushes. The lab libvirt image is the other exception: main mints every keeper it offers, and a pin left on an older major could only move by a digest copied by hand. The lab soak runner image disables minor updates too, because its tag follows the lab's Kubernetes minor. The OVN pin disables minor updates as well, because the image is held on the 26.03 LTS line: 26.09 is a minor bump and 27.03 a major one, and both leave that line. Patch bumps there wait the 3-day cooldown but are not automerged, for the reason below. build-images.yaml runs on every pull_request touching images/**, builds the amd64 image and runs tests/container-images/verify_ovn.sh against it, so the reviewer merges on a green build.

Go module PRs run two post-update options. gomodTidy runs go mod tidy in the module whose go.mod Renovate updates. gomodTidyAll then runs it in every module that points at that module through a local replace directive, in dependency order: every operator module replaces internal/common, and operators/c5c3 also replaces the service operators. A bump in internal/common that raises an indirect requirement of the operators therefore arrives tidy in every module. Both options stay listed because the validator does not check postUpdateOptions values: a hosted bot too old for gomodTidyAll would silently skip it, and gomodTidy still runs. go.work.sum is not tracked (see .gitignore). The go command appends a checksum to it whenever a workspace build needs one that no member go.sum holds, so its content depends on which command ran, and Renovate's gomod manager writes it only when it vendors.

Separately, the native nix manager keeps the root flake fresh: it maintains flake.lock via lock-file maintenance. The lock pins five inputs: nixpkgs (nixos-unstable) feeds the dev shell, nixpkgs-images (nixos-26.05) feeds the Nix-built container images (Nix Image Build), and pyproject-nix, uv2nix and pyproject-build-systems build the Python environments of those images. The nix flake inputs package rule re-locks all five on the Monday schedule of the uv bump and opens one grouped PR. That PR is not automerged: the refs are rolling branches, so the re-lock is not a version bump and the 3-day cooldown cannot gate it — a reviewer confirms the base-runtime bump instead. A re-lock of nixpkgs-images rebuilds the Nix-built images on its pull request. The Nix devshell re-reads the canonical tool pins (ci.yaml, the Makefile, hack/install-test-deps.sh) on entry, so a tool-version bump needs no flake edit — see Nix Development Environment. The spike under nix/spike/ is outside every manager: a packageRules entry with matchFileNames: ["nix/spike/**"] sets enabled: false, so its flake.lock and the generated locks under nix/spike/locks/ are pinned by hand and re-locked by whoever edits them.

Nix locks

The locks under nix/locks/<release>/<name>/, one per service and release plus one per release for Tempest, are derived from five kinds of pin: releases/<release>/source-refs.yaml and test-refs.yaml, releases/<release>/upper-constraints.txt and extra-packages.yaml, overrides/<release>/constraints.txt, the patches under patches/, and nix/base-requirements.txt. hack/nix-lock.sh writes them, and with them the noVNC hash in images/nova/novnc.json and the derived keys of the four image pins images/<name>/source.json. nix/locks/** in ignorePaths keeps every manager from reading the derived files, and a packageRules entry with matchFileNames: ["nix/locks/**"] disables any update to them. A Renovate PR that moves one of those pins, the noVNC pin or an image pin, therefore fails the nix-lock-check job of the build-images workflow and cannot automerge until the locks follow. The fix is one command on the PR branch, committed and pushed:

bash

make nix-lock

It needs Nix with flakes and runs hack/nix-lock.sh --all in the lock-tools dev shell, whose uv wrote the committed locks (Python environments and locks).

Flux HelmRelease chart versions

Renovate's native flux manager reads the Flux manifests through the three flux.managerFilePatterns entries in renovate.json:

/^deploy/flux-system/(releases|sources)/.+\.yaml$/
/^deploy/kind/.+\.yaml$/
/^deploy/lab/metal-stack/ceph/(release|source)\.yaml$/

The manager resolves the sourceRef of a HelmRelease against the HelmRepository manifests it has parsed, so the sources directory has to match as well: deploy/kind/prometheus/release.yaml references the prometheus-community HelmRepository in deploy/flux-system/sources/. Under deploy/lab/ only the HelmRelease and the HelmRepository of the lab Ceph match, because that overlay has no kind sibling to inherit; nothing else under deploy/lab/ or deploy/examples/ does. The patterns use no regex lookahead, because the hosted bot compiles them with RE2, which has none.

It tracks seventeen third-party charts and one image:

Chart	File	Strategy	Automerge
`cert-manager`	`deploy/flux-system/releases/cert-manager.yaml`	`widen`	no
`external-secrets`	`deploy/flux-system/releases/external-secrets.yaml`	`widen`	no
`garage-operator`	`deploy/flux-system/releases/garage-operator.yaml`	`widen`	no
`mariadb-operator`	`deploy/flux-system/releases/mariadb-operator.yaml`	`widen`, group `mariadb-operator chart`	no
`mariadb-operator-crds`	`deploy/flux-system/releases/mariadb-operator-crds.yaml`	`widen`, group `mariadb-operator chart`	no
`openbao`	`deploy/flux-system/releases/openbao.yaml`	`widen`	no
`prometheus-operator-crds`	`deploy/flux-system/releases/prometheus-operator-crds.yaml`	`widen`, group `prometheus-operator charts`	no
`chaos-mesh`	`deploy/kind/chaos-mesh/release.yaml`	`widen`	no
`grafana`	`deploy/kind/dizzy/release-grafana.yaml`	`widen`	no
`victoria-metrics-single`	`deploy/kind/dizzy/release-victoria-metrics.yaml`	`widen`	no
`metrics-server`	`deploy/kind/metrics-server/release.yaml`	`widen`	no
`kube-prometheus-stack`	`deploy/kind/prometheus/release.yaml`	`widen`, group `prometheus-operator charts`	no
`vertical-pod-autoscaler`	`deploy/kind/vpa/release.yaml`	`widen`	no
`gateway-helm`	`deploy/kind/base/envoy-gateway.yaml`	`bump`, group `envoy-gateway`	minor/patch after 3 days
`headlamp`	`deploy/kind/base/headlamp.yaml`	`bump`, group `headlamp`	minor/patch after 3 days
`csi-driver-nfs`	`deploy/kind/nfs/release.yaml`	exact pin, majors disabled, group `csi-driver-nfs chart`	no
`rook-ceph`	`deploy/lab/metal-stack/ceph/release.yaml`	exact pin, majors disabled, group `rook-ceph chart`	no
image `ghcr.io/headlamp-k8s/headlamp-plugin-flux`	`deploy/kind/base/headlamp.yaml` (HelmRelease values)	tag	no

widen is the rangeStrategy of the base rule. A release inside the range changes nothing in Git: Flux installs it on its next reconcile and Renovate stays silent. A release past the ceiling opens a PR that raises the ceiling and keeps the floor, so >=0.10.0 <1.0.0 becomes >=0.10.0 <3.0.0 for external-secrets 2.x. Renovate's default strategy resolves to replace here, which writes a bare <3.0.0 and drops the floor. widen never moves the floor. gateway-helm and headlamp use bump for that: an in-range release rewrites the floor (>=1.9.0 <2.0.0 becomes >=1.9.2 <2.0.0), and a major rewrites both bounds (>=2.0.0 <3.0.0).

Two more rules make the manager work on the sources of this repository:

A chart from a type: oci HelmRepository (c5c3-charts, prometheus-community, garage-operator, envoy-gateway-charts) resolves through the docker datasource, whose docker versioning cannot read a range. Renovate then skips the chart with skipReason: invalid-value. A packageRule sets versioning: helm for a docker dependency of the flux manager whose current value starts with >=. The rule names no chart, so it covers a new OCI chart as long as its range is spelled >=X <Y; a ^X or ~X range is still skipped.
The base rule sets minimumReleaseAgeBehaviour: timestamp-optional. ghcr.io publishes no release timestamp, and under the default, timestamp-required, the 3-day cooldown holds every ghcr.io update as pending forever. Docker Hub and the HTTP chart indexes publish timestamps, so their cooldown still holds. None of the ghcr.io dependencies automerges.

Two rules switch the manager off. The c5c3-charts releases match ghcr.io/c5c3/charts/**, the url of deploy/flux-system/sources/c5c3-charts.yaml without oci://: this repository publishes those charts and sets their ranges itself. The other rule lists every file a customManager reads, plus the K-ORC release, whose image stays untracked and moves with the bump-korc-pin skill:

deploy/flux-system/sources/k-orc.yaml
deploy/flux-system/sources/openbao-operator.yaml
deploy/flux-system/sources/rabbitmq-cluster-operator.yaml
deploy/flux-system/releases/k-orc.yaml
deploy/flux-system/releases/rabbitmq-cluster-operator.yaml
deploy/kind/base/flux-web.yaml
deploy/kind/infrastructure/openbao-instance.yaml
deploy/kind/nfs/nfs-server.yaml

Every pin has one owner. A customManager moves a tag together with its digest or commit in one match, which the flux manager does not do for these pairs, so a file that gets a customManager joins this list. tests/unit/renovate/flux_helmrelease_manager_test.sh fails for a file both managers read, and R8 of the check-renovate-coverage audit reports it as claimed twice.

A gateway-helm or headlamp major opens a PR that is not automerged. The ENVOY_GATEWAY_VERSION CRD pin shares the envoy-gateway group with the chart, but its majors stay disabled like every custom-regex major. The chart comes from Docker Hub and the pin from GitHub releases, so the group holds both only once both releases are visible and past the cooldown. The pin can automerge alone while it stays inside the chart range. A chart PR whose new range leaves the pin behind (a floor bump that arrives first, or a major) fails tests/unit/renovate/envoy_gateway_manager_test.sh until the pin is inside the range.

A PR that raises a ceiling lets Flux install every release up to the new one. Before merging, the reviewer handles the coupling the e2e legs do not see:

victoria-metrics-single (deploy/kind/dizzy/release-victoria-metrics.yaml) and vertical-pod-autoscaler (deploy/kind/vpa/release.yaml) hold a one-minor window on purpose. widen keeps the old floor, so lift the floor by hand on the PR branch and keep the window one minor wide.
A gateway-helm major needs ENVOY_GATEWAY_VERSION in hack/deploy-infra.sh moved by hand on the same branch; the test above stays red until it is.
A mariadb-operator major moves the github.com/mariadb-operator/mariadb-operator Go module (pinned in internal/common/go.mod and the operator modules) along with the CRDs the chart installs.
prometheus-operator-crds owns the CRDs kube-prometheus-stack runs against (deploy/kind/prometheus/release.yaml sets crds.enabled=false), so the two share the prometheus-operator charts group. A PR that raises only the kube-prometheus-stack ceiling needs a check that the CRDs below the prometheus-operator-crds ceiling carry what the new operator expects.

To see what the manager proposes, run Renovate against a copy of deploy/. The npx install of Renovate lacks the optional re2 module and then rejects every matchStrings entry that uses (?m). The flux manager needs no customManager, so the copy drops them:

bash

tmp=$(mktemp -d) && cp -R deploy "$tmp/deploy"
jq 'del(.customManagers) | .enabledManagers = ["flux"]' renovate.json > "$tmp/renovate.json"
git -C "$tmp" init -q && git -C "$tmp" add -A && git -C "$tmp" -c user.name=x -c user.email=x@x commit -qm x
ver=$(sed -nE 's/^RENOVATE_VALIDATOR_VERSION="([^"]+)"/\1/p' tests/unit/renovate/fluxoperator_custommanager_test.sh)
(cd "$tmp" && npx --yes --package node@24 --package "renovate@${ver}" -- \
  renovate --platform=local --report-type=file --report-path="$tmp/report.json")
jq -r '.repositories[].packageFiles.flux[] | .packageFile as $f | .deps[]
  | [$f, .depName, (.currentValue // .currentDigest), (.skipReason // "tracked"),
     ((.updates // []) | map(.updateType + " " + (.newValue // .newDigest)) | join(" | "))] | @tsv' "$tmp/report.json"

$ver is the RENOVATE_VALIDATOR_VERSION that tests/unit/renovate/fluxoperator_custommanager_test.sh pins; that release refuses Node 25, hence --package node@24. Each row names the file, the dependency, its current value, tracked or the skip reason, and any proposed update. The run has two limits. --platform=local stops after the lookup and builds no branches, so the rules scoped by matchUpdateTypes (the csi-driver-nfs major, the two automerge rules) do not show in the report; flux_helmrelease_manager_test.sh covers them. And bump proposes nothing while a floor equals the newest release: lower the floor in the copy, for example to >=1.9.0 <2.0.0 in deploy/kind/base/envoy-gateway.yaml, to see >=1.9.2 <2.0.0 proposed for the renovate/envoy-gateway branch.

Derived keys of the image pins

Renovate moves the keys of the four image pins a datasource can name: version and rev of images/ovn/source.json and images/keystone-federation-proxy/source.json, and rev of images/openstack-hypervisor-operator/source.json and images/kvm-node-agent/source.json. make nix-lock writes the rest:

Pin	Derived keys
`images/ovn/source.json`	`hash`, the NAR hash of the OVN tree at `rev`; `ovs.rev`, the commit the `ovs` submodule gitlink names at `rev`; `ovs.hash`, the NAR hash of that OVS tree
`images/keystone-federation-proxy/source.json`	`hash`, the NAR hash of the mod_auth_openidc tree at `rev`
`images/openstack-hypervisor-operator/source.json`, `images/kvm-node-agent/source.json`	`hash`, the NAR hash of the tree at `rev`; `vendorHash`, the hash of its Go module download

No Renovate datasource can express "the ovs gitlink at tag X" or a NAR hash, so a Renovate PR on one of these pins fails nix-lock-check until someone runs make nix-lock on its branch and pushes the result. hack/nix-lock.sh reads the gitlink from a shallow, blobless fetch of the pinned OVN commit and recomputes a vendorHash only when hash moved or vendorHash is empty. The build fetches each tree by commit and hash, never through the tag, so a moved tag or a changed gitlink cannot change the image without a change in this repository. That step is also why the OVN patch rule does not automerge: an automerged PR would leave nobody to run it, and the group would stall red until someone noticed OVN security patches had stopped landing (Source-built images).

The noVNC pin images/nova/novnc.json has the {version, rev, hash} shape of the proxy pin and follows the same flow: Renovate moves version and rev, and make nix-lock writes hash.

operators/<op>/vendor.json is derived from the Go modules of the operator: inputsHash covers the go.mod and go.sum of internal/common, the operator and the siblings its go.mod replaces, and vendorHash is the hash of their download. No manager reads the file. A Renovate gomod PR fails the verify-codegen job (make nix-lock-check-operators) until someone runs make nix-lock on its branch and pushes the result (Operator images).

Re-cutting the openstack-hypervisor-operator patch

The four patches under images/openstack-hypervisor-operator/patches/ are cut against the pinned upstream commit, each as one commit on top of the ones before:

Patch	Subject	Test command
`0001-eviction-let-nova-choose-block-migration.patch`	`Eviction: let Nova choose block migration`	`go test -count=1 -run '^TestLiveMigrateAutoBody$' ./internal/controller/eviction/`
`0002-hypervisor-make-the-high-availability-default-configurable.patch`	`Hypervisor: make the default of spec.highAvailability configurable`	`go test -count=1 -run '^TestHypervisorCreatedWithDefaultHighAvailability$' ./internal/controller/`
`0003-traits-report-traitsupdated-when-nothing-differs.patch`	`Traits: report TraitsUpdated when no custom trait differs`	`go test -count=1 -run '^TestTraitsInSyncSetsTraitsUpdated$' ./internal/controller/`
`0004-openstack-select-the-catalog-interface-with-os-interface.patch`	`openstack: select the catalog interface with OS_INTERFACE`	`go test -count=1 -run '^TestServiceClientInterface$' ./internal/openstack/`

When a Renovate PR moves the rev of images/openstack-hypervisor-operator/source.json to a commit on which one of them no longer applies, build-hvo fails in the patch phase of the Nix build and nothing is published. If upstream now carries a patch's change, delete that patch on the Renovate branch together with its checks in images/openstack-hypervisor-operator/patch-checks.sh: for 0001 check_live_migrate_opts and the TestLiveMigrateAutoBody run, for 0002 and 0003 the test's name in the controller go test run and its require_pass, for 0004 the TestServiceClientInterface run and its require_pass. Each test passes only while upstream ships it. With no patch left, patches/ can go too. Otherwise re-cut the patches in order in a scratch clone that holds both the old and the new commit, so a three-way apply finds the blobs each patch was cut from:

bash

old=<commit before the move>; new=<commit the Renovate PR pins>
git init /tmp/hvo && cd /tmp/hvo
git remote add origin https://github.com/cobaltcore-dev/openstack-hypervisor-operator.git
git fetch --depth 1 origin "$old" "$new" && git checkout --detach "$new"
git apply --3way <repo>/images/openstack-hypervisor-operator/patches/0001-*.patch
# resolve the conflicts, then prove the test still passes
go test -count=1 -run '^TestLiveMigrateAutoBody$' ./internal/controller/eviction/
git commit -am "Eviction: let Nova choose block migration"
# the same for 0002, 0003 and 0004, each with its test command and subject from the table
git format-patch -4 --no-signature -o /tmp/hvo-patches

git am -3 does not take the checked-in files: each carries the house header instead of a mail envelope, and git am stops at the missing author. Turn each git format-patch output back into the house header (the SPDX pair, the bare subject, the rationale, Applies-to: naming the new commit, Upstream status:, and no diffstat; see patches/cinder/2026.1/0001-nfs-run-qemu-img-info-as-the-service-user.patch for the form), run make nix-lock for hash and vendorHash, commit the files onto the Renovate branch and let build-hvo prove them.

Re-cutting the kvm-node-agent patch

images/kvm-node-agent/patches/0001-certificates-restrict-private-key-file-modes.patch is cut against the pinned upstream commit, and the recipe is the one above with the kna names. When a Renovate PR moves the rev of images/kvm-node-agent/source.json to a commit on which the patch no longer applies, build-kna fails in the patch phase of the Nix build and nothing is published. If upstream now carries the change, delete the patch on the Renovate branch together with its two test runs in images/kvm-node-agent/patch-checks.sh, and with the last patch patches/. Otherwise re-cut it in a scratch clone that holds both commits:

bash

old=<commit before the move>; new=<commit the Renovate PR pins>
git init /tmp/kna && cd /tmp/kna
git remote add origin https://github.com/cobaltcore-dev/kvm-node-agent.git
git fetch --depth 1 origin "$old" "$new" && git checkout --detach "$new"
git apply --3way <repo>/images/kvm-node-agent/patches/0001-*.patch
# resolve the conflicts, then prove both tests still pass
go test -count=1 -run '^TestUpdateTLSCertificateKeyMode$' ./internal/certificates/
go test -count=1 -run '^TestUpdateTLSCertificateKeyGroupNotPermitted$' ./internal/certificates/
git commit -am "Certificates: restrict the mode of private key files"
git format-patch -1 --no-signature --stdout > /tmp/0001.patch

Run the second test as a user other than root: it skips as root, because a root process may give a file any group. The image build runs both tests as a Nix build user and fails unless both print their --- PASS: line. Turn the git format-patch output into the house header as for the hvo patch (the SPDX pair, the bare subject, the rationale, Applies-to: naming the new commit, Upstream status:, and no diffstat), move the chart ref in deploy/lab/metal-stack/hypervisor/sources.yaml to the chart of the new commit, run make nix-lock, commit the result onto the Renovate branch and let build-kna prove the patch.

Go version upgrades

Cadence and support window

The Go team ships a new minor release roughly every six months and supports the two most recent minor versions. We track upstream:

Stay on a supported minor at all times.
Pick up patch releases (1.X.Y → 1.X.(Y+1)) within the regular Renovate flow — these are low-risk and grouped with other patch bumps.
Plan minor upgrades (1.X → 1.(X+1)) within roughly one month of GA so the deprecation window before the previous minor falls out of support is comfortable.

Where the Go version lives

A minor upgrade must touch every location in the same PR — partial bumps fail CI because go.work and the per-module go.mod directives must agree.

File	What to update	Notes
`go.work`	`go 1.X.Y`	Single source of truth used by `actions/setup-go` in CI.
`internal/common/go.mod`	`go 1.X.Y`	Shared library module.
`operators/keystone/go.mod`	`go 1.X.Y`	Keystone operator module.
`operators/c5c3/go.mod`	`go 1.X.Y`	C5C3 ControlPlane orchestrator module.
`flake.lock`	No change for a patch bump.	The operator images build with `go_1_27` of the locked `nixpkgs-images`; `nix eval --raw .#packages.x86_64-linux.keystone-operator-image.manager.go.version` prints its version.
`.github/workflows/ci.yaml`	No change.	All `actions/setup-go` steps use `go-version-file: go.work`.

There is no toolchain directive anywhere in the workspace. We rely on the directive in go.mod/go.work plus the toolchain that ships with the CI runner and the go_1_27 the operator images build with. If the team ever needs to support contributors on an older local Go, add a toolchain go1.X.Y line to go.work rather than to individual go.mod files.

Worked example: 1.25.10 → 1.26.3

An illustrative past minor upgrade (the workspace has since moved on — at the time of writing all four files are on go 1.27.2). Substitute the current and target versions for your own bump; the mechanics below are unchanged.

bash

# 1. Bump the version in all four files. Each file has exactly one
#    `go 1.25.10` line; replace with `go 1.26.3`.
sed -i.bak 's/^go 1\.25\.10$/go 1.26.3/' \
  go.work \
  internal/common/go.mod \
  operators/keystone/go.mod \
  operators/c5c3/go.mod
rm -f go.work.bak internal/common/go.mod.bak operators/*/go.mod.bak

# 2. Resync the workspace. This refreshes the indirect requirement lists
#    in each `go.mod`.
go work sync

# 3. Build every module from its own directory (the workspace cannot be
#    built from the repository root because the modules are independent).
(cd internal/common         && go build ./... && go vet ./...)
(cd operators/keystone      && go build ./... && go vet ./...)
(cd operators/c5c3          && go build ./... && go vet ./...)

# 4. Run unit tests for all three modules — each has its own suite.
(cd internal/common    && go test -short -timeout 5m ./...)
(cd operators/keystone && go test -short -timeout 5m ./...)
(cd operators/c5c3     && go test -short -timeout 5m ./...)

The operator images build with go_1_27 of the locked nixpkgs-images, and buildGoModule sets GOTOOLCHAIN=local. Before opening the bump PR, confirm that the nix eval of the table above prints a version at or above the new directive. Otherwise checks.<system>.go-directive fails nix flake check in nix-flake-check, and every operator image build fails; wait for the Monday re-lock of flake.lock to bring the channel's newer Go. A minor bump also moves buildGo127Module in nix/lib/operator-image.nix and the go_1_27 of the go-directive check in flake.nix to the new minor.

Commit messages follow the repository convention (English, no Co-Authored-By, SAP AI-assisted / On-behalf-of / Signed-off-by trailers).

Verification

Required CI checks gate the upgrade:

lint — make lint via golangci-lint. New analyzers may light up after a minor bump; fix in the same PR or temporarily disable the offending linter in .golangci.yml with a TODO referencing the upgrade PR.
format-check — gofumpt -l must be silent.
test — per-module go test ./....
govulncheck — runs against the new toolchain.
e2e-operator (keystone) / e2e-operator (c5c3), e2e-infra, e2e-prometheus, e2e-chaos — Chainsaw suites against a kind cluster (e2e runs as a per-operator matrix); these catch runtime regressions that pure Go-level tests miss.

For local pre-flight, the minimal smoke is go work sync && (cd internal/common && go test -short -timeout 5m ./...).

Deprecated APIs and new vet checks

Each Go minor release adds analyzers and may flip previously-warned APIs to errors. After go vet ./... runs clean from each module, check the release notes for:

New vet analyzers. A minor bump may add an analyzer that triggers on patterns the codebase has tolerated for years. If the warning is load-bearing, fix it; if it is noise on the path the project has chosen, add a targeted exclusion in .golangci.yml.
stdlib deprecations. Search for newly-deprecated stdlib symbols (grep -rn 'pkg.OldFunc\|pkg.OldType' --include='*.go') and replace them. Avoid making this part of the upgrade PR if the replacement is invasive — open a follow-up.
Toolchain selection changes. go.work and go.mod only set the minimum required Go. They do not pin the toolchain that builds the binary. Production builds use the go_1_27 of nixpkgs-images, with GOTOOLCHAIN=local.

Rollback

If a minor upgrade lands on main and a regression surfaces in CI or production:

Revert the single upgrade commit (git revert <sha>); the revert is self-contained because all four files moved together.
If the regression is from a new vet analyzer, prefer adding the analyzer exclusion to .golangci.yml and rolling forward — revert is cheap, but the deprecation window of the previous minor will eventually force the upgrade.
Record the regression in the upgrade issue with a reproducer; do not re-attempt the minor bump until the regression has an upstream fix or a local workaround.

Library and dependency updates

Classification

Type	Renovate behaviour	Reviewer depth
Patch	Grouped into a PR; not auto-merged — a human merges.	Skim the diff for unexpected indirect bumps, then merge once green.
Minor	Grouped into a PR; not auto-merged.	Confirm the CHANGELOG mentions no behavior change in modules we depend on, then merge.
Major	Always opened, never auto-merged.	Read full upstream release notes; check for migrations; run e2e locally.

This is the behaviour for the native managers (Go modules, Docker, Actions, npm, Flux). The custom-regex managers (OpenStack tags, the hack/ test tooling, kind deploy components, Go build tooling) and the two floor-tracked Flux charts, gateway-helm and headlamp, are the only ones that do auto-merge: patch/minor auto-merge after the 3-day cooldown. Majors are disabled for the custom-regex managers; a gateway-helm or headlamp major opens a PR that a human merges.

Coupled stacks (k8s)

Kubernetes Go packages are versioned in lockstep. Never bump just one of:

k8s.io/api
k8s.io/apimachinery
k8s.io/client-go
k8s.io/apiextensions-apiserver
k8s.io/component-base

…without bumping the others to the same v0.X.Y. Renovate groups these under a single PR, but if you cherry-pick from a Renovate PR or open a manual bump, keep them aligned. The same applies to sigs.k8s.io/controller-runtime — it has a hard compatibility matrix with the k8s.io/* packages; consult the controller-runtime release notes for the supported pairing before bumping either side.

OpenBao / Vault clients

github.com/openbao/openbao/api is API-compatible with github.com/hashicorp/vault/api but the modules version independently. When OpenBao publishes a new client release:

Patch/minor: handle via Renovate.
Major: open a dedicated PR that also re-runs tests/e2e/keystone/openbao-* Chainsaw suites and reviews internal/common/bootstrap/ for any signature changes.

Kubebuilder / operator-sdk markers

Marker generation (controller-gen) is pinned via CONTROLLER_GEN_VERSION in .github/workflows/ci.yaml. A bump to controller-gen may change CRD generation output (e.g. new schema fields for +kubebuilder:validation:* markers). Treat this as a major-style review:

Bump the env var in ci.yaml.
Run make manifests locally and inspect the diff in operators/keystone/config/crd/.
Confirm no CRD field is silently removed (would break running clusters on upgrade).

Container image digests

No Dockerfile is left in the repository; the flake builds every image. An image reference Renovate maintains keeps the floating tag and the SHA digest together, as the docker://nixos/nix:<tag>@sha256:… container of the build-e2e-images job does: the tag is human-readable, the digest is the immutable pin, and both move together in Renovate PRs. Manual edits should preserve this shape and never drop the digest.

GitHub Actions pinned by SHA

Policy: every uses: reference pins to a commit SHA, annotated with the human-readable tag for review:

yaml

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6

The tag in the trailing comment is for humans. The SHA is what GitHub resolves. Renovate keeps both in sync. Never replace a SHA with a floating tag — supply-chain integrity for third-party actions depends on the SHA pin.

Security updates (CVEs)

CVE-driven bumps bypass the 3-day Renovate cooldown — merge as soon as CI is green. Triage order:

CVEs in our direct dependencies (anything listed in a go.mod's direct require ( … ) block) — top priority. Renovate flags these with a high-priority label.
CVEs in indirect dependencies that affect us — confirm via govulncheck (run automatically in CI) before deciding urgency.
CVEs in base images — bump the image digest and re-trigger the image-build pipeline.

If govulncheck reports a vulnerable indirect dependency that has no upstream fix yet, document it in the corresponding issue and add a replace directive in go.mod only as a last resort.

Worked example: a library major upgrade

Major bumps always need human review. Take a hypothetical controller-runtime v0.23 → v1.0 PR:

Read the upstream release notes in full. controller-runtime typically gates Kubernetes API compatibility on its minor versions — confirm the supported k8s.io pairing matches the version we ship.

Confirm the API surface we touch. Search the codebase:

bash

grep -rn 'controller-runtime\|sigs.k8s.io/controller-runtime' \
  internal/common operators

Run the upgrade locally, mirroring what Renovate would do:

bash

(cd operators/keystone && go get sigs.k8s.io/controller-runtime@v1.0.0 && go mod tidy)
(cd operators/c5c3      && go get sigs.k8s.io/controller-runtime@v1.0.0 && go mod tidy)
(cd internal/common     && go get sigs.k8s.io/controller-runtime@v1.0.0 && go mod tidy)
go work sync

Compile and vet from each module. Compile errors are the easy feedback; for behavior changes, read the controller-runtime CHANGELOG for the entries between the two versions.
Run unit tests and the keystone e2e suite locally (make test, make e2e) before pushing — major operator-framework bumps regularly surface only at reconcile-time, not at compile-time.
Open a dedicated PR, not a Renovate edit. Link the upstream release notes, the list of API surfaces touched, and the e2e run evidence.

Process

Renovate PRs vs. dedicated PRs

Situation	Channel
Routine patch/minor bump (Go module, Docker digest, GitHub Action).	Renovate PR.
Major bump for a leaf dependency with a small API surface.	Renovate PR + review.
Major bump for `controller-runtime`, `client-go`, `k8s.io/*`, OpenBao.	Dedicated PR.
Go minor upgrade.	Dedicated PR + issue.
CVE-driven bump.	Renovate PR, fast-merge.
Anything that requires CRD or Helm chart migration.	Dedicated PR + issue.

A dedicated PR is preferred whenever the change requires a write-up that does not fit in a Renovate commit message: migrations, deprecation follow-ups, e2e-suite changes, or a coordinated bump across multiple modules.

Required checks before merge

Renovate performs the merge itself (platformAutomerge is disabled) and waits until every check on the PR has succeeded. The checks below are the ones to watch when merging a dependency PR by hand:

lint, format-check, govulncheck
test (all three modules)
e2e-operator (keystone) / e2e-operator (c5c3), e2e-infra (and e2e-prometheus, e2e-chaos when their path filters trigger)
helm-validate (when chart files change)
build-e2e-images (when image or workflow files change)

Renovate's gate covers every check that runs on the PR. A path filter can keep a job from starting at all; a manual merge holds itself to the same bar and treats such a job as unverified, not as passed.

Keystone

Horizon

Glance

Placement

Barbican

OVN

Neutron

Cinder

Nova

Dependency Management

Renovate at a glance

Nix locks

Flux HelmRelease chart versions

Derived keys of the image pins

Re-cutting the openstack-hypervisor-operator patch

Re-cutting the kvm-node-agent patch

Go version upgrades

Cadence and support window

Where the Go version lives

Worked example: 1.25.10 → 1.26.3

Verification

Deprecated APIs and new vet checks

Rollback

Library and dependency updates

Classification

Coupled stacks (k8s)

OpenBao / Vault clients

Kubebuilder / operator-sdk markers

Container image digests

GitHub Actions pinned by SHA

Security updates (CVEs)

Worked example: a library major upgrade

Process

Renovate PRs vs. dedicated PRs

Required checks before merge

See also

Dependency Management ​

Renovate at a glance ​

Nix locks ​

Flux HelmRelease chart versions ​

Derived keys of the image pins ​

Re-cutting the openstack-hypervisor-operator patch ​

Re-cutting the kvm-node-agent patch ​

Go version upgrades ​

Cadence and support window ​

Where the Go version lives ​

Worked example: 1.25.10 → 1.26.3 ​

Verification ​

Deprecated APIs and new vet checks ​

Rollback ​

Library and dependency updates ​

Classification ​

Coupled stacks (k8s) ​

OpenBao / Vault clients ​

Kubebuilder / operator-sdk markers ​

Container image digests ​

GitHub Actions pinned by SHA ​

Security updates (CVEs) ​

Worked example: a library major upgrade ​

Process ​

Renovate PRs vs. dedicated PRs ​

Required checks before merge ​

See also ​

Dependency Management

Renovate at a glance

Nix locks

Flux HelmRelease chart versions

Derived keys of the image pins

Re-cutting the openstack-hypervisor-operator patch

Re-cutting the kvm-node-agent patch

Go version upgrades

Cadence and support window

Where the Go version lives

Worked example: 1.25.10 → 1.26.3

Verification

Deprecated APIs and new vet checks

Rollback

Library and dependency updates

Classification

Coupled stacks (k8s)

OpenBao / Vault clients

Kubebuilder / operator-sdk markers

Container image digests

GitHub Actions pinned by SHA

Security updates (CVEs)

Worked example: a library major upgrade

Process

Renovate PRs vs. dedicated PRs

Required checks before merge

See also