Keystone Operator Prometheus Metrics
Reference catalogue for every Prometheus collector the Keystone operator exposes on the controller-runtime metrics endpoint. Every metric name, label set, and histogram bucket list below is authoritative. The sub-reconciler duration and error metrics are verified by the tests in internal/common/instrumentation; the per-CR collectors by collectors_test.go; and dashboards/dashboard_test.go fails the build if the bundled Grafana dashboard references a metric the operator does not register.
The sub-reconciler duration/error pair (keystone_operator_reconcile_duration_seconds and keystone_operator_reconcile_errors_total) is shared with the other CobaltCore operators. It lives in the internal/common/instrumentation package and is exposed as the instrumenter declared beside the glue in internal/controller/instrumentation.go. Both the sub-reconciler vectors and the per-CR collectors (rotation age and db_sync) register on the controller-runtime registry once at operator startup through RegisterMetrics(), which main.go calls before wiring the controllers. Registration returns an error rather than panicking mid-reconcile, so a duplicate-registration fails startup cleanly. The per-CR collectors live in operators/keystone/internal/metrics/collectors.go and are registered by RegisterMetrics() via their Register() function. All collectors attach to the controller-runtime registry (sigs.k8s.io/controller-runtime/pkg/metrics) and are served on the operator's metrics listener at :8080/metrics by default; see How to enable the Keystone operator metrics endpoint for cluster-side wiring.
Metric summary
| Metric | Type | Labels | Purpose |
|---|---|---|---|
keystone_operator_reconcile_duration_seconds | Histogram | sub_reconciler | Latency of each sub-reconciler invocation |
keystone_operator_reconcile_errors_total | Counter | sub_reconciler, condition_type | Count of sub-reconciler failures per Ready sub-condition |
keystone_operator_key_rotation_age_seconds | Gauge | keystone, namespace, key_type | Age of the most recent successful Fernet/credential/admin-password rotation |
keystone_operator_db_sync_total | Counter | keystone, namespace, result | Terminal db_sync Job outcomes |
keystone_operator_db_sync_duration_seconds | Histogram | keystone, namespace | Wall-clock duration of terminated db_sync Jobs |
controller_runtime_reconcile_time_seconds | Histogram | controller | End-to-end Reconcile latency (externally provided by controller-runtime) |
The last row is not a keystone_operator_* metric: controller-runtime exports controller_runtime_reconcile_time_seconds out of the box for every controller, keyed by the controller label (keystone for this operator). Unlike keystone_operator_reconcile_duration_seconds, which samples only the sub-reconciler functions, this histogram wraps the whole Reconcile call — orchestration, the status update, and watch handling — so it is the signal to use for end-to-end reconcile-latency SLOs.
keystone_operator_reconcile_duration_seconds
Wall-clock duration of a single sub-reconciler invocation in seconds.
| Attribute | Value |
|---|---|
| Type | Histogram |
| Labels | sub_reconciler |
| Buckets (seconds) | 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 |
| Call site | instrumenter.Instrument in operators/keystone/internal/controller/instrumentation.go |
Label cardinality. The sub_reconciler value is drawn from a closed set of 16 strings — the keys of subReconcilerConditionTypes — so the series count is bounded regardless of how many Keystone CRs exist. The metric deliberately omits a keystone/namespace label to keep cardinality fleet-independent; per-CR attribution is available via logs.
Example PromQL. p95 per sub-reconciler over the last 5 minutes:
histogram_quantile(
0.95,
sum by (sub_reconciler, le) (
rate(keystone_operator_reconcile_duration_seconds_bucket[5m])
)
)keystone_operator_reconcile_errors_total
Count of sub-reconciler errors, partitioned by the offending sub-reconciler and the status.conditions[] type it failed to drive to True.
| Attribute | Value |
|---|---|
| Type | Counter |
| Labels | sub_reconciler, condition_type |
| Call site | instrumenter.Instrument (on non-nil error return) |
Label cardinality. Both labels are drawn from closed sets: the 16 sub-reconciler names and the 14 Ready sub-condition types listed in subConditionTypes. The sub_reconciler → condition_type mapping is one-to-one for most entries; the exceptions (Secrets, DBConnectionSecret, Config all map to SecretsReady) collapse into fewer distinct (sub_reconciler, condition_type) pairs in practice.
The counter is not pre-created; a (sub_reconciler, condition_type) series only appears after its first error, so a healthy fleet emits no series for this metric.
Example PromQL. Error rate per sub-reconciler over the last 5 minutes:
sum by (sub_reconciler) (
rate(keystone_operator_reconcile_errors_total[5m])
)keystone_operator_key_rotation_age_seconds
Age in seconds of the most recent successful key rotation for a given Keystone CR and key type, measured as time.Since(rotation-completed-at) at reconcile time.
| Attribute | Value |
|---|---|
| Type | Gauge |
| Labels | keystone, namespace, key_type |
| Call site | reconcileFernetKeys / reconcileCredentialKeys / reconcilePasswordRotation on every reconcile pass |
key_type values. fernet, credential, or admin-password.
admin-password caveat. For key_type="admin-password" the gauge measures age since the operator committed the rotated password to its push-source Secret — not since the password went live. The new password becomes live only after ESO mirrors it to OpenBao, the keystone-admin ExternalSecret syncs it back, and bootstrap re-runs (~1h+). The gauge therefore under-reports time-since-live for admin-password relative to fernet and credential, which go live the instant the operator updates the keys Secret.
Annotation source. The gauge reads the cobaltcore.c5c3.io/rotation-completed-at annotation from the production keys Secret first; applyRotationOutput stamps that annotation on the production Secret on every successful apply, so the timestamp is durable across the inter-rotation steady state and the gauge value (recomputed as time.Since(completedAt) on every reconcile) tracks wall-clock age correctly even after the staging Secret has been deleted. If the production Secret has no annotation yet — i.e. the very-first rotation has not been applied — the lookup falls back to the staging Secret to cover the post-CronJob-PATCH/pre-apply window.
When the gauge is NOT set. Both lookups skip the gauge on absent or malformed cobaltcore.c5c3.io/rotation-completed-at annotations: a missing annotation on both Secrets means rotation has never completed (alert via PromQL absent(...)); a malformed annotation is a script bug surfaced via the RotationAnnotationInvalid event. Validation rejections of a staging payload are tracked via the RotationRejected event (see Keystone Controller Events) and indirectly via keystone_operator_reconcile_errors_total{condition_type="FernetKeysReady"}.
Series lifecycle. When a Keystone CR is deleted, every rotation-age series tagged with that (keystone, namespace) pair is dropped by DeleteForKeystone in reconcileDelete, preventing orphan series from accumulating over the CR lifetime.
Example PromQL. Alert if any rotation is older than 14 days:
max by (keystone, namespace, key_type) (
keystone_operator_key_rotation_age_seconds
) > (14 * 86400)keystone_operator_db_sync_total
Count of DB-related Job terminal transitions per Keystone CR, labelled by the terminal state.
| Attribute | Value |
|---|---|
| Type | Counter |
| Labels | keystone, namespace, result |
| Call site | reconcileDatabase, reconcileExpand, reconcileMigrate, reconcileContract on each Job's first terminal transition |
Covered Jobs. The counter aggregates terminal transitions from every DB-related Job the operator runs against the Keystone CR:
<keystone>-db-sync— non-upgrade schema sync (reconcileDatabase).<keystone>-schema-check— post-db_syncdrift detection.<keystone>-db-expand/-db-migrate/-db-contract— the three phases of the expand-migrate-contract upgrade flow.
This deliberate aggregation keeps the dashboard panel and failure-rate alerts populated during upgrades. The counter does NOT carry a per-phase label; use the matching DatabaseReady condition reasons (DBSyncFailed, ExpandFailed, MigrateFailed, ContractFailed, SchemaDriftDetected) on the CR to attribute a failure to a specific phase.
result values. succeeded (Job Complete=True) or failed (Job Failed=True).
Single-emit guarantee. Each phase is guarded by an independent transition predicate keyed on the Job UID and persisted via the cobaltcore.c5c3.io/last-<phase>-job-uid annotation on the Keystone CR; repeated reconciles observing the same terminated Job contribute at most one increment, so polling does not inflate the counter. The annotation patch is committed BEFORE the metric is recorded so a transient apiserver failure cannot cause the next reconcile to double-count the same Job UID; on patch failure the metric is deferred to the next reconcile, an Info-level log line is emitted, and a Warning event with reason DBSyncMetricEmissionDeferred is recorded on the Keystone CR so persistent apiserver failures are visible at default log verbosity and via kubectl describe keystone.
Series lifecycle. Deleted alongside the per-CR rotation-age gauge in DeleteForKeystone when the Keystone CR is removed.
Example PromQL. DB-related failure rate (any phase) over the last hour:
sum by (keystone, namespace) (
rate(keystone_operator_db_sync_total{result="failed"}[1h])
)keystone_operator_db_sync_duration_seconds
Duration in seconds of terminated DB-related Jobs, measured as condition.LastTransitionTime − Job.CreationTimestamp.
| Attribute | Value |
|---|---|
| Type | Histogram |
| Labels | keystone, namespace |
| Buckets (seconds) | 1, 5, 10, 30, 60, 120, 300, 600 |
| Call site | reconcileDatabase, reconcileExpand, reconcileMigrate, reconcileContract on each Job's first terminal transition |
The histogram records one sample per terminated DB-related Job (db_sync, schema-check, db-expand, db-migrate, db-contract), regardless of result. Use keystone_operator_db_sync_total{result="failed"} to scope alerting to failures only; use this histogram to reason about how long a sync took when it did run.
Example PromQL. p95 db_sync duration over the last 24 hours:
histogram_quantile(
0.95,
sum by (keystone, namespace, le) (
rate(keystone_operator_db_sync_duration_seconds_bucket[24h])
)
)Reconcile duration SLOs
These service-level objectives make the reconcile hot path measurable, so a latency regression shows up against a documented target instead of only as a subjective "the operator feels slow". They are expressed against the same histograms the bundled dashboard plots.
| Regime | Signal | Objective |
|---|---|---|
Steady state (all conditions True, single CR) | keystone_operator_reconcile_duration_seconds per sub_reconciler | p95 ≤ 300 ms |
Steady state (all conditions True, single CR) | controller_runtime_reconcile_time_seconds{controller="keystone"} | p95 ≤ 2 s |
Active rotation or db_sync/bootstrap wait | keystone_operator_reconcile_duration_seconds per sub_reconciler | p95 ≤ 2 s |
The steady-state per-sub-reconciler target is what the hot-path optimisations (probe caching, config-render gating, redundant-read elimination) are measured against; the end-to-end target covers the orchestration and status-update work that keystone_operator_reconcile_duration_seconds does not sample, which is why it uses the built-in controller_runtime_reconcile_time_seconds histogram.
Steady-state per-sub-reconciler p95:
histogram_quantile(
0.95,
sum by (sub_reconciler, le) (
rate(keystone_operator_reconcile_duration_seconds_bucket[5m])
)
)End-to-end p95:
histogram_quantile(
0.95,
sum by (le) (
rate(controller_runtime_reconcile_time_seconds_bucket{controller="keystone"}[5m])
)
)The reconcile-performance benchmark scripts a 1/5/25-CR run and reports these percentiles, and can gate on the steady-state end-to-end p95 as a regression check.
Enabling the ServiceMonitor
The Helm chart ships an opt-in monitoring.coreos.com/v1 ServiceMonitor. Enable it via values.yaml:
# values.yaml fragment
monitoring:
serviceMonitor:
enabled: true
interval: 30sThe ServiceMonitor selector targets the operator Service by operator-library.selectorLabels on port metrics, scraping the path /metrics at the configured interval. Prometheus-operator CRDs (ServiceMonitor) must be installed in the cluster; the chart does not install them. For the end-to-end operator-side enablement flow, see How to enable the Keystone operator metrics endpoint.
Related
- Keystone Reconciler Architecture — Metrics Instrumentation — the
instrumenter.Instrumentcontract and the rule that every new sub-reconciler must be wrapped. - Observability & Diagnostics — human-facing status, conditions, and events.
- How to enable the Keystone operator metrics endpoint — cluster-side Prometheus/Grafana wiring.
operators/keystone/dashboards/keystone-operator.json— reference Grafana dashboard consuming the metrics above.