Skip to content

Architecture Overview ​

This document explains the architecture of the Memcached Operator, its design principles, how the reconciliation loop works, and its role within the CobaltCore ecosystem.


CobaltCore Context ​

The Memcached Operator is part of CobaltCore (C5C3), a Kubernetes-native OpenStack distribution for operating Hosted Control Planes on bare-metal infrastructure.

CobaltCore follows a strict operator-per-service architecture: every component gets its own dedicated Kubernetes operator. The central c5c3-operator orchestrates the overall lifecycle by creating Custom Resources that the individual operators reconcile.

Memcached is one of four infrastructure services in the Control Plane Cluster, deployed in Phase 1 of the dependency graph -- before any OpenStack service can start:

text
Phase 1: Infrastructure    --> MariaDB, RabbitMQ, Valkey, Memcached
Phase 2: Identity          --> Keystone (depends: MariaDB, Memcached)
Phase 3: Service Catalog   --> K-ORC
Phase 4: Core              --> Glance, Placement
Phase 5: Compute           --> Nova, Neutron, Cinder

The primary consumer of Memcached is Keystone, which uses it for token caching via the dogpile.cache.pymemcache backend. Every Keystone instance connects to the individual Memcached pods by address (e.g., memcached-0.memcached.openstack.svc:11211), so the operator runs the pods in a StatefulSet behind a headless Service (clusterIP: None). The operator lists one address per pod in status.serverList.


Operator Architecture ​

text
+---------------------------------------------------------------------+
|                        Kubernetes Cluster                           |
|                                                                     |
|  +--------------------------------------------------------------+   |
|  |                    Operator Manager                          |   |
|  |                                                              |   |
|  |  +---------------------+    +-----------------------------+  |   |
|  |  |  MemcachedReconciler|    |  Webhook Server             |  |   |
|  |  |                     |    |  - Defaulting               |  |   |
|  |  |  Watches:           |    |  - Validation               |  |   |
|  |  |  - Memcached CR     |    +-----------------------------+  |   |
|  |  |  - StatefulSets     |                                     |   |
|  |  |  - Services         |    +-----------------------------+  |   |
|  |  |  - PDBs             |    |  Metrics Server             |  |   |
|  |  |  - ServiceMonitors  |    |  :8443 /metrics             |  |   |
|  |  |  - NetworkPolicies  |    +-----------------------------+  |   |
|  |  +--------+------------+                                     |   |
|  +-----------|-------------------------------------------------+    |
|              | creates / updates / deletes                          |
|              v                                                      |
|  +--------------------------------------------------------------+   |
|  |                    Managed Resources                         |   |
|  |                                                              |   |
|  |  +--------------+  +----------+  +------+  +---------------+  |   |
|  |  | StatefulSet  |  | Service  |  | PDB  |  | ServiceMonitor|  |   |
|  |  |              |  | (headless|  |      |  |               |  |   |
|  |  | +----------+ |  | clusterIP|  |min:1 |  | prometheus    |  |   |
|  |  | |memcached | |  | :11211   |  |      |  | scrape config |  |   |
|  |  | |:11211    | |  +----------+  +------+  +---------------+  |   |
|  |  | +----------+ |                                              |   |
|  |  | |exporter  | |  +---------------+                           |   |
|  |  | |:9150     | |  | NetworkPolicy |                           |   |
|  |  | +----------+ |  |               |                           |   |
|  |  +--------------+  +---------------+                           |   |
|  +--------------------------------------------------------------+   |
+---------------------------------------------------------------------+

The operator runs as a single manager process in the cluster containing:

  • MemcachedReconciler -- the core controller that watches the Memcached Custom Resource and all owned child resources, driving them toward the declared desired state.
  • Webhook Server -- handles admission requests from the Kubernetes API server, providing defaulting (setting sensible defaults for omitted fields) and validation (enforcing complex cross-field constraints).
  • Metrics Server -- exposes controller-runtime metrics on :8443/metrics for Prometheus scraping.

Leader election (ID: d4f3c8a2.c5c3.io) ensures only one instance of the operator is active at a time when running multiple replicas.


Design Principles ​

Declarative ​

Users describe the desired state through a Memcached Custom Resource. The operator continuously drives reality toward that state. Users never issue imperative commands -- they declare what they want and the operator figures out how to get there.

Idempotent ​

Every reconciliation produces the same outcome regardless of how many times it runs. The operator uses the controllerutil.CreateOrUpdate pattern: it builds the desired state for each managed resource and applies it, creating the resource if it does not exist or updating it if it differs.

Kubernetes-native ​

The operator follows standard Kubernetes API conventions:

  • Owner references for automatic garbage collection of managed resources
  • Status conditions (Available, Progressing, Degraded) following the standard condition format
  • ObservedGeneration tracking to distinguish stale status from up-to-date status
  • Standard labels (app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/managed-by)

Level-triggered ​

Reconciliation reacts to the current state, not to a sequence of events. If the operator misses an event or is restarted, the next reconciliation will still converge to the correct state by comparing the declared desired state against the observed actual state. This makes the operator resilient to missed notifications, restarts, and race conditions.


Watched Resources ​

The controller sets up watches for the following resources. All child resource watches are filtered to resources owned by a Memcached CR via owner references.

ResourceAPI GroupWatch Reason
Memcachedmemcached.c5c3.ioPrimary resource; triggers reconciliation on changes
StatefulSetappsDetect drift, observe rollout status
Servicecore ("")Detect drift
PDBpolicyDetect drift
ServiceMonitormonitoring.coreos.comDetect drift (if monitoring enabled)
NetworkPolicynetworking.k8s.ioDetect drift (if network policy enabled)

When any watched resource changes, the controller enqueues the owning Memcached CR for reconciliation. This ensures that if someone manually edits a managed StatefulSet or Service, the operator will detect the drift and restore the desired state.


Reconciliation Flow ​

The MemcachedReconciler follows a structured, sequential reconciliation flow that runs every time the state of a watched resource changes.

text
Reconcile(ctx, req)
|
+-- 1. Fetch the Memcached CR
|   +-- Not found? --> return (deleted, nothing to do)
|   +-- Found --> continue
|
+-- 2. Set status condition: Progressing=True
|
+-- 3. Reconcile StatefulSet
|   +-- Build desired StatefulSet spec
|   +-- CreateOrUpdate (controllerutil)
|   |   +-- Create if not exists
|   |   +-- Update if spec differs
|   +-- Set owner reference on StatefulSet
|   +-- Delete the legacy Deployment named like the CR (only if the CR controls it)
|
+-- 4. Reconcile Service (headless, clusterIP: None)
|   +-- Build desired Service spec
|   +-- CreateOrUpdate
|   +-- Set owner reference
|
+-- 5. Reconcile PodDisruptionBudget (if HA PDB enabled)
|   +-- Build desired PDB spec
|   +-- CreateOrUpdate
|   +-- Set owner reference
|
+-- 6. Reconcile ServiceMonitor (if monitoring enabled)
|   +-- Check if ServiceMonitor CRD exists
|   +-- Build desired ServiceMonitor spec
|   +-- CreateOrUpdate
|   +-- Set owner reference
|
+-- 7. Reconcile NetworkPolicy (if security.networkPolicy enabled)
|   +-- Build desired NetworkPolicy spec
|   +-- CreateOrUpdate
|   +-- Set owner reference
|
+-- 8. Update Status
|   +-- Read StatefulSet status (readyReplicas)
|   +-- Set conditions: Available, Progressing, Degraded, Ready
|   +-- Set serverList, one entry per pod from the StatefulSet replica count
|   +-- Set observedGeneration
|   +-- Patch status sub-resource
|
+-- 9. Return result
    +-- No timed requeue; a change to the owned StatefulSet's status triggers the next reconcile through the watch
    +-- Exception: a 409 Conflict on the status write requeues the pass after 1 s without an error

Create-or-Update Pattern ​

All managed resources follow the create-or-update pattern using controllerutil.CreateOrUpdate:

go
sts := &appsv1.StatefulSet{
    ObjectMeta: metav1.ObjectMeta{
        Name:      statefulSetName(mc.Name),
        Namespace: mc.Namespace,
    },
}

op, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
    // Keep the stored containers, because constructStatefulSet replaces the spec
    storedContainers := sts.Spec.Template.Spec.Containers
    // Set the desired state on the resource
    constructStatefulSet(mc, sts, secretHash, restartTrigger)
    // Add the values the API server would default
    setStatefulSetServerDefaults(sts, storedContainers)
    // Set owner reference for garbage collection
    return controllerutil.SetControllerReference(mc, sts, r.Scheme)
})

This pattern is idempotent: if the resource already exists and matches the desired state, the update is a no-op. If it differs, the resource is updated in place. If it does not exist, it is created.

The API server fills StatefulSet fields that the builder leaves empty, such as revisionHistoryLimit, the pod's restartPolicy, dnsPolicy, and schedulerName, the container's termination message settings, the probe timeout and thresholds, and the Secret volume mode. setStatefulSetServerDefaults writes those values itself, so the desired state matches the stored StatefulSet when nothing changed. A container keeps the image pull policy the API server chose for it while its image is unchanged; after an image change the API server chooses again. A pass with nothing to change therefore sends no update, emits no event, and counts as unchanged in memcached_operator_reconcile_resource_total.


Managed Resources Detail ​

StatefulSet ​

The operator creates a StatefulSet for each Memcached CR with the following characteristics:

  • Name: The CR name. A name longer than 52 characters is shortened to its first 43 characters, -, and the first 8 hex characters of the SHA-256 of the name, because Kubernetes adds the StatefulSet name to a pod label that is limited to 63 characters
  • Service name: serviceName is the CR name, so the headless Service governs the StatefulSet and every pod gets the stable DNS name <statefulset>-<i>.<name>.<namespace>.svc
  • Replicas: Configured via spec.replicas (default: 1, range: 0-64). With autoscaling enabled, the operator keeps the replica count the HPA set on the StatefulSet, and a new StatefulSet starts with spec.autoscaling.minReplicas, or 1 when that is unset
  • Pod management: Parallel, so pods are created and deleted together instead of one after another
  • Update strategy: RollingUpdate, which replaces one pod at a time without a surge pod: each pod is deleted and then recreated under the same name. A single-replica instance has no ready pod while its pod is replaced
  • Memcached container: Runs the Memcached server with command-line arguments derived from spec.memcached fields
  • Exporter sidecar (optional): When spec.monitoring.enabled is true, a prom/memcached-exporter sidecar is injected, exposing metrics on port 9150
  • Health probes:
    • Liveness: TCP socket on port 11211, initialDelaySeconds=10, periodSeconds=10
    • Readiness: TCP socket on port 11211, initialDelaySeconds=5, periodSeconds=5
  • Labels: app.kubernetes.io/name=memcached, app.kubernetes.io/instance=<name>, app.kubernetes.io/managed-by=memcached-operator

Operator releases before the StatefulSet ran the pods in a Deployment named like the CR. After the StatefulSet is reconciled, the reconciler deletes that Deployment with background propagation and emits a Normal event with reason Deleted. It deletes the Deployment only when the Memcached CR controls it, so a Deployment with the same name that the CR does not own is left alone.

Headless Service ​

A headless Service (clusterIP: None) is created for each Memcached CR, exposing:

  • Port 11211 (memcached) for client connections
  • Port 9150 (metrics) for Prometheus scraping (when monitoring is enabled)

The headless Service enables direct pod discovery by DNS, which is critical for clients like Keystone's pymemcache that connect to individual pod addresses. The StatefulSet names this Service in serviceName, so each pod gets its own DNS record, and the operator publishes those names in status.serverList.

PodDisruptionBudget ​

Created when spec.highAvailability.podDisruptionBudget.enabled is true. Supports both minAvailable and maxUnavailable configurations. The controller defaults minAvailable to 1 when neither field is set.

ServiceMonitor ​

Created when spec.monitoring.enabled is true and the ServiceMonitor CRD exists in the cluster. The controller checks for CRD availability at reconciliation time and gracefully skips ServiceMonitor creation if the Prometheus Operator CRDs are not installed.

NetworkPolicy ​

Created when spec.security.networkPolicy.enabled is true. Restricts ingress traffic to the Memcached port (11211) from only the specified allowedSources. When allowedSources is empty, all sources are allowed.


Status Conditions ​

The operator uses standard Kubernetes conditions to communicate the state of a Memcached instance:

ConditionMeaning
AvailableTrue when the StatefulSet has at least one ready pod
ProgressingTrue when a rollout or scale operation is in progress
DegradedTrue when fewer replicas than desired are ready

Condition updates use meta.SetStatusCondition to handle lastTransitionTime correctly -- the transition time is only updated when the condition's status value actually changes.

The status.observedGeneration field is set to the CR's metadata.generation after each successful reconciliation, allowing clients to determine whether the status reflects the latest spec changes.


Owner References and Garbage Collection ​

Every managed resource (StatefulSet, HPA, Service, PDB, ServiceMonitor, NetworkPolicy) gets a controller owner reference pointing back to the Memcached CR:

yaml
ownerReferences:
  - apiVersion: memcached.c5c3.io/v1alpha1
    kind: Memcached
    name: my-cache
    controller: true
    blockOwnerDeletion: true

This enables Kubernetes garbage collection: when the Memcached CR is deleted, the garbage collector automatically deletes all owned resources. No finalizer is required for this basic cleanup.


Requeue Strategy ​

The reconciler does not poll on a timer. The watch on the owned StatefulSet enqueues the Memcached CR whenever the StatefulSet or its status changes, so rollout progress is tracked through events. A reconcile that returns an error is requeued with exponential backoff. The one timed requeue follows a 409 Conflict on the status write: the pass drops the status it computed, logs Status update conflict, requeueing at Info level, and returns RequeueAfter: 1s without an error, so the next pass recomputes the status from a fresh read.


Labels ​

All managed resources are labeled with standard Kubernetes recommended labels:

LabelValue
app.kubernetes.io/namememcached
app.kubernetes.io/instance<Memcached CR name>
app.kubernetes.io/managed-bymemcached-operator

These labels are used for:

  • StatefulSet selector matching
  • Service selector matching
  • PDB selector matching
  • Pod anti-affinity label selectors
  • ServiceMonitor selector matching
  • NetworkPolicy pod selectors