Skip to content

HPA Reconciliation ​

Reference documentation for the HorizontalPodAutoscaler (HPA) reconciliation logic that enables automatic scaling of Memcached pods based on resource utilization metrics.

Source: internal/controller/hpa.go, internal/controller/memcached_controller.go

Overview ​

When spec.autoscaling.enabled is true, the reconciler ensures a matching HorizontalPodAutoscaler exists in the same namespace with the same name as the Memcached CR. The HPA is constructed from the CR spec using a pure builder function, then applied via controllerutil.CreateOrUpdate for idempotent create/update semantics. A controller owner reference on the HPA enables automatic garbage collection when the Memcached CR is deleted.

When autoscaling is disabled (or the spec.autoscaling field is removed), any existing HPA owned by the CR is deleted, and the operator resumes managing replica count via spec.replicas.

The HPA is opt-in — it is only created when explicitly enabled.


CRD Field Path ​

text
spec.autoscaling

Defined in api/v1alpha1/memcached_types.go on the AutoscalingSpec struct:

go
type AutoscalingSpec struct {
    Enabled     bool                                          `json:"enabled,omitempty"`
    MinReplicas *int32                                        `json:"minReplicas,omitempty,omitzero"`
    MaxReplicas int32                                         `json:"maxReplicas,omitempty"`
    Metrics     []autoscalingv2.MetricSpec                    `json:"metrics,omitempty,omitzero"`
    Behavior    *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty,omitzero"`
}
FieldTypeRequiredDefaultDescription
enabledboolNofalseControls whether an HPA is created
minReplicas*int32Nonil (HPA default: 1)Lower limit for autoscaler replica count
maxReplicasint32Yes—Upper limit for autoscaler replica count
metrics[]autoscalingv2.MetricSpecNo80% CPU utilization (via webhook)Metrics used to calculate desired replica count
behavior*autoscalingv2.HorizontalPodAutoscalerBehaviorNo300s scaleDown stabilization (via webhook)Scaling behavior for up and down directions

HPA Construction ​

constructHPA(mc *Memcached, hpa *HorizontalPodAutoscaler) sets the desired state of the HPA in-place. It is called within the controllerutil.CreateOrUpdate mutate function so that both creation and updates use identical logic.

ScaleTargetRef ​

The HPA always targets the StatefulSet managed by the same Memcached CR:

go
hpa.Spec.ScaleTargetRef = autoscalingv2.CrossVersionObjectReference{
    APIVersion: "apps/v1",
    Kind:       "StatefulSet",
    Name:       statefulSetName(mc.Name),
}

statefulSetName returns the CR name unless it has more than 52 characters, in which case the StatefulSet name is shortened (see StatefulSet Reconciliation). The HPA itself keeps the full CR name. The StatefulSet's scale subresource carries the label selector the HPA needs to find the pods.

Webhook Defaults ​

The defaulting webhook (applied before the controller sees the CR) provides sensible defaults when autoscaling is enabled:

  • Metrics: If metrics is empty, the webhook injects a CPU utilization metric targeting 80% average utilization.
  • Behavior: If behavior is nil, the webhook injects a scaleDown stabilization window of 300 seconds to prevent cache stampedes during scale-down events.

Labels ​

The HPA uses the same standard Kubernetes recommended labels as the StatefulSet and Service, generated by labelsForMemcached(name):

Label KeyValuePurpose
app.kubernetes.io/namememcachedIdentifies the application
app.kubernetes.io/instance<cr-name>Distinguishes instances of the same application
app.kubernetes.io/managed-bymemcached-operatorIdentifies the managing controller

Reconciliation Method ​

reconcileHPA(ctx, mc *Memcached) on MemcachedReconciler ensures the HPA matches the desired state:

go
func (r *MemcachedReconciler) reconcileHPA(ctx context.Context, mc *memcachedv1beta1.Memcached) error {
    if !mc.IsAutoscalingEnabled() {
        return r.deleteOwnedResource(ctx, &autoscalingv2.HorizontalPodAutoscaler{
            ObjectMeta: metav1.ObjectMeta{Name: mc.Name, Namespace: mc.Namespace},
        }, "HorizontalPodAutoscaler")
    }

    hpa := &autoscalingv2.HorizontalPodAutoscaler{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
        },
    }

    _, err := r.reconcileResource(ctx, mc, hpa, func() error {
        constructHPA(mc, hpa)
        return nil
    }, "HorizontalPodAutoscaler")
    return err
}

Skip and Deletion Logic ​

The mc.IsAutoscalingEnabled() guard returns false (triggering HPA deletion) when:

  • spec.autoscaling is nil
  • spec.autoscaling.enabled is false

When IsAutoscalingEnabled returns false, the controller calls deleteOwnedResource to remove any existing HPA. This is idempotent — no error occurs if the HPA does not exist.

Owner Reference ​

The reconcileResource helper calls controllerutil.SetControllerReference, adding an owner reference to the HPA's metadata:

FieldValue
apiVersionmemcached.c5c3.io/v1beta1
kindMemcached
name<cr-name>
uid<cr-uid>
controllertrue
blockOwnerDeletiontrue

This enables:

  • Garbage collection: Deleting the Memcached CR automatically deletes the owned HPA via Kubernetes' owner reference cascade.
  • Watch filtering: The Owns(&autoscalingv2.HorizontalPodAutoscaler{}) watch on the controller maps HPA events back to the owning Memcached CR for reconciliation.

Reconciliation Order ​

reconcileHPA is called between reconcileStatefulSet and reconcileService in the main Reconcile function.


StatefulSet Replicas Interaction ​

When autoscaling is enabled, the HPA controller sets the StatefulSet's spec.replicas through the scale subresource. The operator keeps that count: constructStatefulSet reads the stored spec.replicas before it replaces the spec and passes it to statefulSetReplicas as current, which returns it unchanged while autoscaling is on. A StatefulSet created while autoscaling is on starts at spec.autoscaling.minReplicas, or 1 when that is nil, and the HPA scales it from there.

Autoscaling StateStatefulSet spec.replicas
Enabled, StatefulSet existsKept at the count the HPA last set
Enabled, new StatefulSetspec.autoscaling.minReplicas, or 1 if nil
DisabledRestored from spec.replicas (default: 1 if nil)

statefulSetReplicas never returns nil:

go
func statefulSetReplicas(mc *memcachedv1beta1.Memcached, current *int32) *int32 {
    replicas := memcachedv1beta1.DefaultReplicas
    switch {
    case !mc.IsAutoscalingEnabled():
        if mc.Spec.Replicas != nil {
            replicas = *mc.Spec.Replicas
        }
    case current != nil:
        replicas = *current
    case mc.Spec.Autoscaling.MinReplicas != nil:
        replicas = *mc.Spec.Autoscaling.MinReplicas
    }
    return &replicas
}

A nil replicas in the desired spec would not leave the count to the HPA. The API server defaults a nil replicas to 1, so controllerutil.CreateOrUpdate would see a difference from the stored count on every reconcile and send an update that undoes the HPA's scale-up. On a StatefulSet that deletes every pod above ordinal 0 and shrinks status.serverList.

status.serverList follows the StatefulSet's spec.replicas, so a scale change the HPA makes reaches serverList on the next reconcile, which the Owns(&appsv1.StatefulSet{}) watch triggers.

After an upgrade from a release that ran the pods in a Deployment, the new StatefulSet starts at minReplicas. The count the HPA had set on the old Deployment is not carried over; the HPA scales the StatefulSet back up within its sync loop.


Status Conditions ​

When autoscaling is enabled, status conditions reflect HPA-managed scaling:

  • Available: Message includes "(HPA-managed)" suffix.
  • Progressing: Desired replica count is sourced from StatefulSet.status.replicas (the HPA-managed count) rather than spec.replicas.
  • Degraded: Comparison uses the HPA-managed desired count.

When autoscaling is disabled, conditions revert to normal operator-managed messages.


Metrics ​

No new metric registration is needed. The existing reconcile_resource_total counter is automatically recorded by reconcileResource() with resource_kind=HorizontalPodAutoscaler. The same result label pattern (success/error) is used as for all other reconciled resources.


RBAC ​

The controller requires RBAC permissions for the autoscaling API group:

go
// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete

This generates a ClusterRole rule in config/rbac/role.yaml:

yaml
- apiGroups:
    - autoscaling
  resources:
    - horizontalpodautoscalers
  verbs:
    - create
    - delete
    - get
    - list
    - patch
    - update
    - watch

CR Examples ​

HPA with Full Autoscaling Spec ​

yaml
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
  namespace: default
spec:
  replicas: 3
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 70
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 600

Produces:

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-cache
  namespace: default
  labels:
    app.kubernetes.io/name: memcached
    app.kubernetes.io/instance: my-cache
    app.kubernetes.io/managed-by: memcached-operator
  ownerReferences:
    - apiVersion: memcached.c5c3.io/v1beta1
      kind: Memcached
      name: my-cache
      controller: true
      blockOwnerDeletion: true
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-cache
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600

HPA with Webhook Defaults ​

yaml
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
spec:
  autoscaling:
    enabled: true
    maxReplicas: 5

After the defaulting webhook applies defaults, the HPA receives:

  • minReplicas: nil (HPA default: 1)
  • maxReplicas: 5
  • metrics: 80% CPU utilization (injected by webhook)
  • behavior: 300s scaleDown stabilization window (injected by webhook)

Autoscaling Disabled (Default) ​

yaml
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
spec:
  replicas: 3

No HPA is created. The StatefulSet spec.replicas is set to 3. If an HPA previously existed (autoscaling was enabled then disabled), it is deleted.


Runtime Behavior ​

ActionResult
Enable autoscaling (enabled: true)HPA created, StatefulSet spec.replicas kept, then set by the HPA
HPA scales the StatefulSetCount kept on every reconcile; status.serverList follows it
Change maxReplicasHPA updated on next reconcile
Change minReplicasHPA updated on next reconcile
Change metricsHPA updated on next reconcile
Change behaviorHPA updated on next reconcile
Disable autoscaling (enabled: false)HPA deleted, StatefulSet spec.replicas restored from CR
Remove spec.autoscaling entirelyHPA deleted, StatefulSet spec.replicas restored from CR
Delete Memcached CRHPA deleted via garbage collection (owner reference)
Reconcile twice with same specNo HPA update (idempotent)
External drift (manual HPA edit)Corrected on next reconciliation cycle

Implementation ​

The constructHPA function in internal/controller/hpa.go is a pure function that sets HPA desired state in-place:

go
func constructHPA(mc *Memcached, hpa *HorizontalPodAutoscaler)
  • Sets metadata.labels using labelsForMemcached
  • Sets spec.scaleTargetRef to the managed StatefulSet (apps/v1, StatefulSet, statefulSetName(<cr-name>))
  • Copies minReplicas, maxReplicas, metrics, and behavior from spec.autoscaling

The IsAutoscalingEnabled method on the Memcached type is a pure guard:

go
func (mc *Memcached) IsAutoscalingEnabled() bool
  • Returns false when spec.autoscaling is nil
  • Returns false when enabled is false
  • Returns true only when enabled is true