HPA Reconciliation

Reference documentation for the HorizontalPodAutoscaler (HPA) reconciliation logic that enables automatic scaling of Memcached pods based on resource utilization metrics.

Source: internal/controller/hpa.go, internal/controller/memcached_controller.go

Overview

When spec.autoscaling.enabled is true, the reconciler ensures a matching HorizontalPodAutoscaler exists in the same namespace with the same name as the Memcached CR. The HPA is constructed from the CR spec using a pure builder function, then applied via controllerutil.CreateOrUpdate for idempotent create/update semantics. A controller owner reference on the HPA enables automatic garbage collection when the Memcached CR is deleted.

When autoscaling is disabled (or the spec.autoscaling field is removed), any existing HPA owned by the CR is deleted, and the operator resumes managing replica count via spec.replicas.

The HPA is opt-in — it is only created when explicitly enabled.

CRD Field Path

text

spec.autoscaling

Defined in api/v1alpha1/memcached_types.go on the AutoscalingSpec struct:

type AutoscalingSpec struct {
    Enabled     bool                                          `json:"enabled,omitempty"`
    MinReplicas *int32                                        `json:"minReplicas,omitempty,omitzero"`
    MaxReplicas int32                                         `json:"maxReplicas,omitempty"`
    Metrics     []autoscalingv2.MetricSpec                    `json:"metrics,omitempty,omitzero"`
    Behavior    *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty,omitzero"`
}

Field	Type	Required	Default	Description
`enabled`	`bool`	No	`false`	Controls whether an HPA is created
`minReplicas`	`*int32`	No	`nil` (HPA default: 1)	Lower limit for autoscaler replica count
`maxReplicas`	`int32`	Yes	—	Upper limit for autoscaler replica count
`metrics`	`[]autoscalingv2.MetricSpec`	No	80% CPU utilization (via webhook)	Metrics used to calculate desired replica count
`behavior`	`*autoscalingv2.HorizontalPodAutoscalerBehavior`	No	300s scaleDown stabilization (via webhook)	Scaling behavior for up and down directions

HPA Construction

constructHPA(mc *Memcached, hpa *HorizontalPodAutoscaler) sets the desired state of the HPA in-place. It is called within the controllerutil.CreateOrUpdate mutate function so that both creation and updates use identical logic.

ScaleTargetRef

The HPA always targets the Deployment managed by the same Memcached CR:

hpa.Spec.ScaleTargetRef = autoscalingv2.CrossVersionObjectReference{
    APIVersion: "apps/v1",
    Kind:       "Deployment",
    Name:       mc.Name,
}

Webhook Defaults

The defaulting webhook (applied before the controller sees the CR) provides sensible defaults when autoscaling is enabled:

Metrics: If metrics is empty, the webhook injects a CPU utilization metric targeting 80% average utilization.
Behavior: If behavior is nil, the webhook injects a scaleDown stabilization window of 300 seconds to prevent cache stampedes during scale-down events.

Labels

The HPA uses the same standard Kubernetes recommended labels as the Deployment and Service, generated by labelsForMemcached(name):

Label Key	Value	Purpose
`app.kubernetes.io/name`	`memcached`	Identifies the application
`app.kubernetes.io/instance`	`<cr-name>`	Distinguishes instances of the same application
`app.kubernetes.io/managed-by`	`memcached-operator`	Identifies the managing controller

Reconciliation Method

reconcileHPA(ctx, mc *Memcached) on MemcachedReconciler ensures the HPA matches the desired state:

func (r *MemcachedReconciler) reconcileHPA(ctx context.Context, mc *memcachedv1alpha1.Memcached) error {
    if !hpaEnabled(mc) {
        return r.deleteOwnedResource(ctx, &autoscalingv2.HorizontalPodAutoscaler{
            ObjectMeta: metav1.ObjectMeta{Name: mc.Name, Namespace: mc.Namespace},
        }, "HorizontalPodAutoscaler")
    }

    hpa := &autoscalingv2.HorizontalPodAutoscaler{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
        },
    }

    _, err := r.reconcileResource(ctx, mc, hpa, func() error {
        constructHPA(mc, hpa)
        return nil
    }, "HorizontalPodAutoscaler")
    return err
}

Skip and Deletion Logic

The hpaEnabled guard returns false (triggering HPA deletion) when:

spec.autoscaling is nil
spec.autoscaling.enabled is false

When hpaEnabled returns false, the controller calls deleteOwnedResource to remove any existing HPA. This is idempotent — no error occurs if the HPA does not exist.

Owner Reference

The reconcileResource helper calls controllerutil.SetControllerReference, adding an owner reference to the HPA's metadata:

Field	Value
`apiVersion`	`memcached.c5c3.io/v1alpha1`
`kind`	`Memcached`
`name`	`<cr-name>`
`uid`	`<cr-uid>`
`controller`	`true`
`blockOwnerDeletion`	`true`

This enables:

Garbage collection: Deleting the Memcached CR automatically deletes the owned HPA via Kubernetes' owner reference cascade.
Watch filtering: The Owns(&autoscalingv2.HorizontalPodAutoscaler{}) watch on the controller maps HPA events back to the owning Memcached CR for reconciliation.

Reconciliation Order

reconcileHPA is called between reconcileDeployment and reconcileService in the main Reconcile function.

Deployment Replicas Interaction

When autoscaling is enabled, the operator sets Deployment.spec.replicas to nil, ceding replica management to the HPA controller. This prevents the operator from conflicting with HPA scaling decisions.

Autoscaling State	Deployment spec.replicas
Enabled	`nil` (HPA controls scaling)
Disabled	Restored from `spec.replicas` (default: 1 if nil)

The constructDeployment function checks hpaEnabled(mc) to determine whether to set replicas:

var replicasPtr *int32
if !hpaEnabled(mc) {
    replicas := int32(1)
    if mc.Spec.Replicas != nil {
        replicas = *mc.Spec.Replicas
    }
    replicasPtr = &replicas
}

Status Conditions

When autoscaling is enabled, status conditions reflect HPA-managed scaling:

Available: Message includes "(HPA-managed)" suffix.
Progressing: Desired replica count is sourced from Deployment.status.replicas (the HPA-managed count) rather than spec.replicas.
Degraded: Comparison uses the HPA-managed desired count.

When autoscaling is disabled, conditions revert to normal operator-managed messages.

Metrics

No new metric registration is needed. The existing reconcile_resource_total counter is automatically recorded by reconcileResource() with resource_kind=HorizontalPodAutoscaler. The same result label pattern (success/error) is used as for all other reconciled resources.

RBAC

The controller requires RBAC permissions for the autoscaling API group:

// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete

This generates a ClusterRole rule in config/rbac/role.yaml:

yaml

- apiGroups:
    - autoscaling
  resources:
    - horizontalpodautoscalers
  verbs:
    - create
    - delete
    - get
    - list
    - patch
    - update
    - watch

CR Examples

HPA with Full Autoscaling Spec

yaml

apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
  namespace: default
spec:
  replicas: 3
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 70
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 600

Produces:

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-cache
  namespace: default
  labels:
    app.kubernetes.io/name: memcached
    app.kubernetes.io/instance: my-cache
    app.kubernetes.io/managed-by: memcached-operator
  ownerReferences:
    - apiVersion: memcached.c5c3.io/v1alpha1
      kind: Memcached
      name: my-cache
      controller: true
      blockOwnerDeletion: true
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-cache
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600

HPA with Webhook Defaults

yaml

apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
spec:
  autoscaling:
    enabled: true
    maxReplicas: 5

After the defaulting webhook applies defaults, the HPA receives:

minReplicas: nil (HPA default: 1)
maxReplicas: 5
metrics: 80% CPU utilization (injected by webhook)
behavior: 300s scaleDown stabilization window (injected by webhook)

Autoscaling Disabled (Default)

yaml

apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
spec:
  replicas: 3

No HPA is created. The Deployment spec.replicas is set to 3. If an HPA previously existed (autoscaling was enabled then disabled), it is deleted.

Runtime Behavior

Action	Result
Enable autoscaling (`enabled: true`)	HPA created, Deployment `spec.replicas` set to `nil`
Change `maxReplicas`	HPA updated on next reconcile
Change `minReplicas`	HPA updated on next reconcile
Change `metrics`	HPA updated on next reconcile
Change `behavior`	HPA updated on next reconcile
Disable autoscaling (`enabled: false`)	HPA deleted, Deployment `spec.replicas` restored from CR
Remove `spec.autoscaling` entirely	HPA deleted, Deployment `spec.replicas` restored from CR
Delete Memcached CR	HPA deleted via garbage collection (owner reference)
Reconcile twice with same spec	No HPA update (idempotent)
External drift (manual HPA edit)	Corrected on next reconciliation cycle

Implementation

The constructHPA function in internal/controller/hpa.go is a pure function that sets HPA desired state in-place:

func constructHPA(mc *Memcached, hpa *HorizontalPodAutoscaler)

Sets metadata.labels using labelsForMemcached
Sets spec.scaleTargetRef to the managed Deployment (apps/v1, Deployment, <cr-name>)
Copies minReplicas, maxReplicas, metrics, and behavior from spec.autoscaling

The hpaEnabled function is a pure guard:

func hpaEnabled(mc *Memcached) bool

Returns false when spec.autoscaling is nil
Returns false when enabled is false
Returns true only when enabled is true

HPA Reconciliation ​

Overview ​

CRD Field Path ​

HPA Construction ​

ScaleTargetRef ​

Webhook Defaults ​

Labels ​

Reconciliation Method ​

Skip and Deletion Logic ​

Owner Reference ​

Reconciliation Order ​

Deployment Replicas Interaction ​

Status Conditions ​

Metrics ​

RBAC ​

CR Examples ​

HPA with Full Autoscaling Spec ​

HPA with Webhook Defaults ​

Autoscaling Disabled (Default) ​

Runtime Behavior ​

Implementation ​

HPA Reconciliation

Overview

CRD Field Path

HPA Construction

ScaleTargetRef

Webhook Defaults

Labels

Reconciliation Method

Skip and Deletion Logic

Owner Reference

Reconciliation Order

Deployment Replicas Interaction

Status Conditions

Metrics

RBAC

CR Examples

HPA with Full Autoscaling Spec

HPA with Webhook Defaults

Autoscaling Disabled (Default)

Runtime Behavior

Implementation