Skip to content

Headless Service Reconciliation ​

Reference documentation for the headless Service reconciliation logic that creates and updates a v1 Service with clusterIP: None for each Memcached CR.

Source: internal/controller/service.go, internal/controller/memcached_controller.go

Overview ​

When a Memcached CR is created or updated, the reconciler ensures a matching headless Service exists in the same namespace with the same name. The headless Service enables DNS-based pod discovery — clients such as pymemcache can resolve individual pod IPs via DNS SRV records for consistent hashing. The Service also governs the memcached StatefulSet through its spec.serviceName, which gives every pod a stable DNS name (see Per-Pod DNS Names). The Service is constructed from the CR spec using a pure builder function, then applied via controllerutil.CreateOrUpdate for idempotent create/update semantics. A controller owner reference on the Service enables automatic garbage collection when the Memcached CR is deleted.

The headless Service is always created for every Memcached CR. It is not opt-in.


Service Construction ​

constructService(mc *Memcached, svc *Service) sets the desired state of the headless Service in-place. It is called within the controllerutil.CreateOrUpdate mutate function so that both creation and updates use identical logic.

ClusterIP ​

The Service is always headless:

go
svc.Spec.ClusterIP = corev1.ClusterIPNone  // "None"

Setting clusterIP: None means Kubernetes does not allocate a virtual IP. Instead, DNS queries for the Service name return A records for each ready pod IP, enabling direct pod-to-pod communication.

Note: The ClusterIP field is immutable after Service creation. Because constructService sets it in the mutate function, it is applied during the initial Create call and left unchanged on subsequent Updates.

Port Configuration ​

The Service exposes a single port:

go
corev1.ServicePort{
    Name:       "memcached",
    Port:       11211,
    TargetPort: intstr.FromString("memcached"),
    Protocol:   corev1.ProtocolTCP,
}
FieldValueDescription
namememcachedNamed port for DNS SRV record lookup
port11211Memcached default TCP port
targetPortmemcached (named)References the container port name on the pod
protocolTCPMemcached uses TCP

The targetPort uses the named port memcached (matching the container port defined in the StatefulSet pod template) rather than a numeric value, so the Service remains valid even if the container port number changes.

Note: The metrics port (9150) is not included in this Service. It will be added in Phase 4 (ServiceMonitor Reconciliation) when monitoring is enabled.

Labels ​

The Service uses the same standard Kubernetes recommended labels as the StatefulSet, generated by labelsForMemcached(name):

Label KeyValuePurpose
app.kubernetes.io/namememcachedIdentifies the application
app.kubernetes.io/instance<cr-name>Distinguishes instances of the same application
app.kubernetes.io/managed-bymemcached-operatorIdentifies the managing controller

These labels are set on the Service's metadata.labels and on spec.selector, ensuring the Service selects pods managed by the same Memcached CR instance.

Selector ​

The Service selector uses the same label set as the StatefulSet's spec.selector.matchLabels. This ensures the headless Service resolves to the exact same pods that the StatefulSet manages:

go
svc.Spec.Selector = labelsForMemcached(mc.Name)

Per-Pod DNS Names ​

The memcached StatefulSet sets spec.serviceName to the CR name, which is this Service's name (see StatefulSet Reconciliation). Kubernetes gives each StatefulSet pod the hostname <statefulset>-<i> and the subdomain <cr-name>, so the cluster DNS publishes one A record per pod:

text
<statefulset>-<i>.<cr-name>.<namespace>.svc

<statefulset> is the StatefulSet's name, which equals the CR name unless the CR name has more than 52 characters (see StatefulSet Name). The Service always keeps the full CR name. <i> is the pod's ordinal, from 0 to replicas - 1.

For a CR my-cache in namespace default with three replicas, the names are:

text
my-cache-0.my-cache.default.svc
my-cache-1.my-cache.default.svc
my-cache-2.my-cache.default.svc

A pod keeps its name across restarts and rolling updates, so its DNS name stays the same. The reconciler publishes these names, with port 11211, in status.serverList (see Status Conditions).

The Service leaves publishNotReadyAddresses unset, so a pod's record resolves only while the pod is ready. An unready memcached refuses connections anyway.

Custom Annotations ​

Custom annotations from spec.service.annotations are applied to the Service metadata. This supports integration with external tools such as Prometheus scraping, load balancer configuration, or service mesh injection.

go
if mc.Spec.Service != nil && len(mc.Spec.Service.Annotations) > 0 {
    svc.Annotations = mc.Spec.Service.Annotations
} else {
    svc.Annotations = nil
}
CR StateService Annotations
spec.service is nilNo annotations (nil)
spec.service.annotations emptyNo annotations (nil)
spec.service.annotations setExact copy of the annotation map

Annotations are replaced entirely on each reconciliation. The operator does not merge with existing annotations — whatever is in spec.service.annotations becomes the Service's annotation set.


Reconciliation Method ​

reconcileService(ctx, mc *Memcached) on MemcachedReconciler ensures the headless Service matches the desired state:

go
func (r *MemcachedReconciler) reconcileService(ctx context.Context, mc *memcachedv1alpha1.Memcached) error {
    logger := log.FromContext(ctx)

    svc := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      mc.Name,
            Namespace: mc.Namespace,
        },
    }

    result, err := controllerutil.CreateOrUpdate(ctx, r.Client, svc, func() error {
        constructService(mc, svc)
        return controllerutil.SetControllerReference(mc, svc, r.Scheme)
    })
    if err != nil {
        return fmt.Errorf("reconciling Service: %w", err)
    }

    logger.Info("Service reconciled", "name", svc.Name, "operation", result)
    return nil
}

CreateOrUpdate Behavior ​

controllerutil.CreateOrUpdate performs a server-side get-or-create:

ScenarioMutate Function CalledAPI Operationresult Value
Service does not existYesCreatecontrollerutil.OperationResultCreated
Service exists, spec differsYesUpdatecontrollerutil.OperationResultUpdated
Service exists, spec matchesYes(no-op)controllerutil.OperationResultNone

The mutate function runs before every create or update, ensuring the Service always reflects the current CR spec. External drift (manual edits) is corrected on the next reconciliation cycle.

Owner Reference ​

controllerutil.SetControllerReference adds an owner reference to the Service's metadata:

FieldValue
apiVersionmemcached.c5c3.io/v1alpha1
kindMemcached
name<cr-name>
uid<cr-uid>
controllertrue
blockOwnerDeletiontrue

This enables:

  • Garbage collection: Deleting the Memcached CR automatically deletes the owned Service via Kubernetes' owner reference cascade.
  • Watch filtering: The Owns(&corev1.Service{}) watch on the controller maps Service events back to the owning Memcached CR for reconciliation.

Error Handling ​

Error ScenarioBehavior
API server unreachableError returned, controller-runtime requeues with backoff
Service create/update failsError wrapped with context, returned for requeue
Owner reference conflictError from SetControllerReference, returned for requeue

Errors are wrapped with fmt.Errorf("reconciling Service: %w", err) to provide context in logs while preserving the original error for apierrors.IsXxx() checks upstream.


Reconcile Integration ​

The Reconcile method calls reconcileService after reconcileStatefulSet and reconcileHPA:

go
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    memcached := &memcachedv1beta1.Memcached{}
    if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    var missingSecrets []string
    missingSecrets, reconcileErr = r.reconcileStatefulSet(ctx, memcached)
    if reconcileErr != nil {
        return ctrl.Result{}, reconcileErr
    }

    if reconcileErr = r.reconcileHPA(ctx, memcached); reconcileErr != nil {
        return ctrl.Result{}, reconcileErr
    }

    if reconcileErr = r.reconcileService(ctx, memcached); reconcileErr != nil {
        return ctrl.Result{}, reconcileErr
    }

    // ... reconcilePDB, reconcileServiceMonitor, reconcileNetworkPolicy, reconcileStatus ...

    return statusResult, nil
}

On the first reconcile of a new CR, the StatefulSet is created before the Service. The StatefulSet does not need the Service to exist; the per-pod DNS records appear once the Service is created later in the same reconcile.

ScenarioReturn ValueEffect
CR not found (deleted)ctrl.Result{}, nilNo requeue; owner ref cascade handles Service cleanup
CR fetch failsctrl.Result{}, errRequeue with exponential backoff
StatefulSet reconcile failsctrl.Result{}, errRequeue; Service reconcile skipped
HPA reconcile failsctrl.Result{}, errRequeue; Service reconcile skipped
Service reconcile succeedsctrl.Result{}, nilReconcile continues with the PDB
Service reconcile failsctrl.Result{}, errRequeue with exponential backoff

Reconciliation Flow ​

text
  Memcached CR created/updated
            │
            ▼
  ┌─────────────────────────────┐
  │  Reconcile                  │
  │  1. Fetch Memcached CR      │
  │  2. If NotFound → return    │
  │  3. If error → requeue      │
  └────────────┬────────────────┘
               │
               ▼
  ┌─────────────────────────────┐
  │  reconcileStatefulSet       │
  │  (see deployment-           │
  │   reconciliation.md)        │
  └────────────┬────────────────┘
               │
               ▼
  ┌─────────────────────────────┐
  │  reconcileHPA               │
  │  (see hpa-                  │
  │   reconciliation.md)        │
  └────────────┬────────────────┘
               │
               ▼
  ┌─────────────────────────────┐
  │  reconcileService           │
  │                             │
  │  CreateOrUpdate:            │
  │    ┌──────────────────────┐ │
  │    │ Mutate function      │ │
  │    │  constructService    │ │
  │    │  SetControllerRef    │ │
  │    └──────────────────────┘ │
  │                             │
  │  Service (headless)         │
  │  ├─ Name: <cr-name>        │
  │  ├─ Namespace: <cr-ns>     │
  │  ├─ ClusterIP: None        │
  │  ├─ Port: 11211/TCP        │
  │  ├─ Selector: std labels   │
  │  ├─ Annotations: from spec │
  │  └─ OwnerRef → Memcached CR│
  └─────────────────────────────┘

CRD Spec: ServiceSpec ​

The spec.service field on the Memcached CR controls Service customization:

go
// ServiceSpec defines configuration for the headless Service.
type ServiceSpec struct {
    Annotations map[string]string `json:"annotations,omitempty,omitzero"`
}
FieldTypeRequiredDefaultDescription
spec.service*ServiceSpecNonilService configuration block
spec.service.annotationsmap[string]stringNonilCustom annotations for the Service

Service Manifest Examples ​

Minimal CR ​

yaml
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
  namespace: default
spec: {}

Produces:

yaml
apiVersion: v1
kind: Service
metadata:
  name: my-cache
  namespace: default
  labels:
    app.kubernetes.io/name: memcached
    app.kubernetes.io/instance: my-cache
    app.kubernetes.io/managed-by: memcached-operator
  ownerReferences:
    - apiVersion: memcached.c5c3.io/v1alpha1
      kind: Memcached
      name: my-cache
      controller: true
      blockOwnerDeletion: true
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: memcached
    app.kubernetes.io/instance: my-cache
    app.kubernetes.io/managed-by: memcached-operator
  ports:
    - name: memcached
      port: 11211
      targetPort: memcached
      protocol: TCP

CR with Custom Annotations ​

yaml
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
  name: my-cache
  namespace: default
spec:
  service:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "11211"

Produces a Service identical to the minimal example, with the addition of:

yaml
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "11211"