Headless Service Reconciliation
Reference documentation for the headless Service reconciliation logic that creates and updates a v1 Service with clusterIP: None for each Memcached CR.
Source: internal/controller/service.go, internal/controller/memcached_controller.go
Overview
When a Memcached CR is created or updated, the reconciler ensures a matching headless Service exists in the same namespace with the same name. The headless Service enables DNS-based pod discovery — clients such as pymemcache can resolve individual pod IPs via DNS SRV records for consistent hashing. The Service also governs the memcached StatefulSet through its spec.serviceName, which gives every pod a stable DNS name (see Per-Pod DNS Names). The Service is constructed from the CR spec using a pure builder function, then applied via controllerutil.CreateOrUpdate for idempotent create/update semantics. A controller owner reference on the Service enables automatic garbage collection when the Memcached CR is deleted.
The headless Service is always created for every Memcached CR. It is not opt-in.
Service Construction
constructService(mc *Memcached, svc *Service) sets the desired state of the headless Service in-place. It is called within the controllerutil.CreateOrUpdate mutate function so that both creation and updates use identical logic.
ClusterIP
The Service is always headless:
svc.Spec.ClusterIP = corev1.ClusterIPNone // "None"Setting clusterIP: None means Kubernetes does not allocate a virtual IP. Instead, DNS queries for the Service name return A records for each ready pod IP, enabling direct pod-to-pod communication.
Note: The
ClusterIPfield is immutable after Service creation. BecauseconstructServicesets it in the mutate function, it is applied during the initial Create call and left unchanged on subsequent Updates.
Port Configuration
The Service exposes a single port:
corev1.ServicePort{
Name: "memcached",
Port: 11211,
TargetPort: intstr.FromString("memcached"),
Protocol: corev1.ProtocolTCP,
}| Field | Value | Description |
|---|---|---|
name | memcached | Named port for DNS SRV record lookup |
port | 11211 | Memcached default TCP port |
targetPort | memcached (named) | References the container port name on the pod |
protocol | TCP | Memcached uses TCP |
The targetPort uses the named port memcached (matching the container port defined in the StatefulSet pod template) rather than a numeric value, so the Service remains valid even if the container port number changes.
Note: The metrics port (9150) is not included in this Service. It will be added in Phase 4 (ServiceMonitor Reconciliation) when monitoring is enabled.
Labels
The Service uses the same standard Kubernetes recommended labels as the StatefulSet, generated by labelsForMemcached(name):
| Label Key | Value | Purpose |
|---|---|---|
app.kubernetes.io/name | memcached | Identifies the application |
app.kubernetes.io/instance | <cr-name> | Distinguishes instances of the same application |
app.kubernetes.io/managed-by | memcached-operator | Identifies the managing controller |
These labels are set on the Service's metadata.labels and on spec.selector, ensuring the Service selects pods managed by the same Memcached CR instance.
Selector
The Service selector uses the same label set as the StatefulSet's spec.selector.matchLabels. This ensures the headless Service resolves to the exact same pods that the StatefulSet manages:
svc.Spec.Selector = labelsForMemcached(mc.Name)Per-Pod DNS Names
The memcached StatefulSet sets spec.serviceName to the CR name, which is this Service's name (see StatefulSet Reconciliation). Kubernetes gives each StatefulSet pod the hostname <statefulset>-<i> and the subdomain <cr-name>, so the cluster DNS publishes one A record per pod:
<statefulset>-<i>.<cr-name>.<namespace>.svc<statefulset> is the StatefulSet's name, which equals the CR name unless the CR name has more than 52 characters (see StatefulSet Name). The Service always keeps the full CR name. <i> is the pod's ordinal, from 0 to replicas - 1.
For a CR my-cache in namespace default with three replicas, the names are:
my-cache-0.my-cache.default.svc
my-cache-1.my-cache.default.svc
my-cache-2.my-cache.default.svcA pod keeps its name across restarts and rolling updates, so its DNS name stays the same. The reconciler publishes these names, with port 11211, in status.serverList (see Status Conditions).
The Service leaves publishNotReadyAddresses unset, so a pod's record resolves only while the pod is ready. An unready memcached refuses connections anyway.
Custom Annotations
Custom annotations from spec.service.annotations are applied to the Service metadata. This supports integration with external tools such as Prometheus scraping, load balancer configuration, or service mesh injection.
if mc.Spec.Service != nil && len(mc.Spec.Service.Annotations) > 0 {
svc.Annotations = mc.Spec.Service.Annotations
} else {
svc.Annotations = nil
}| CR State | Service Annotations |
|---|---|
spec.service is nil | No annotations (nil) |
spec.service.annotations empty | No annotations (nil) |
spec.service.annotations set | Exact copy of the annotation map |
Annotations are replaced entirely on each reconciliation. The operator does not merge with existing annotations — whatever is in spec.service.annotations becomes the Service's annotation set.
Reconciliation Method
reconcileService(ctx, mc *Memcached) on MemcachedReconciler ensures the headless Service matches the desired state:
func (r *MemcachedReconciler) reconcileService(ctx context.Context, mc *memcachedv1alpha1.Memcached) error {
logger := log.FromContext(ctx)
svc := &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: mc.Name,
Namespace: mc.Namespace,
},
}
result, err := controllerutil.CreateOrUpdate(ctx, r.Client, svc, func() error {
constructService(mc, svc)
return controllerutil.SetControllerReference(mc, svc, r.Scheme)
})
if err != nil {
return fmt.Errorf("reconciling Service: %w", err)
}
logger.Info("Service reconciled", "name", svc.Name, "operation", result)
return nil
}CreateOrUpdate Behavior
controllerutil.CreateOrUpdate performs a server-side get-or-create:
| Scenario | Mutate Function Called | API Operation | result Value |
|---|---|---|---|
| Service does not exist | Yes | Create | controllerutil.OperationResultCreated |
| Service exists, spec differs | Yes | Update | controllerutil.OperationResultUpdated |
| Service exists, spec matches | Yes | (no-op) | controllerutil.OperationResultNone |
The mutate function runs before every create or update, ensuring the Service always reflects the current CR spec. External drift (manual edits) is corrected on the next reconciliation cycle.
Owner Reference
controllerutil.SetControllerReference adds an owner reference to the Service's metadata:
| Field | Value |
|---|---|
apiVersion | memcached.c5c3.io/v1alpha1 |
kind | Memcached |
name | <cr-name> |
uid | <cr-uid> |
controller | true |
blockOwnerDeletion | true |
This enables:
- Garbage collection: Deleting the Memcached CR automatically deletes the owned Service via Kubernetes' owner reference cascade.
- Watch filtering: The
Owns(&corev1.Service{})watch on the controller maps Service events back to the owning Memcached CR for reconciliation.
Error Handling
| Error Scenario | Behavior |
|---|---|
| API server unreachable | Error returned, controller-runtime requeues with backoff |
| Service create/update fails | Error wrapped with context, returned for requeue |
| Owner reference conflict | Error from SetControllerReference, returned for requeue |
Errors are wrapped with fmt.Errorf("reconciling Service: %w", err) to provide context in logs while preserving the original error for apierrors.IsXxx() checks upstream.
Reconcile Integration
The Reconcile method calls reconcileService after reconcileStatefulSet and reconcileHPA:
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
memcached := &memcachedv1beta1.Memcached{}
if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
if apierrors.IsNotFound(err) {
return ctrl.Result{}, nil
}
return ctrl.Result{}, err
}
var missingSecrets []string
missingSecrets, reconcileErr = r.reconcileStatefulSet(ctx, memcached)
if reconcileErr != nil {
return ctrl.Result{}, reconcileErr
}
if reconcileErr = r.reconcileHPA(ctx, memcached); reconcileErr != nil {
return ctrl.Result{}, reconcileErr
}
if reconcileErr = r.reconcileService(ctx, memcached); reconcileErr != nil {
return ctrl.Result{}, reconcileErr
}
// ... reconcilePDB, reconcileServiceMonitor, reconcileNetworkPolicy, reconcileStatus ...
return statusResult, nil
}On the first reconcile of a new CR, the StatefulSet is created before the Service. The StatefulSet does not need the Service to exist; the per-pod DNS records appear once the Service is created later in the same reconcile.
| Scenario | Return Value | Effect |
|---|---|---|
| CR not found (deleted) | ctrl.Result{}, nil | No requeue; owner ref cascade handles Service cleanup |
| CR fetch fails | ctrl.Result{}, err | Requeue with exponential backoff |
| StatefulSet reconcile fails | ctrl.Result{}, err | Requeue; Service reconcile skipped |
| HPA reconcile fails | ctrl.Result{}, err | Requeue; Service reconcile skipped |
| Service reconcile succeeds | ctrl.Result{}, nil | Reconcile continues with the PDB |
| Service reconcile fails | ctrl.Result{}, err | Requeue with exponential backoff |
Reconciliation Flow
Memcached CR created/updated
│
▼
┌─────────────────────────────┐
│ Reconcile │
│ 1. Fetch Memcached CR │
│ 2. If NotFound → return │
│ 3. If error → requeue │
└────────────┬────────────────┘
│
▼
┌─────────────────────────────┐
│ reconcileStatefulSet │
│ (see deployment- │
│ reconciliation.md) │
└────────────┬────────────────┘
│
▼
┌─────────────────────────────┐
│ reconcileHPA │
│ (see hpa- │
│ reconciliation.md) │
└────────────┬────────────────┘
│
▼
┌─────────────────────────────┐
│ reconcileService │
│ │
│ CreateOrUpdate: │
│ ┌──────────────────────┐ │
│ │ Mutate function │ │
│ │ constructService │ │
│ │ SetControllerRef │ │
│ └──────────────────────┘ │
│ │
│ Service (headless) │
│ ├─ Name: <cr-name> │
│ ├─ Namespace: <cr-ns> │
│ ├─ ClusterIP: None │
│ ├─ Port: 11211/TCP │
│ ├─ Selector: std labels │
│ ├─ Annotations: from spec │
│ └─ OwnerRef → Memcached CR│
└─────────────────────────────┘CRD Spec: ServiceSpec
The spec.service field on the Memcached CR controls Service customization:
// ServiceSpec defines configuration for the headless Service.
type ServiceSpec struct {
Annotations map[string]string `json:"annotations,omitempty,omitzero"`
}| Field | Type | Required | Default | Description |
|---|---|---|---|---|
spec.service | *ServiceSpec | No | nil | Service configuration block |
spec.service.annotations | map[string]string | No | nil | Custom annotations for the Service |
Service Manifest Examples
Minimal CR
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
name: my-cache
namespace: default
spec: {}Produces:
apiVersion: v1
kind: Service
metadata:
name: my-cache
namespace: default
labels:
app.kubernetes.io/name: memcached
app.kubernetes.io/instance: my-cache
app.kubernetes.io/managed-by: memcached-operator
ownerReferences:
- apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
name: my-cache
controller: true
blockOwnerDeletion: true
spec:
clusterIP: None
selector:
app.kubernetes.io/name: memcached
app.kubernetes.io/instance: my-cache
app.kubernetes.io/managed-by: memcached-operator
ports:
- name: memcached
port: 11211
targetPort: memcached
protocol: TCPCR with Custom Annotations
apiVersion: memcached.c5c3.io/v1alpha1
kind: Memcached
metadata:
name: my-cache
namespace: default
spec:
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "11211"Produces a Service identical to the minimal example, with the addition of:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "11211"