Skip to content

Envtest Integration Tests ​

Reference documentation for the envtest integration test suite that validates the full reconciliation loop against a real API server, covering resource creation, spec propagation, idempotency, deletion, status conditions, and multi-instance isolation.

Source: internal/controller/memcached_envtest_integration_test.go

Overview ​

The envtest integration tests exercise the reconciler end-to-end by running reconcileOnce() against a real API server (provided by controller-runtime's envtest framework). Unlike unit tests that use fake clients or test individual builder functions, these tests create actual Memcached CRs via the API server with webhook validation active, invoke the reconciler, and verify the resulting Kubernetes resources.

All integration tests live in a single file (memcached_envtest_integration_test.go) within the controller_test package, sharing the envtest bootstrap from suite_test.go.


Test Infrastructure ​

Envtest Bootstrap (suite_test.go) ​

The test suite configures a real API server with:

  • CRD installation from config/crd/bases and config/crd/thirdparty (includes Prometheus Operator CRDs for ServiceMonitor)
  • Webhook server from config/webhook (defaulting and validation webhooks are active during all tests)
  • Controller manager started in a background goroutine (enables garbage collection via owner references)
text
suite_test.go variables available to all tests:
├── k8sClient  client.Client     — envtest API client
├── ctx        context.Context    — cancellable context
├── cfg        *rest.Config       — API server config
└── testEnv    *envtest.Environment

Shared Helper Functions ​

Helpers are defined across existing test files and reused by integration tests:

HelperDefined InPurpose
uniqueName(prefix)memcached_crd_validation_test.goGenerates prefix-<uuid8> for test isolation
validMemcached(name)memcached_crd_validation_test.goReturns a minimal valid CR in default namespace
int32Ptr(i)memcached_crd_validation_test.goReturns *int32 for spec fields
strPtr(s)memcached_crd_validation_test.goReturns *string for spec fields
reconcileOnce(mc)memcached_deployment_reconcile_test.goCreates a reconciler and runs one reconcile cycle
fetchStatefulSet(mc)memcached_deployment_reconcile_test.goGets the StatefulSet with same name/namespace as CR
fetchService(mc)memcached_service_reconcile_test.goGets the Service with same name/namespace as CR
fetchPDB(mc)memcached_pdb_reconcile_test.goGets the PDB with same name/namespace as CR
fetchServiceMonitor(mc)memcached_servicemonitor_reconcile_test.goGets the ServiceMonitor with same name/namespace as CR
fetchNetworkPolicy(mc)memcached_networkpolicy_reconcile_test.goGets the NetworkPolicy with same name/namespace as CR
findCondition(conditions, type)memcached_status_reconcile_test.goFinds a status condition by type string

reconcileOnce Implementation ​

go
func reconcileOnce(mc *memcachedv1alpha1.Memcached) (ctrl.Result, error) {
    r := &controller.MemcachedReconciler{
        Client: k8sClient,
        Scheme: scheme.Scheme,
    }
    return r.Reconcile(ctx, ctrl.Request{
        NamespacedName: client.ObjectKeyFromObject(mc),
    })
}

This creates a fresh reconciler for each call, using the envtest k8sClient. It does not set up watches or event recording — it invokes the Reconcile method directly with a ctrl.Request.


Test Organization ​

The integration tests are organized into Ginkgo Describe blocks by concern:

Describe BlockConcernREQ Coverage
Full reconciliation loop: minimal CRStatefulSet + Service creation, no optional resourcesREQ-001, REQ-002
Full reconciliation loop: full-featured CRAll five resources created in one reconcileREQ-001, REQ-002
Spec update propagationReplicas, image, monitoring changes propagateREQ-003
Optional resource enable/disable lifecyclePDB, ServiceMonitor, NetworkPolicy toggle on/offREQ-001, REQ-003
Full idempotencyThree consecutive reconciles without resource version changesREQ-006
Status conditions lifecycleAvailable, Progressing, Degraded through lifecycle stagesREQ-005
CR deletion and garbage collectionOwner references enable GC, reconcile returns success for deleted CRREQ-004
Owned resource recreation after external deletionDrift correction recreates deleted resourcesREQ-007
Multi-instance isolationTwo CRs in same namespace with independent resourcesREQ-008
Cross-resource consistencyMonitoring toggle updates StatefulSet, Service, NetworkPolicy atomicallyREQ-003
Full create-update-delete lifecycleEnd-to-end lifecycle in a single testREQ-001, REQ-003, REQ-004, REQ-005

Test Patterns ​

Standard Test Structure ​

Every integration test follows this pattern:

go
var _ = Describe("Category", func() {
    Context("scenario description", func() {
        var mc *memcachedv1alpha1.Memcached

        BeforeEach(func() {
            mc = validMemcached(uniqueName("prefix"))
            // Configure spec fields...
            Expect(k8sClient.Create(ctx, mc)).To(Succeed())
            Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(mc), mc)).To(Succeed())

            result, err := reconcileOnce(mc)
            Expect(err).NotTo(HaveOccurred())
            Expect(result).To(Equal(ctrl.Result{}))
        })

        It("should verify resource state", func() {
            sts := fetchStatefulSet(mc)
            Expect(*sts.Spec.Replicas).To(Equal(int32(1)))
        })
    })
})

Key conventions:

  1. uniqueName() for every CR — prevents cross-test interference
  2. validMemcached() as the starting point — webhook-valid minimal CR
  3. BeforeEach creates the CR and runs the initial reconcile
  4. It blocks verify specific aspects of the resulting state
  5. Re-fetch before mutation — k8sClient.Get() before k8sClient.Update() to get a fresh resourceVersion

Spec Update Pattern ​

go
// Always re-fetch before updating to get current resourceVersion.
Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(mc), mc)).To(Succeed())
mc.Spec.Replicas = int32Ptr(3)
Expect(k8sClient.Update(ctx, mc)).To(Succeed())

_, err = reconcileOnce(mc)
Expect(err).NotTo(HaveOccurred())

sts = fetchStatefulSet(mc)
Expect(*sts.Spec.Replicas).To(Equal(int32(3)))

Resource Absence Verification ​

go
pdb := &policyv1.PodDisruptionBudget{}
err := k8sClient.Get(ctx, client.ObjectKeyFromObject(mc), pdb)
Expect(apierrors.IsNotFound(err)).To(BeTrue())

Idempotency Verification ​

go
sts1 := fetchStatefulSet(mc)
rv := sts1.ResourceVersion

_, err := reconcileOnce(mc)
Expect(err).NotTo(HaveOccurred())

sts2 := fetchStatefulSet(mc)
Expect(sts2.ResourceVersion).To(Equal(rv))

Owner Reference Verification ​

go
for _, obj := range []client.Object{sts, svc, pdb, sm, np} {
    refs := obj.GetOwnerReferences()
    Expect(refs).To(HaveLen(1))
    Expect(refs[0].Name).To(Equal(mc.Name))
    Expect(refs[0].UID).To(Equal(mc.UID))
    Expect(refs[0].Kind).To(Equal("Memcached"))
    Expect(*refs[0].Controller).To(BeTrue())
    Expect(*refs[0].BlockOwnerDeletion).To(BeTrue())
}

Label Consistency Verification ​

go
for _, obj := range []client.Object{sts, svc, pdb, sm, np} {
    labels := obj.GetLabels()
    Expect(labels).To(HaveKeyWithValue("app.kubernetes.io/name", "memcached"))
    Expect(labels).To(HaveKeyWithValue("app.kubernetes.io/instance", mc.Name))
    Expect(labels).To(HaveKeyWithValue("app.kubernetes.io/managed-by", "memcached-operator"))
}

Envtest Pitfalls ​

PitfallImpactMitigation
No kubelet in envtestreadyReplicas always remains 0 in StatefulSet statusTests assert Degraded=True for non-zero desired replicas; use replicas=0 to test Available=True
Webhook validation is activeCR mutations that violate CRD validation will be rejectedAlways start from validMemcached() and make incremental changes
Resource names must be uniqueShared envtest instance across all tests in the suiteAlways use uniqueName() with a descriptive prefix
Reconciler does not delete optional resourcesDisabling a feature flag does not remove the resourceTests verify the reconciler succeeds (no error) but do not assert NotFound for disabled resources
GC requires controller managerOwner reference cascade only works when the manager is runningsuite_test.go starts the manager; GC-dependent tests use Eventually()
Re-fetch before updatek8sClient.Update() requires current resourceVersionAlways call k8sClient.Get() before k8sClient.Update()

Requirement Coverage Matrix ​

REQ-IDRequirementTest Scenarios
REQ-001Create all required resources per CRMinimal CR: StatefulSet + Service; Full CR: all five resources; Optional resources NotFound when disabled
REQ-002Owner references with controller=true, blockOwnerDeletion=trueVerified on all resources for both minimal and full-featured CRs
REQ-003Spec changes propagate in single reconcileReplicas, image, monitoring enable/disable, cross-resource consistency (StatefulSet + Service + NetworkPolicy updated atomically)
REQ-004CR deletion handled gracefullyReconcile returns ctrl.Result{} + nil for deleted CR; owner references enable GC
REQ-005Status conditions set correctlyInitial: Available=False/Progressing=True/Degraded=True; Zero replicas: Available=True/Progressing=False/Degraded=False; ObservedGeneration tracks changes
REQ-006Idempotent reconciliationThree consecutive reconciles on full-featured CR: no resource version changes after first; post-update idempotency verified
REQ-007Recreate externally deleted resourcesStatefulSet, Service, PDB, ServiceMonitor, NetworkPolicy each tested individually; multi-resource simultaneous deletion tested
REQ-008Multi-instance isolationTwo CRs: independent resources, distinct labels, deletion of one does not affect the other

Adding a New Integration Test ​

To add a new integration test scenario:

1. Choose the test file ​

All integration tests go in internal/controller/memcached_envtest_integration_test.go.

2. Write the test ​

go
var _ = Describe("New Feature Integration", func() {
    Context("when the new feature is enabled", func() {
        It("should produce the expected resource state", func() {
            mc := validMemcached(uniqueName("integ-new-feature"))
            // Configure the CR spec for your scenario.
            mc.Spec.NewField = someValue

            Expect(k8sClient.Create(ctx, mc)).To(Succeed())
            Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(mc), mc)).To(Succeed())

            _, err := reconcileOnce(mc)
            Expect(err).NotTo(HaveOccurred())

            // Verify the expected resource state.
            sts := fetchStatefulSet(mc)
            Expect(sts.Spec.SomeField).To(Equal(expectedValue))
        })
    })
})

3. Follow conventions ​

  • Use uniqueName("integ-<short-descriptor>") for the CR name
  • Start from validMemcached() and add only the fields your test needs
  • Use fetch* helpers for resource retrieval (they fail the test on NotFound)
  • Use apierrors.IsNotFound(err) when asserting resource absence
  • Re-fetch the CR with k8sClient.Get() before any k8sClient.Update()
  • If adding a new fetch* helper, define it in the corresponding *_reconcile_test.go file using ExpectWithOffset(1, ...) for correct failure line reporting

4. Run the tests ​

bash
make test

All integration tests run as part of the Controller Suite Ginkgo suite alongside unit and reconcile-level tests.