all posts
2025-03-17 · 15 min read

Manage k8s resources using controllers. Part 3

Happy spring everyone! I'm Vlad and today we'll look at the final topic, which is crucial from the perspective of professional development in general and k8s operators and controllers in particular. As a reminder, in the previous article, we started the practical part of our task and wrote a small controller that manages valkey (open-source redis).

Other articles in the series:

Part 1. Theory

Part 2. Controller Implementation

Introduction

Testing is a critical stage in the development of any application, as tests provide confidence that everything works as expected. Kubernetes operators and Custom Resource Definition (CRD) controllers are no exception. Quality, multi-level testing ensures the reliability, stability, and predictability of operator behavior under various conditions and usage scenarios.

Kubernetes operators often manage critical resources and processes in the cluster, so errors in their operation can have serious consequences. Properly organized testing helps identify potential problems in the early stages of development.

In this article, we will explore different approaches to testing k8s operators: from simple unit tests to full-scale end-to-end tests in a test cluster environment. We will also discuss tools and frameworks that help automate the testing process and ensure high code quality.

Special attention will be paid to specific aspects of testing CRD controllers, including verification of reconciliation logic, validation of Custom Resource objects, and testing interaction with the Kubernetes API.

Resources for Practice

All code related to the practical part can be found on our GitHub:

https://github.com/uagolang/k8s-operator

Testing Theory

Let's dive a bit (but not too deep) into testing theory to make sure we're all on the same page. While different programming languages have their own names for various types of tests, standardizing them, I would highlight 3 main types.

Types of Tests

  • Unit tests are the basic level of testing that verifies individual functions and methods for correct operation. For more complex cases, table-driven tests are often used. They allow testing a function with different input data and edge cases.
  • Integration tests are verify the interaction between different system components and their correct operation together. Unlike unit tests that test individual functions in isolation, integration tests verify how different parts of code work together. For the test environment, we'll use the testenv package from Kubernetes.
  • End-to-end (e2e) tests are the highest level of testing that checks the entire system as a whole, simulating real program usage with real interaction with external services, testing through user interfaces (API endpoints, CLI, etc.) and verifying the entire data flow through the system.

Mocking

Mocking - is a technique in testing where real objects are replaced with their imitations to isolate the tested code from external dependencies. In Go, there are several approaches to mocking:

  • Manual mock creation - writing custom structures that implement required interfaces
  • Using mocking libraries - automated mock generation using special tools

In our task, I will use mock from Uber. It's a powerful tool for generating mocks in Go. It allows automatically generating mocks based on interfaces. Its main advantages, in my opinion, are:

  • Automatic mock code generation
  • Built-in matcher for argument verification
  • Ability to set expected behavior
  • Method call verification
  • Installing mockgen

First, you need to install the CLI mockgen:

bash
go install go.uber.org/mock/mockgen@latest

To generate a mock for some interface, you need to execute the command:

bash
mockgen -destination {destination} -package {package_name} -mock_names {interface_name}={mock_name} {interface_folder_path} {interface_name}

Let's generate all the mocks we need this way:

bash
# valkey service mock
mockgen -destination ./mocks/mock_valkey_service.go -package mocks -mock_names Service=MockValkeyService ./internal/services/valkey Service
 
# flow mock
mockgen -destination ./mocks/mock_flow.go -package mocks -mock_names Flow=MockFlow ./internal/controller/flows Flow
 
# k8s client
mockgen -destination ./mocks/mock_k8s_client.go -package mocks -mock_names Client=MockK8sClient sigs.k8s.io/controller-runtime/pkg/client Client
 
# k8s status client mocks
mockgen -destination ./mocks/mock_k8s_status_client.go -package mocks -mock_names SubResourceWriter=MockK8sStatusClient sigs.k8s.io/controller-runtime/pkg/client SubResourceWriter

They should be generated in the mocks folder in the root directory of the project. Now let's look at an example of their usage. So, let's choose one of the tests in the repository:

go
t.Run("success", func(t *testing.T) {
			// what should mock returns
			deployment := &appsv1.Deployment{
				Status: appsv1.DeploymentStatus{
					ReadyReplicas: 1,
				},
			}
 
			// imitate k8s Get method run with returning mocked deployment
			k8sClient.EXPECT().Get(ctx, types.NamespacedName{
				Name:      createRequest.CrdName,
				Namespace: createRequest.Namespace,
			}, gomock.AssignableToTypeOf(deployment)).DoAndReturn(
				func(_ context.Context, _ types.NamespacedName, obj runtimeclient.Object, _ ...runtimeclient.GetOption) error {
					*obj.(*appsv1.Deployment) = *(deployment)
					return nil
				})
		})
 
		// run tested method and check results
		ready, readyReplicas, err := s.IsReady(ctx, isReadyRequest)
		require.NoError(t, err)
		require.True(t, ready)
		require.Equal(t, int32(1), readyReplicas)
})

Let's break down the structure of the EXPECT builder when working with mocks:

go
mockObject.EXPECT().      // start expectation
    MethodName(arg1, arg2).  // method name with arguments
    Times(1).               // how many times it should be called
    Return(result, nil)     // what the method returns

Main builder components:

  • EXPECT() - start of mock expectations chain
  • MethodName(args...) - specifies which method and with what arguments is expected
  • Times(n) - how many times the method should be called:
    • Times(1) - exactly once
    • MinTimes(1) - at least once
    • MaxTimes(3) - maximum three times
    • AnyTimes() - any number of times
  • Return(...) - what the method should return

Additional useful builder methods:

  • Do(func) - execute arbitrary function on call
  • DoAndReturn(func) - execute function and return its result
  • After(other) - indicates that this call should happen after another

Tests, Tests, Tests!

Before I start commenting on examples from the project, I want to warn that I won't be going into too much detail about each test I've written, as this could stretch into another series of articles. I'll only look at selected tests and focus on interesting features and capabilities in writing them.

Since not all examples from the repository will be shown here (but you can easily check everything yourself) - I want to demonstrate a screenshot of running the make test command, which runs the tests and reports on test coverage:

make test results

make test results

As we can see, internal/controller has 97.5% coverage. If you look at the execution result in your IDE (I hope you're not writing in notepad), you'll see this picture:

test suite results with cover flag

test suite results with cover flag

The green vertical line here represents test-covered cases. I simply don't have enough screen space to show everything in one screenshot, but the red area - not covered by tests - is the only place I didn't spend time on, as there's no business logic in this method. However, this Reconcile isn't the only place with business logic. I would even say it's a kind of "skeleton" that allows isolating the real business logic, but which definitely needs testing. This is the level of integration tests. We actually run the Kubernetes envtest test environment, simulating the Kubernetes server operation and integration of other levels (flow, service) of the controller and application as a whole.

As for the business logic of bringing the object to the expected state, I described it as a Flow interface and its coverage is 100%! Yes, it has only one Run method, but there are enough use cases to thoroughly test the code. This interface will allow writing separate flows for each CRD. Its tests can be considered unit tests since they don't require a local environment to run. They simply verify the method's logic, and for dependencies (service) - I use mocks. Similarly with the service - its coverage is 100%! And I also consider its tests to be unit tests.

Table-Driven tests

go
package validator
 
import (
	"errors"
	"reflect"
	"testing"
)
 
func TestGetErrors(t *testing.T) {
	type args struct {
		err error
	}
	tests := []struct {
		name     string
		args     args
		wantErrs Errors
	}{
		{
			name: "no validation errors",
			args: args{
				err: errors.New("internal error"),
			},
			wantErrs: Errors{},
		},
		{
			name: "custom validation errors",
			args: args{
				err: Errors{
					{Field: "foo", Message: "bar"},
				},
			},
			wantErrs: Errors{
				{Field: "foo", Message: "bar"},
			},
		},
		{
			name: "custom validation error",
			args: args{
				err: Error{Field: "foo", Message: "bar"},
			},
			wantErrs: Errors{
				{Field: "foo", Message: "bar"},
			},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if gotErrs := GetErrors(tt.args.err); !reflect.DeepEqual(gotErrs, tt.wantErrs) {
				t.Errorf("GetErrors() = %v, want %v", gotErrs, tt.wantErrs)
			}
		})
	}
}

The tested function takes an error and returns validation errors if they exist. I simply iterate through all test cases and return an error if the received value differs from the expected one.

Service tests

go
// tests for service
func TestService(t *testing.T) {
	ctx := context.Background()
 
	// mock controller
	ctrl := gomock.NewController(t)
	defer ctrl.Finish()
 
	// mock some data or get needed mocks
	mockErr := errors.New("mock error")
	storage := "300Mi"
	k8sClient := mocks.NewMockK8sClient(ctrl)
	s := valkey.NewValkeyService(valkey.WithK8sClient(k8sClient))
 
	// predefined requests that
	// will be used in test cases
	createRequest := &valkey.CreateRequest{
		CrdName:   "valkey",
		Namespace: "default",
		Image:     "nesymno/k8s-operator:latest",
		User:      "user",
		Password:  "password",
		Replicas:  1,
		Volume: v1alpha1.Volume{
			Enabled: true,
			Storage: storage,
		},
		Resource: v1alpha1.Resource{
			CPU:     "100m",
			Memory:  "200Mi",
			Storage: storage,
		},
	}
 
	...
}

The test begins with its declaration and initialization of necessary data and dependencies.

go
// test cases for the method Create
t.Run("create", func(t *testing.T) {
 
		// success test case with creating
		// all needed k8s resources using mocks
		t.Run("success", func(t *testing.T) {
			k8sClient.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
			k8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
 
			err := s.Create(ctx, createRequest)
			require.NoError(t, err)
		})
 
		// handle error case when
		// request data is invalid
		t.Run("with validation errors", func(t *testing.T) {
			// make copy of object, don't use pointer
			// because value will be updated in createRequest
			req := *createRequest
			// invalidate some request data
			req.CrdName = ""
			req.Namespace = ""
			req.Image = ""
 
			// pointer to copied object
			err := s.Create(ctx, &req)
			require.Error(t, err)
 
			// get validator errors
			errs := validatorlib.GetErrors(err)
			if len(errs) == 0 {
				require.Errorf(t, err, "expected validation errors")
			}
 
			// check that got 3 errors
			// and what exact we expected
			require.Len(t, errs, 3)
			require.Equal(t, validatorlib.Errors{
				{
					Field:   "crd_name",
					Message: "crd_name is a required field",
				},
				{
					Field:   "namespace",
					Message: "namespace is a required field",
				},
				{
					Field:   "image",
					Message: "image is a required field",
				},
			}, errs)
		})
 
		// error case when we are waiting for
		// secret to be created, but for the first
		// check it was not found (not created yet)
		// and then on next check there was some error
		t.Run("wait secret failed", func(t *testing.T) {
			k8sClient.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
			notFoundErr := k8serrors.NewNotFound(schema.GroupResource{
				Group:    "",
				Resource: "secrets",
			}, createRequest.CrdName)
			k8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(notFoundErr)
			k8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(mockErr)
 
			err := s.Create(ctx, createRequest)
			require.Error(t, err)
		})
 
})
 
t.Run("is_ready", func(t *testing.T) {
 
		t.Run("success", func(t *testing.T) {
			// copy of that object will be returned
			deployment := &appsv1.Deployment{
				Status: appsv1.DeploymentStatus{
					ReadyReplicas: 1,
				},
			}
 
			// this is how you can use getter in mock
			// to return needed mocked data
			k8sClient.EXPECT().Get(ctx, types.NamespacedName{
				Name:      createRequest.CrdName,
				Namespace: createRequest.Namespace,
			}, gomock.AssignableToTypeOf(deployment)).DoAndReturn(
				func(_ context.Context, _ types.NamespacedName, obj runtimeclient.Object, _ ...runtimeclient.GetOption) error {
					*obj.(*appsv1.Deployment) = *(deployment)
					return nil
				})
		})
 
		ready, readyReplicas, err := s.IsReady(ctx, isReadyRequest)
		require.NoError(t, err)
		require.True(t, ready)
		require.Equal(t, int32(1), readyReplicas)
	})
 
})

All other service tests are built in a similar way, we just need to cover different cases (which is done in the repository).

Flow tests

go
func TestFlowRun(t *testing.T) {
	ctx := context.Background()
	ctrl := gomock.NewController(t)
	defer ctrl.Finish()
 
	const (
		resourceName     = "test-resource"
		defaultNamespace = "default"
	)
 
	mockErr := errors.New("mock error")
	mockK8sClient := mocks.NewMockK8sClient(ctrl)
	mockValkeySvc := mocks.NewMockValkeyService(ctrl)
 
	flow := valkey.NewFlow(
		valkey.WithK8sClient(mockK8sClient),
		valkey.WithValkeySvc(mockValkeySvc),
	)
}

At this level, everything follows similar logic as in the service, but with a few more mocks.

As for the tests themselves - they are also quite straightforward:

go
t.Run("healthcheck error", func(t *testing.T) {
		// imitate method call
		mockValkeySvc.EXPECT().Update(gomock.Any(), gomock.Any()).Return(nil)
		mockValkeySvc.EXPECT().IsReady(gomock.Any(), gomock.Any()).Return(false, int32(0), mockErr)
 
		status, finalizers, err := flow.Run(ctx, databasev1alpha1.Valkey{
			ObjectMeta: metav1.ObjectMeta{
				Name:       resourceName,
				Namespace:  defaultNamespace,
				Finalizers: []string{valkey.Finalizer},
			},
		})
 
		// checking expected results
		require.Nil(t, status)
		require.Nil(t, finalizers)
		require.Error(t, err)
})

Controller tests

First, let's look at internal/controller/suite_test.go. This is the entry point for tests in this package that are united by certain component interactions.

go
By("bootstrapping test environment")
testEnv = &envtest.Environment{
	CRDDirectoryPaths:     []string{filepath.Join("..", "..", "config", "crd", "bases")},
	ErrorIfCRDPathMissing: true,
 
	// The BinaryAssetsDirectory is only required if you want to run the tests directly
	// without call the makefile target test. If not informed it will look for the
	// default path defined in controller-runtime which is /usr/local/kubebuilder/.
	// Note that you must have the required binaries setup under the bin directory to perform
	// the tests directly. When we run make test it will be setup and used automatically.
	BinaryAssetsDirectory: filepath.Join("..", "..", "bin", "k8s",
		fmt.Sprintf("1.29.0-%s-%s", runtime.GOOS, runtime.GOARCH)),
}
 
var err error
// cfg is defined in this file globally.
cfg, err = testEnv.Start()
Expect(err).NotTo(HaveOccurred())
Expect(cfg).NotTo(BeNil())
 
err = databasev1alpha1.AddToScheme(scheme.Scheme)
Expect(err).NotTo(HaveOccurred())
 
// use fake k8s client (in-memory)
// instead of real cluster connection
k8sClient = fake.NewClientBuilder().
	WithScheme(scheme.Scheme).
	// you can add some object for seed cluster
	// with needed resource objects
	WithRuntimeObjects(flows.FakeComponents...).
	// register Status sub-resources to have an ability
	// to Update resource Status object
	WithStatusSubresource(
		&databasev1alpha1.Valkey{},
	).
	Build()
// just test that fake client was initialized
Expect(k8sClient).NotTo(BeNil())
 
mockCtrl := gomock.NewController(GinkgoT())
 
// init mocks
mockK8sClient = mocks.NewMockK8sClient(mockCtrl)
mockFlow = mocks.NewMockFlow(mockCtrl)
mockK8sStatusClient = mocks.NewMockK8sStatusClient(mockCtrl)
 
// init crd controllers
controllerValkey = &ValkeyReconciler{
	Client: k8sClient,
	Scheme: k8sClient.Scheme(),
	Flow:   mockFlow,
}

Was created a test environment using Kubernetes' testenv. Also added our CRD to the scheme. Created a fake k8s client (it stores everything in memory) and configure the controller. That's all.

I added 2 additional methods to the reconciler itself that allow me to replace the fake client with a mock:

go
// ValkeyReconciler reconciles a Valkey object
type ValkeyReconciler struct {
	client.Client
 
	fakeClient client.Client
 
	Scheme *runtime.Scheme
	Flow   flows.Flow
}
 
func (r *ValkeyReconciler) SetK8sClient(c *mocks.MockK8sClient) {
	r.fakeClient = r.Client
	r.Client = c
}
 
func (r *ValkeyReconciler) RollbackK8sClient() {
	if r.fakeClient != nil {
		r.Client = r.fakeClient
		r.fakeClient = nil
	}
}

This approach is necessary for testing certain cases, for example, when we need to verify errors from k8s itself, so we need to be able to simulate them. Let's look at a test example:

go
It("get resource not found error", func() {
	controllerValkey.SetK8sClient(mockK8sClient)
	defer controllerValkey.RollbackK8sClient()
 
	notFoundErr := k8serrors.NewNotFound(schema.GroupResource{
		Group:    "",
		Resource: "valkey",
	}, resourceName)
	mockK8sClient.EXPECT().Get(gomock.Any(), gomock.Any(), gomock.Any()).Return(notFoundErr)
 
	_, err := controllerValkey.Reconcile(ctx, reconcile.Request{
		NamespacedName: typeNamespacedName,
	})
	Expect(err).To(HaveOccurred())
})

As we can see, first I pass the mock client to the SetK8sClient method and immediately make a rollback to the fake one in defer, to ensure that after this test is complete, subsequent tests (which use controllerValkey.Client) will have access to the fake client, not the mock.

Here's the case this test corresponds to:

go
item := new(v1alpha1.Valkey)
if err := r.Get(ctx, req.NamespacedName, item); err != nil {
	if k8serrors.IsNotFound(err) { // this is the case
		return emptyResp, reconcile.TerminalError(err)
	} else {
		return emptyResp, err
	}
}

So, here we need to test the case when the object we are looking for does not exist. The not found error is sent by Kubernetes itself, and to simulate this error, I use a mock instead of a fake client.

Similarly, we need to cover other cases in the code with tests.

End-to-end tests

The E2E tests were generated during project creation, and I haven't modified them since there was no need. These tests run the operator itself rather than execute business logic. However, if the operator configuration is non-trivial, necessary cases should be covered with tests.

go
It("should run successfully", func() {
	var controllerPodName string
	var err error
 
	// projectimage stores the name of the image used in the example
	var projectimage = "example.com/k8s-operator:v0.0.1"
 
	By("building the manager(Operator) image")
	cmd := exec.Command("make", "docker-build", fmt.Sprintf("IMG=%s", projectimage))
	_, err = utils.Run(cmd)
	ExpectWithOffset(1, err).NotTo(HaveOccurred())
 
	By("loading the the manager(Operator) image on Kind")
	err = utils.LoadImageToKindClusterWithName(projectimage)
	ExpectWithOffset(1, err).NotTo(HaveOccurred())
 
	By("installing CRDs")
	cmd = exec.Command("make", "install")
	_, err = utils.Run(cmd)
	ExpectWithOffset(1, err).NotTo(HaveOccurred())
 
	By("deploying the controller-manager")
	cmd = exec.Command("make", "deploy", fmt.Sprintf("IMG=%s", projectimage))
	_, err = utils.Run(cmd)
	ExpectWithOffset(1, err).NotTo(HaveOccurred())
 
	By("validating that the controller-manager pod is running as expected")
	verifyControllerUp := func() error {
		// Get pod name
 
		cmd = exec.Command("kubectl", "get",
			"pods", "-l", "control-plane=controller-manager",
			"-o", "go-template={{ range .items }}"+
				"{{ if not .metadata.deletionTimestamp }}"+
				"{{ .metadata.name }}"+
				"{{ \"\\n\" }}{{ end }}{{ end }}",
			"-n", namespace,
		)
 
		podOutput, err := utils.Run(cmd)
		ExpectWithOffset(2, err).NotTo(HaveOccurred())
		podNames := utils.GetNonEmptyLines(string(podOutput))
		if len(podNames) != 1 {
			return fmt.Errorf("expect 1 controller pods running, but got %d", len(podNames))
		}
		controllerPodName = podNames[0]
		ExpectWithOffset(2, controllerPodName).Should(ContainSubstring("controller-manager"))
 
		// Validate pod status
		cmd = exec.Command("kubectl", "get",
			"pods", controllerPodName, "-o", "jsonpath={.status.phase}",
			"-n", namespace,
		)
		status, err := utils.Run(cmd)
		ExpectWithOffset(2, err).NotTo(HaveOccurred())
		if string(status) != "Running" {
			return fmt.Errorf("controller pod in %s status", status)
		}
		return nil
	}
	EventuallyWithOffset(1, verifyControllerUp, time.Minute, time.Second).Should(Succeed())
 
})

Conclusions

In this series of articles, we have thoroughly examined the tools for creating and operating principles of a Kubernetes operator that manages custom resources. We've traveled a challenging path from theoretical understanding of k8s resource operations and principles of bringing resources to the desired state, to practical implementation of our own controller and its test coverage!

This series of articles demonstrated the complete development cycle of a Kubernetes operator, from concept to implementation and testing. Key achievements include:

  • Detailed overview of k8s operators architecture and operating principles
  • Practical implementation of a controller using modern tools and approaches
  • Comprehensive code coverage with tests to ensure reliability

This experience can be valuable for developers who plan to create their own operators or simply want to better understand the internal mechanisms of Kubernetes. It's important to remember that creating an operator is a complex but structured process that requires understanding both theoretical foundations and practical aspects of development.

Key recommendations for those planning to develop their own operators:

  • Pay sufficient attention to architecture planning
  • Follow the principles of idempotency and declarative approach
  • Ensure proper test coverage
  • Use available tools and frameworks

I hope this series of articles was interesting and brought something new and exciting to your life as a software engineer and, possibly, future Platform Engineer 😊