Manage k8s resources using controllers. Part 1

Hi! As always, this is Vlad, and today I'd like to explore the topic of managing k8s resources using the CRD (Custom Resource Definition) operator in a k8s cluster.

Sounds massive, doesn't it? 😄 Actually, it is. This is quite an extensive topic that I became closely familiar with while working on a product aimed at enabling developers to maintain k8s clusters themselves without the need for DevOps specialists, or to reduce their number to the necessary minimum.

In my opinion, this article will be extremely useful and informative both for beginners who are just starting their journey in the world of Kubernetes, and for experienced professionals who already have practical experience working with Kubernetes API. The material is structured in such a way that every reader, regardless of their level of preparation, can gain valuable knowledge and practical advice about working with this powerful container orchestration platform. Docker and Kubernetes are written in Golang. This turns out to be a huge plus for gophers regarding native integration with Kubernetes API, which provides extensive opportunities for business use.

I would like to describe each aspect of this interesting process as thoroughly as possible. I'm planning a series of three articles, each of which will introduce you to certain cycles of development and testing of k8s operators:

1. Managing k8s resources using operators. Part 1
   1. Why did I decide to get familiar with this?
   2. Custom Kubernetes APIs
   3. Resource Reconciliation Process
2. Managing k8s resources using operators. Part 2
   1. Example of operator implementation for k8s
3. Managing k8s resources using operators. Part 3
   1. Testing k8s operator controllers

Introduction

Kubernetes (k8s) is rapidly evolving and becoming increasingly popular for deploying and managing containerised applications. However, with the growing complexity of infrastructure and number of resources, certain challenges arise in managing them.

The main problems that developers face when working with k8s resources:

• Complexity of manually managing a large number of interconnected resources
• Need for constant monitoring of resource status and responding to changes
• Lack of automation for typical resource operations
• Difficulty in maintaining consistency between desired and actual resource states

One of the effective solutions to these problems is creating custom Kubernetes operators.

Operator is an application that uses the Kubernetes API to manage custom resources (Custom Resource Definitions, CRD).

Operators are an important link in the chain of k8s resource management, as they allow:

• Automate routine operations with resources
• Implement custom business logic for resource management
• Ensure resource state consistency
• Simplify deployment and management of complex applications
• Properly covering the reconciliation process with tests

Let's dive into the world of Kubernetes (k8s) and get closer to the possibilities for Go developers, as Docker and Kubernetes themselves (if you didn't know) are written in Golang!

Why did I decide to get familiar with this?

Overall, in my practice, I worked with Kubernetes (k8s) at the level of local project deployment and knew some basic kubectl commands like apply, delete, etc. It was never a problem for me to ask a DevOps for help in deploying certain environments locally. But there are objective factors and project stages that don't allow having DevOps specialists, or having a very limited, insufficient number of them. Of course, the first thoughts about a startup! 🤦‍♂️

It all started in December 2023. About a couple of months before that, the project I had been working on for the last 3 years (also essentially a startup) failed, and the development team underwent severe cuts, as usual 😃 So, (imagine 💭) December 2023. I had been without work for 2 months already, and the pre-New Year and New Year holidays play a cruel joke every year on people trying to get an offer in the passing year. And I love jokes, even dark ones! Perhaps that's why higher powers decided among themselves: "better give this guy a chance, or he might become a gangster!" 🤙 Somewhere in mid-December, Anton, who is now my partner, wrote to me on LinkedIn and offered to join his team developing a startup in the Internal Development Platform niche, a fairly new direction but one that's rapidly gaining popularity and is used by large corporations to manage internal projects. All large (not all, but also not very large) projects exist in Kubernetes clusters.

The whole point of Internal Development Platform is that they manage all resources in Kubernetes (k8s), and the main beneficiaries are developers who can configure their cross-cloud infrastructure and manage various projects using a simple and understandable user interface. This is where I got to know Kubernetes (k8s) quite closely. I learned about the principles of its API operation, what CRD is, and why they are an important tool that allows effectively tracking and responding to events related to cloud infrastructure monitoring.

Custom Kubernetes APIs

Before moving on to examining tools that allow effective management and monitoring of resources in a k8s cluster, I'd like to focus more on the concept of CRD.

What is CRD in k8s and why do we, developers, need it?

CRD (Custom Resource Definition) is a mechanism in Kubernetes that allows developers to create their own, specific to their needs types of resources, extending the standard Kubernetes API.

For simpler understanding, it's similar to how developers create their own data types in a programming language - for example, when you define a new structure in Go or a class in Java to represent specific business entities. Just as these custom types extend the programming language's capabilities to solve specific tasks, CRDs extend Kubernetes capabilities to manage specific resources of your infrastructure.

CRD perform the following main functions:

• Define the structure and validation of custom resources
• Allow storing application-specific configurations in the cluster
• Integrate with existing Kubernetes tools
• Provide a declarative API for resource management

Speaking specifically about us, developers, CRD provide us with advantages and solve certain problems in resource management as proposed above and:

• Enable automation of complex operations through custom controllers

For example, if you're developing a platform for microservices management, you can create a Application CRD that will contain all the necessary configuration for deploying and managing your application, including scaling settings, network policies, resource limitations, and dependencies between services. This approach significantly simplifies the deployment process as it allows other developers to use a single YAML file to deploy complex microservice applications, instead of manually configuring and coordinating multiple standard Kubernetes resources such as Deployments, Services, ConfigMaps, and Ingress rules, etc.

KubeBuilder

KubeBuilder is a framework for developing Kubernetes operators that significantly simplifies the process of creating custom controllers and CRD. It provides developers with a set of tools and templates for quickly starting operator development.

The main problems that KubeBuilder solves:

• Automation of base code creation for operators
• Simplification of CRD and API types generation process
• Provision of ready-made templates for typical development patterns
• Integration with testing and deployment tools

The principle of KubeBuilder operation is based on code generation and the use of ready-made abstractions:

• Code generation: KubeBuilder uses Go markers for automatic generation of CRD code, API types, and validation, etc
• Project templates: Provides project structure and basic controller implementation
• Reconcile cycle: Implements a standard pattern for reconciling desired and current resource states
• Integration with controller-runtime: Uses the controller-runtime library for interaction with Kubernetes API

KubeBuilder allows developers to focus on implementing operator business logic rather than basic infrastructure and boilerplate code. This significantly speeds up development and reduces the number of potential errors.

You can read more about KubeBuilder capabilities on the official website.

But there is another project that acts as a framework for kubebuilder and aims to facilitate the development of new CRDs and APIs.

Operator SDK

Operator SDK is a tool that builds on top of KubeBuilder and simplifies the process of developing, testing, and maintaining Kubernetes operators. It combines best practices and operator development patterns into a unified framework.

Let's look at a table comparing the capabilities of KubeBuilder and Operator SDK:

Operator SDK solves these key problems:

• Reduces boilerplate code compared to pure KubeBuilder
• Simplifies the process of deploying and testing operators
• Provides a unified interface for working with different types of operators
• Automates the creation and management of CRD

Operator SDK is an extremely useful tool for teams just starting their journey in Kubernetes operators development. It not only provides a clearly structured and methodical approach to development, but also serves as a guide that helps teams follow time-tested practices and avoid common mistakes. Thanks to built-in templates and automated processes, teams can focus on implementing business logic instead of spending time configuring basic infrastructure and solving technical nuances.

You can read more about Operator SDK on the official website.

Resource Reconciliation Process

Resource Reconciliation is a key concept in Kubernetes responsible for maintaining the desired system state. It is a continuous process during which the controller compares the current state of a resource with the desired state and performs necessary actions to synchronize them.

Problems Solved by Resource Reconciliation

• Automatic Recovery: The system automatically corrects discrepancies between desired and current states
• Fault Tolerance: Even if an operation fails, the system continues attempting to achieve the desired state
• Scalability: Enables efficient management of a large number of resources in the cluster

Let's look at a simple example to better understand this concept. Imagine you have an operator that manages the deployment of a MongoDB database. Here's how the reconciliation process works:

• Desired State: User creates a Custom Resource specifying they want MongoDB with:
  • 3 replicas
  • version 8.0
  • 10GB storage for each replica
• Current State: There are no MongoDB pods in the cluster yet

The reconciliation process of CRD from the example follows this algorithm:

1. The operator notices a new Custom Resource and starts the Reconciliation cycle
2. Compares the current state (0 replicas) with the desired state (3 replicas)
3. Creates necessary Kubernetes resources: StatefulSet, Service, ConfigMap, etc.
4. Periodically checks if all pods are running and healthy
5. If something goes wrong (for example, a pod crashes), the operator automatically attempts to restore the desired state

This process is essentially an infinite loop. If a user modifies the Custom Resource (for example, increases the number of replicas to 5), the Reconciliation cycle will start again to bring the system to the new desired state.

For better understanding of the resource reconciliation process, let's examine its main principles. These principles form the foundation upon which the entire resource management system in Kubernetes is based.

I would highlight 3 main principles of Resource Reconciliation:

1. Declarative means that the developer simply describes the result to be achieved, not the implementation of how to do it. For example, we say: "I want 3 database copies," and the system figures out how to implement it.
2. Idempotency, which tells us that repeated actions on any object no longer change the result. It's like pressing an elevator button - no matter how many times you press "1st floor," the elevator will still go to the first floor.
3. Asynchronicity, which allows the system to work independently of other processes. It's similar to how a washing machine works on its own while you do other things.

Operator developers need to consider several key features when working with Resource Reconciliation. First, since the Reconciliation cycle can run in parallel for the same resource, the code must be thread-safe. Second, it's important to properly handle temporary errors and network failures by implementing retry mechanisms. Third, reconciliation logic needs to be optimized to avoid unnecessary operations and reduce load on the Kubernetes API server. It's also critically important to maintain the current resource state through status and condition updates so other system components can properly react to changes.

Let's summarize the key features of Resource Reconciliation:

• Reconciliation Cycle: The controller periodically checks resource states, so it's important to consider the possibility of parallel operation execution
• Error Handling: Errors must be properly handled and retry mechanisms implemented
• Optimization: It's important to avoid unnecessary operations and optimize reconciliation logic to reduce cluster load
• Statuses and Conditions: Resource statuses must be properly updated and appropriate conditions set to reflect reconciliation progress

When developing operators, it's important to understand that the Reconciliation cycle can be triggered for various reasons:

• Resource Change - it's important to note that a new Reconciliation cycle will be triggered both when changing the full object using the Update() method and when changing the Status field using the Status().Update() method. It's very important to understand this and not implement business logic where multiple calls to these methods exist in one cycle iteration (in one attempt), which can lead to additional errors and collisions.
• Periodic Check - this isn't mandatory (but I consider it good practice), it all depends on the specific task that the CRD solves. Again, any object change will restart the cycle.
• Adding a new CRD - starting the cycle for the created object.

We'll examine all of this in more detail with a specific example from the perspective of writing an operator in Golang, but just to understand the general algorithm - let's look at a small example:

// example of a simple reconcile function
func (r *MyController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // get resource
    resource := &myresv1.MyResource{}
    if err := r.Get(ctx, req.NamespacedName, resource); err != nil {
		    // end cycle if we got NotFound error
		    // or restart it for any other error
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
 
    // check current state
    currentState := getCurrentState()
 
    // compare with desired state
    if !isDesiredState(currentState, resource.Status) {
        // perform necessary actions to achieve desired state
        if err := applyChanges(ctx, resource); err != nil {
            // return error for retry after specified interval
            return ctrl.Result{RequeueAfter: time.Second * 5}, err
        }
    }
 
		// if resource is in desired state - end cycle
    return ctrl.Result{}, nil
}

This code demonstrates the basic structure of a reconcile function that checks the current state of a resource and performs necessary actions to achieve the desired state. In case of an error, the operation will be retried after 5 seconds.

Conclusion

In this article, we've examined important aspects of managing k8s resources using operators and CRD. Key points to note:

• Declarative Approach: Kubernetes uses a declarative model where we describe the desired state, and the system determines how to achieve it
• Resource Reconciliation: Key mechanism ensuring continuous reconciliation of current state with desired state
• Important Principles: Idempotency and asynchronicity are fundamental principles of operator operation
• Technical Aspects: Need for thread-safe code, proper error handling, and optimization of reconciliation logic

In the next article, we'll focus more deeply on creating a k8s operator from a Golang developer's perspective and examine some features that need to be known and considered when developing controllers for resources.