Kubernetes Webhooks: The Basics Every DevOps Engineer Should Know

By /

Kubernetes Webhooks: The Basics Every DevOps Engineer Should Know

Kubernetes webhooks are a powerful mechanism for extending and customizing the behavior of the Kubernetes API server. They act as interceptors for API requests, allowing you to inject custom logic before those requests are processed by the API server’s core components. This ability to intercept and modify requests makes webhooks an essential tool for DevOps engineers looking to implement custom validation, mutation, and policy enforcement.

This article covers the foundational concepts every DevOps engineer should understand about Kubernetes webhooks.

1. What are Kubernetes Webhooks?

At their core, Kubernetes webhooks are HTTP callbacks. When a relevant API request occurs (e.g., a request to create a Pod, Deployment, or Service), the Kubernetes API server sends an HTTP POST request to a configured webhook endpoint. This endpoint, typically a service running within your cluster (though it can be external), processes the request, potentially performs actions, and returns a response to the API server. The API server then uses this response to decide whether to proceed with the original request, modify it, or reject it altogether.

2. Types of Webhooks

There are two primary types of Kubernetes webhooks:

  • Mutating Webhooks: These webhooks are invoked before the request is validated and stored in etcd. They can modify the object contained in the request. This is incredibly useful for tasks such as:

    • Defaulting: Setting default values for fields that are not explicitly specified in the request. For example, automatically adding resource requests/limits to Pods.
    • Labeling: Adding specific labels to resources based on certain criteria. For instance, labeling Pods based on their namespace or owner.
    • Injection: Injecting sidecar containers into Pods. A common example is automatically injecting service mesh proxies (like Istio or Linkerd) into newly created Pods.
    • Adding Annotations: Similar to labeling, you can add annotations.

    Mutating webhooks are processed in the order they are defined. The output of one mutating webhook becomes the input to the next.

  • Validating Webhooks: These webhooks are invoked after the object is mutated (if any mutating webhooks are configured) and before it’s persisted to etcd. They cannot modify the object; their purpose is purely to validate it. Validating webhooks are crucial for:

    • Policy Enforcement: Enforcing custom policies. For example, restricting the use of specific container images, ensuring that all Pods have resource limits, or requiring specific annotations.
    • Data Validation: Validating that the requested configuration conforms to custom rules beyond the standard Kubernetes schema validation. For instance, checking that a ConfigMap contains specific keys or that a Service uses a valid port.
    • Security Checks: Performing security-related checks, such as preventing the use of insecure capabilities.
    • Dry Runs: Webhooks support DryRun, which enables validating/mutating an object without committing changes to etcd.

If a validating webhook rejects a request, the API server returns an error to the client (e.g., kubectl), and the object is not created/updated/deleted. Like mutating webhooks, validating webhooks are processed in the order defined. A failure in any validating webhook will cause the entire request to fail.

3. Key Components of Webhook Configuration

To define a webhook, you need to create a Kubernetes resource of kind MutatingWebhookConfiguration or ValidatingWebhookConfiguration. Here are the critical parts of these configurations:

  • webhooks: This is a list of webhook definitions. Each webhook definition includes:
    • name: A unique name for the webhook.
    • clientConfig: Specifies how the API server connects to the webhook service. This can be done in two ways:
      • service: References a Kubernetes Service that exposes the webhook endpoint. This is the most common approach. It includes:
        • namespace: The namespace of the service.
        • name: The name of the service.
        • path: The URL path within the service where the webhook endpoint is located (e.g., /validate).
        • port: The port number the service is listening on.
      • url: Provides a direct URL to the webhook endpoint. This is used less frequently, primarily for externally hosted webhooks. It must be an HTTPS URL.
    • rules: Defines which API requests the webhook should intercept. It includes:
      • operations: A list of operations (e.g., “CREATE”, “UPDATE”, “DELETE”, “CONNECT”).
      • apiGroups: A list of API groups (e.g., “”, “apps”, “networking.k8s.io”).
      • apiVersions: A list of API versions (e.g., “v1”, “v1beta1”).
      • resources: A list of resources (e.g., “pods”, “deployments”, “services”).
      • scope: Specifies the scope of resources. Options include:
      • *: All resources.
      • Cluster: Cluster-scoped resources.
      • Namespaced: Namespace-scoped resources.
    • failurePolicy: Defines what happens if the API server cannot contact the webhook. Options are:
      • Ignore: The API server ignores the failure and proceeds with the request.
      • Fail: The API server rejects the request.
    • matchPolicy: Specifies how the rules field should be interpreted. Options are:
      • Exact: The request must match the rule exactly.
      • Equivalent: The request may be converted to match the rule (e.g., a v1beta1 request might be converted to v1).
    • namespaceSelector: A label selector that filters the namespaces for which the webhook should be invoked. This allows you to apply webhooks selectively to specific namespaces.
    • objectSelector: A label selector that filters the objects for which the webhook should be invoked, within the selected namespaces (if namespaceSelector is used).
    • timeoutSeconds: The maximum amount of time the API server will wait for a response from the webhook. The default is 10s, and the maximum is 30s.
    • sideEffects: Declare if the webhook has side effects. Possible values are: None, NoneOnDryRun, Some and Unknown. Webhooks with side effects MUST implement a reconciliation system, because the apiserver does not guarantee that the webhook will be called.
    • admissionReviewVersions: A list of AdmissionReview versions the webhook accepts, e.g. v1, v1beta1. The Kubernetes API server will use the first version in the list it supports.

4. Example: Mutating Webhook (Injecting a Sidecar)

Let’s create a simplified example of a mutating webhook that injects a simple sidecar container into every Pod created in the default namespace.

First, the webhook server (simplified Python/Flask example):

“`python
from flask import Flask, request, jsonify

app = Flask(name)

@app.route(‘/mutate’, methods=[‘POST’])
def mutate():
admission_review = request.json
request_object = admission_review[‘request’][‘object’]

if request_object['kind'] == 'Pod':
    # Inject a sidecar container
    if 'containers' not in request_object['spec']:
        request_object['spec']['containers'] = []

    request_object['spec']['containers'].append({
        'name': 'sidecar-container',
        'image': 'busybox:latest',
        'command': ['sleep', '3600']
    })

    admission_response = {
        'apiVersion': 'admission.k8s.io/v1',
        'kind': 'AdmissionReview',
        'response': {
            'uid': admission_review['request']['uid'],
            'allowed': True,
            'patchType': 'JSONPatch',
            'patch':  b64encode(json.dumps([
                {"op": "replace", "path": "/spec/containers", "value": request_object['spec']['containers']}
            ]).encode()).decode() # Base64 encoded patch

        }
    }
    return jsonify(admission_response)
else:
    return jsonify({'response': {'allowed': True}}) # Allow other kinds

if name == ‘main‘:
# In a real-world scenario, you’d use HTTPS with valid certificates.
app.run(host=’0.0.0.0’, port=443, ssl_context=(‘cert.pem’, ‘key.pem’))

import json
from base64 import b64encode

“`

This simple server listens for POST requests on /mutate. It expects an AdmissionReview object, checks if the request is for a Pod, and if so, appends a sidecar container definition to the spec.containers array. Crucially, it uses a JSONPatch to modify the object. The patch is base64 encoded.

Next, the MutatingWebhookConfiguration:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: sidecar-injector
webhooks:
- name: sidecar-injector.example.com
clientConfig:
service:
name: webhook-service
namespace: default
path: /mutate
port: 443
caBundle: <BASE64_ENCODED_CA_BUNDLE> # Replace with the base64 encoded CA bundle.
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
scope: "Namespaced"
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: default # Apply only to the default namespace
sideEffects: None
admissionReviewVersions: ["v1", "v1beta1"]
timeoutSeconds: 5

This configuration defines a webhook named sidecar-injector.example.com that targets Pod creation (CREATE operation) in the default namespace. It points to a service named webhook-service in the default namespace. The caBundle field contains the base64-encoded CA certificate used by the webhook server’s TLS certificate. You can generate this CA bundle using openssl.

Finally, you would need a Deployment and Service for your webhook server:

“`yaml

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
name: webhook-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: webhook-server
template:
metadata:
labels:
app: webhook-server
spec:
containers:
– name: webhook-server
image: # Replace with your Docker image
ports:
– containerPort: 443
volumeMounts:
– name: webhook-certs
mountPath: /certs
readOnly: true
volumes:
– name: webhook-certs
secret:
secretName: webhook-server-certs

Service

apiVersion: v1
kind: Service
metadata:
name: webhook-service
namespace: default
spec:
selector:
app: webhook-server
ports:
– protocol: TCP
port: 443
targetPort: 443
“`

This Deployment and Service expose your Python webhook server. The Secret webhook-server-certs should contain tls.crt and tls.key with your server’s certificate and private key.

5. Best Practices and Considerations

  • HTTPS is Mandatory: All communication between the API server and the webhook endpoint must be over HTTPS. You need to provide valid TLS certificates.
  • Idempotency: Webhooks should be idempotent. The API server might call the webhook multiple times for the same request (e.g., due to retries). The webhook should produce the same result regardless of how many times it’s called.
  • Failure Policy: Carefully consider your failurePolicy. Ignore is generally safer for initial testing and development, but Fail is often preferred in production to ensure policies are enforced.
  • Performance: Webhooks add latency to API requests. Keep your webhook logic as efficient as possible. Avoid unnecessary external calls or complex computations.
  • Testing: Thoroughly test your webhooks. Use a dedicated testing namespace and create various resources to ensure your webhook behaves as expected.
  • Resource Limits: Set resource requests and limits for your webhook deployments to prevent them from consuming excessive resources.
  • Monitoring: Monitor your webhook deployments for errors and performance issues.
  • Reconciliation (for Side Effects): If your webhook has side effects (e.g., it creates other resources), implement a reconciliation loop (e.g., using a Kubernetes controller) to ensure that the desired state is maintained even if the webhook fails or is temporarily unavailable.
  • AdmissionReview Versions: Ensure your webhook server and the admissionReviewVersions in your configuration are compatible.

6. Conclusion

Kubernetes webhooks are a vital tool for any DevOps engineer working with Kubernetes. They provide a flexible and powerful way to customize the behavior of the API server, enabling you to enforce policies, automate tasks, and build more robust and secure Kubernetes deployments. By understanding the basics of mutating and validating webhooks, their configuration, and best practices, you can leverage this capability to significantly enhance your Kubernetes workflows.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top