kubernetes-operator

---
name: kubernetes-operator
description: Use when building a Kubernetes Operator (custom controller reconciling CRD state). Covers CRD design, reconcile-loop correctness, framework choice, and OperatorHub capability levels.
category: Engineering
version: 1.0.0
tools: []
---

# Kubernetes Operator

Guide teams building operators — custom controllers that reconcile CRD state — so they reconcile correctly. Most operator bugs are not Kubernetes bugs; they're reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants. Catch these deterministically before they reach a cluster.

## When to use / not use

Use for: building a new operator (controller for a CRD), reviewing an existing operator for capability-level gaps, auditing a CRD spec for status/conditions/finalizer correctness, choosing a framework, designing a Custom Resource's API surface, or hardening RBAC/leader election/webhook validation.

Not for: plain Helm chart packaging, standard kubectl operations or blue-green deploys, general k8s security posture, or "I want to run a workload" (that's a Deployment or Job, not an operator).

## Core principle: an operator is a reconcile loop, not a script

The loop: observe actual state → read desired state from spec → diff actual vs. desired → act → update status → requeue or done.

Operators fail when they:
1. Treat reconcile as imperative ("do this, then this, then this") instead of declarative (make actual = desired, idempotently)
2. Don't requeue transient failures
3. Skip finalizers, leaving orphan resources
4. Mutate spec instead of status
5. Skip the status subresource (status updates then trigger spec reconciles → infinite loop)
6. Block in reconcile (long HTTP calls, locks)
7. Forget leader election → split-brain on multi-replica deploys

## Reviewing a CRD design

Walk the design against these checks rather than assuming a linter is available: every version declares `subresources.status`; `spec.scope` is `Namespaced` rather than `Cluster` unless cluster scope is explicitly justified; singular and listKind are defined; the OpenAPI v3 schema defines real types (no `x-kubernetes-preserve-unknown-fields: true` at the top level); exactly one version is marked `served: true` AND `storage: true`; the schema includes a Conditions array (to support `metav1.Conditions`); printer columns include `Age` and `Status`/`Phase` for `kubectl get`.

## Reviewing a Go reconcile function

Check for these anti-patterns: returns not shaped `(ctrl.Result, error)`; errors that don't trigger a non-zero requeue (`ctrl.Result{Requeue: true}, err`); `client.Update()` called on the spec object instead of `Status().Update()`; `time.Sleep` inside reconcile (blocks other reconciles — use `RequeueAfter`); HTTP calls without context cancellation; a finalizer add without a matching `defer`; conditions defined in the CRD schema but never set via `IsConditionTrue`/`SetCondition` (status drift); a reconcile function over roughly 80 lines that should be split into subroutines.

## Scoring operator maturity: OperatorHub Capability Levels

| Level | Name | What it requires |
|---|---|---|
| L1 | Basic Install | CRD defined, controller deploys it |
| L2 | Seamless Upgrades | PDBs, conversion webhooks, version skew strategy |
| L3 | Full Lifecycle | Backups, restores, failure recovery |
| L4 | Deep Insights | Metrics endpoint, Prometheus rules, alerts |
| L5 | Auto Pilot | Auto-scaling, auto-tuning, anomaly detection |

Assess current level from what's implemented, then name concrete next steps to advance one level at a time.

## Tooling landscape

| Framework | Language | Best for | Maintenance |
|---|---|---|---|
| controller-runtime | Go | Production-grade, low-level control | Active (sig-api-machinery) |
| kubebuilder | Go | Standard scaffolding, opinionated | Active (Kubernetes SIGs) |
| operator-sdk | Go / Helm / Ansible | OpenShift / mixed-paradigm teams | Active (Red Hat) |
| metacontroller | Any (webhook-based) | Polyglot teams, avoiding Go | Less active |
| KOPF | Python | Python shops, async-first | Active (community) |
| java-operator-sdk | Java | JVM shops | Active (Red Hat / Java SIG) |

Decision rules:
- New operator + Go shop → kubebuilder
- New operator + Python shop → KOPF
- New operator + can't pick a language → metacontroller
- OpenShift target → operator-sdk

## CRD design principles

Status is the source of truth for the controller's view of the world — spec is what the user wants, status is what the controller observed. Use the status subresource (without it, status updates re-trigger reconcile, causing an infinite loop). Use Conditions (`Ready`, `Reconciling`, `Degraded`, each with a reason and message). Add finalizers, or deletion races the controller and orphans external resources. Version the CRD from day one (`v1alpha1` → `v1beta1` → `v1`) and plan a conversion webhook. Validate via the OpenAPI v3 schema rather than relying on the controller to reject bad input. Use `additionalPrinterColumns` so `kubectl get` shows at minimum `Age`, `Phase`, `Ready`. Namespace CRDs unless they manage cluster-scoped resources.

## Reconcile loop principles

Reconcile must be idempotent — reconciling the same state twice gives the same result with zero side effects. Read once, decide, act; don't re-observe the world repeatedly mid-reconcile. Update status, not spec — spec belongs to the user. Return errors that requeue, using `ctrl.Result{RequeueAfter: ...}` for known transient cases. Never block: no `time.Sleep`, no long HTTP calls without context. Read via the controller's cached client, only bypassing the cache for a specific, documented reason. Leader-elect when running more than one replica; otherwise run single-replica mode. Set OwnerReferences so cascading deletion — the operator pattern's built-in benefit — works.

## Workflows

**Bootstrap a new operator (Go + kubebuilder):** pick a Group/Version/Kind (e.g. `apps.example.com/v1alpha1`, kind `MyApp`); scaffold the project and the API; review the generated CRD against the design checks above and fix every warning before writing controller code; implement the reconcile function starting with the simplest correct version; review it against the reconcile anti-pattern checks; assess capability level and confirm L1; test in a sandbox cluster; add status conditions and aim for L2 in the same change.

**Audit an existing operator:** run the capability-level assessment, the CRD design review, and the reconcile-function review; triage findings into FAIL (block release, fix before next deploy) vs. WARN (file an issue, fix within 30 days); document the current capability level; plan one capability-level advancement per quarter.

**Choose a framework:** identify the team's primary language constraint, the deployment target (vanilla Kubernetes vs. OpenShift), and the operator's complexity (single CRD vs. multi-CRD vs. cluster-wide); cross-reference against the tooling table above; build a one-week proof-of-concept before committing.

## Anti-patterns

- `time.Sleep(30 * time.Second)` inside reconcile — blocks other reconciles; use `RequeueAfter`.
- `r.Client.Update(ctx, obj)` to set status — use `r.Status().Update(ctx, obj)` instead.
- No leader election with 2+ replicas — causes split-brain.
- No finalizer — external resources orphan on deletion.
- CRD without a status subresource — status updates trigger spec reconciles (infinite loop).
- Reconcile function over ~200 lines — extract `reconcileXxx` subroutines per condition.
- `x-kubernetes-preserve-unknown-fields: true` on the spec root — defeats validation.
- Imperative reconcile ("if creating, do A; if updating, do B; if deleting, do C") — wrong shape. Reconcile should make actual = desired, regardless of how the state was reached.

## Verifiable success

A team applying this skill should see: every new CRD reviewed against the design checks before merge; every reconcile function reviewed against the anti-pattern checklist; operators reaching OperatorHub Capability Level 3 (Full Lifecycle) before public release; mean time to fix a reconcile bug under 1 day, with no infinite loops in production.