chaos-engineering

---
name: chaos-engineering
description: Use when planning, running, or reviewing a chaos experiment: fault injection, gameday, blast radius, steady state, abort criteria, Chaos Toolkit, Chaos Mesh, Litmus, Gremlin, or AWS FIS.
category: Engineering
version: 1.0.0
tools: []
---

# Chaos Engineering

Design experiments that surface real weaknesses in production systems — without becoming outages. Most "chaos engineering" attempts skip steady-state measurement, define no abort criteria, and have no blast-radius bound. Enforce the discipline that makes chaos experiments safe and useful.

## When to Use

- Planning a chaos experiment (what to break, where, when, how to abort)
- Calculating blast radius before running the experiment
- Reviewing an existing experiment plan for safety
- Choosing a chaos tool (Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS)
- Writing a chaos experiment postmortem
- Running a Game Day exercise

## When Not to Use

- General incident response, once something has already broken
- Threat hunting or red-team exercises
- Performance load testing — a different goal; chaos targets failure modes, not capacity
- Production debugging after the fact — chaos discovers weaknesses preemptively

## Core Principle: Chaos Without Abort Criteria Is an Outage

The 4 Principles of Chaos Engineering (Netflix, 2016):

1. **Build a hypothesis around steady-state behavior.** Not "what breaks?" but "X holds; will it still hold under fault Y?"
2. **Vary real-world events.** Inject realistic failures: kill nodes, slow networks, lose cache, throttle dependencies.
3. **Run experiments in production.** Staging never has the same failure modes. Start small.
4. **Automate experiments to run continuously.** One-off chaos is a press release; continuous chaos is engineering.

Add a fifth: **define abort criteria up front.** A chaos experiment with no abort criteria is an outage by another name.

## Designing an Experiment

Every experiment plan must contain: a hypothesis, the steady-state metric it will be measured against, the attack type and magnitude, the duration, the blast radius, concrete abort criteria, and a rollback procedure.

Example hypothesis: "p99 latency stays <500ms when payment-svc is slow." Example magnitude: inject +200ms of latency. Example abort criteria: abort if "p99 > 1000ms OR error_rate > baseline + 1pp." Concrete thresholds like these — not vague language — are what make an abort criterion enforceable during a live experiment.

## Calculating Blast Radius

Before running anything, estimate blast radius from traffic share, user population, and duration: expected affected users, expected error-budget burn, and a risk score.

- **GREEN** = under 1% of error budget consumed — proceed
- **YELLOW** = 1-10% of error budget consumed — reduce scope before proceeding
- **RED** = over 10% of error budget consumed — abort/redesign the experiment

Recommendation follows the score: PROCEED on GREEN, REDUCE on YELLOW, ABORT on RED.

## Writing the Postmortem

A good postmortem records: a summary, whether the hypothesis was confirmed or refuted, what was learned, what surprised the team, follow-up actions with owners, and a link to the next experiment. Common postmortem failure modes to avoid: no learning recorded, no follow-up actions, and blame-laden language.

## The 7 Attack Types

Different attacks reveal different weaknesses — pick the one that matches the hypothesis. "What happens if X is slow?" → latency. "What happens if X loses network?" → partition.

| Attack | What It Tests | Typical Tooling |
|---|---|---|
| **Latency** | Timeouts, retries, circuit breakers | Chaos Mesh `NetworkChaos` |
| **Error** | Error handling, fallback paths | Chaos Mesh `HTTPChaos`, Toxiproxy |
| **Resource** (CPU, memory, disk) | Saturation handling, autoscaling | Chaos Mesh `StressChaos` |
| **Network partition** | Split-brain, consensus, failover | Chaos Mesh `NetworkChaos` partition |
| **Dependency failure** | Graceful degradation, fallback | Service mesh fault injection |
| **Time** | Clock skew, NTP issues | Chaos Mesh `TimeChaos` |
| **Infrastructure** (kill instance) | Auto-recovery, failover | AWS FIS, Chaos Monkey |

## Choosing a Tool

| Tool | Best For | Pricing | Stack |
|---|---|---|---|
| **Chaos Toolkit** | Lightweight, language-agnostic, JSON experiments | OSS | Any |
| **Chaos Mesh** | Kubernetes-native, rich CRDs, in-cluster | OSS | Kubernetes |
| **Litmus** | Kubernetes, Argo-integrated, large library | OSS + Enterprise | Kubernetes |
| **Gremlin** | Enterprise SaaS, multi-cloud, audit | Paid | Any |
| **AWS FIS** | AWS-native, IAM-integrated, EC2/ECS/EKS | Paid (AWS) | AWS |

Decision rules: k8s-only stack + OSS → Chaos Mesh or Litmus (Litmus has the bigger experiment library); multi-cloud + OSS → Chaos Toolkit; AWS-heavy + simple needs → AWS FIS; enterprise + audit/compliance → Gremlin.

## Workflow: Design and Run a Single Experiment

1. State a hypothesis: "When [fault], steady-state metric X stays within Y."
2. Identify the steady-state metric — it must be measurable BEFORE the experiment.
3. Calculate blast radius — confirm GREEN before proceeding.
4. Draft the experiment plan (hypothesis, attack, magnitude, duration, blast radius, abort criteria, rollback).
5. Get a peer review; confirm abort criteria are concrete.
6. Notify the on-call team.
7. Run the experiment with monitoring open.
8. If abort criteria are hit, abort immediately and record what happened.
9. Write the postmortem to capture learnings.
10. File follow-up actions; link to the next experiment.

## Workflow: Game Day Exercise

1. Pick a scenario (e.g., "primary database fails over").
2. Identify all dependent services that should keep working.
3. Build a multi-experiment plan covering each layer.
4. Schedule with stakeholders; require on-call coverage.
5. Run with a facilitator who manages the scenario.
6. Capture observations in a shared doc as they happen.
7. Write a single combined postmortem covering all observations.
8. Track follow-up actions with owners.

## Workflow: From Game Days to Continuous Chaos

1. Start: weekly Game Day in staging.
2. Move to: weekly Game Day in production with limited blast radius.
3. Mature to: continuous chaos via scheduled experiments.
4. Wire to deployment: every production deploy triggers a baseline chaos sweep.
5. Track experiments per week, weaknesses discovered, and MTTR trend.

## Composing With Other Practices

Kill switches used for feature rollback double as abort triggers here. Operators and reconciliation loops are common chaos targets. A chaos experiment that escalates becomes a regular incident and should be handed to whoever owns incident response.

## Anti-Patterns

- **No hypothesis** — "let's break things" is sabotage, not engineering
- **No steady-state metric** — without a baseline, you can't tell if X broke
- **No blast-radius bound** — a full-prod experiment without limits is an outage
- **No abort criteria** — mandatory, not optional
- **No on-call coverage** — chaos without monitoring is unmonitored production
- **Chaos in staging only** — staging never has production failure modes
- **One-off chaos** — a single experiment is a press release; learning requires recurrence
- **Blame-laden postmortem** — record causes, not blame; teams stop running chaos otherwise

## Verifiable Success

A team following this discipline should see: 100% of chaos experiments with a written hypothesis, abort criteria, and blast-radius calculation; blast radius for any single experiment never exceeding 10% of error budget; mean time between chaos experiments under 14 days; every experiment producing at least one follow-up action that ships; and no chaos experiment escalating to a customer-impacting incident in the trailing 90 days.

Present experiment plans, blast-radius calculations, and postmortems directly in the chat as structured markdown; @mention the on-call teammate before running anything in production.