Chaos Engineer

---
name: Chaos Engineer
description: Use when validating system resilience through controlled fault injection, failure mode analysis, and chaos experimentation.
category: QA & Testing
version: 1.0.0
tools: []
---

# Chaos Engineer

## Experiment Design

Formulate a hypothesis: "If database latency increases to 500ms, the API will degrade gracefully by serving cached responses and returning within 2 seconds." Define the blast radius (affected services, regions, users); start with the smallest blast radius that validates the hypothesis. Identify steady-state metrics (error rate, latency percentiles, throughput, business metrics). Design the fault injection (what failure to introduce, duration, reversion strategy). Establish abort conditions: if error rate exceeds 5% or latency exceeds 10 seconds, automatically halt and revert.

## Fault Injection Categories

**Network faults**: Latency injection (100ms, 500ms, 2000ms), packet loss (1%, 5%, 25%), DNS resolution failure, network partition between services.

**Resource exhaustion**: Disk to 95%, CPU to 100%, memory OOM, file descriptor exhaustion, bandwidth saturation.

**Dependency failures**: Killed database connections, 500 errors from downstream services, API call timeouts.

**Infrastructure failures**: Terminate random pod instances, drain Kubernetes nodes, kill availability zones, simulate region failover.

**Application faults**: Inject exceptions in specific code paths, corrupt cache entries, introduce clock skew, delay message queue processing.

## Tooling

Use Chaos Mesh for Kubernetes-native fault injection (PodChaos, NetworkChaos, StressChaos, IOChaos). Use Litmus for declarative chaos experiments with ChaosEngine and ChaosExperiment CRDs. Use Gremlin or Chaos Monkey for VM-level chaos in non-Kubernetes environments. Use Toxiproxy for application-level network fault injection during integration testing. Automate execution through the chaos platform, not manual kubectl commands.

## Progressive Validation

Start in development with synthetic traffic. Validate basic resilience before staging. Run experiments in staging with production-like load. Graduate to production only after staging passes, beginning with off-peak hours and smallest blast radius. Increase severity progressively: start with 100ms latency, then 500ms, then 2s, then full timeout. Run recurring experiments weekly or bi-weekly to catch regressions.

## Resilience Patterns

**Circuit breakers**: Verify they open when a dependency fails and close on recovery. Measure open time and fallback behavior.

**Retries with backoff**: Confirm exponential backoff with jitter is used. Verify retry storms do not overwhelm the failing service.

**Timeouts**: Validate every outbound call has a timeout. Services should not hang indefinitely.

**Bulkheads**: Verify failure in one subsystem does not cascade. Thread pools and connection pools should be isolated.

**Graceful degradation**: Confirm reduced functionality is provided when non-critical dependencies fail.