release-it

---
name: release-it
description: Use when designing resilient production systems or debugging outages: circuit breakers, bulkheads, timeouts, retries, health checks, chaos engineering, capacity planning, zero-downtime deployments.
category: Business Knowledge
version: 1.0.0
tools: []
---

# Release It! Stability Framework

Design, deploy, and operate production-ready software that expects and survives failure at every level. The software that passes QA is not the software that survives production — production is hostile, and systems must handle failure gracefully rather than collapse catastrophically.

## Core Principle

Every system will eventually be pushed beyond its design limits. The question is not whether failures happen, but whether the system degrades gracefully or fails catastrophically. Production-ready software is resilient, observable, and operates through partial failures without human intervention.

## Scoring

Goal: 8/8. Score a production system with 1 point per "yes" across 8 checks: timeouts, circuit breakers, bulkheads, zero-downtime deploy, deep health checks, correlated telemetry, load-tested past peak, failure injection. Bands: 7-8 = every integration point is bounded, isolated, observable, and deploy/release are decoupled; 4-5 = some patterns present but 3+ diagnostic rows fail (unbounded retries, shared pools, shallow health checks); 2 or below = relies on the happy path, no breakers, no capacity model, no failure testing. Always state the current score, the failing rows, and the specific fix for each.

## 1. Stability Anti-Patterns

Failures propagate through integration points and cascade across system boundaries. The most dangerous patterns are emergent behaviors when systems interact under stress, not bugs in isolated code — audit every integration point by name and close the specific crack rather than hardening at random.

Key insights: integration points (sockets, HTTP calls, queues) are the number-one killer; slow responses are worse than no response because they tie up threads and exhaust pools; unbounded result sets turn a harmless query into an out-of-memory crash once data outgrows test assumptions; users generate load no test predicts (bots, retry storms, flash crowds, "self-denial attacks" from your own marketing); blocked threads are the silent killer — deadlocks and contention show no errors until everything stops.

Apply: assume every remote call can fail, hang, or return garbage — wrap external calls with timeout + circuit breaker. Enforce result-set limits with LIMIT and pagination on list endpoints. Isolate thread/connection pools per dependency (separate pool for payment gateway vs. search). Coordinate marketing launches with capacity planning — pre-scale before Black Friday, queue coupon redemptions.

## 2. Stability Patterns

Counter each anti-pattern with a stability pattern: circuit breakers stop cascades, bulkheads isolate blast radius, timeouts reclaim stuck resources. Together they make a system bend under load instead of breaking. Treat a tripped breaker as expected output, not an incident — page on the breaker staying open, not on it opening.

Key insights: Circuit Breaker has three states (closed, open, half-open) — trips after a failure threshold, periodically tests recovery. Every outbound call needs connect AND read timeouts, propagated up the call chain. Retry with exponential backoff + jitter prevents thundering herd on recovery. Fail Fast rejects requests known to fail instead of wasting resources; Handshaking lets the server decline work before it's sent. Steady State: systems accumulate cruft (logs, sessions, temp files) — design automatic cleanup. Let It Crash: a clean restart often beats limping along in an unknown state.

Apply: Circuit Breaker opens after 5 failures in 60s, half-opens after 30s. Bulkhead: dedicated connection pools for critical vs. non-critical work. Timeout with propagation: connect 1s, read 5s, propagate deadline downstream. Retries: base 100ms backoff, max 3 retries, 20% fleet-wide retry budget. Steady State: purge sessions older than 24h, rotate logs at 500MB.

## 3. Capacity and Availability

Capacity is not one number — it is a multi-dimensional function of CPU, memory, network, disk I/O, connection pools, and threads. Capacity planning means knowing which resource bottlenecks first, and at what load. Untested systems fail at peak load, the worst possible moment; knowing actual (not theoretical) limits lets you set realistic SLAs and scale before users hit the wall.

Key insights: test taxonomy — load test (expected traffic), stress test (beyond limits), soak test (sustained, catches leaks), spike test (sudden bursts). The Universal Scalability Law says throughput never scales linearly: contention and coherence costs cause diminishing returns. Pool exhaustion looks identical to a database outage from the application's perspective — size pools from measured concurrency, not defaults. "The cloud is infinitely scalable" is a myth: auto-scaling has lag, cold starts, and hard limits.

Apply: ramp load to peak then 2x and observe degradation. Size connection pools to measured P99 active connections + 20% headroom. Soak test at 80% capacity for 24-72 hours to catch memory/connection/file-handle leaks. Document the bottleneck per service (e.g. "Service X is memory-bound at 2000 RPS; 4GB per instance").

## 4. Deployment and Release

Deployment (putting code on servers) and release (exposing it to users) are separate operations that should be decoupled — deploy without risk, release with confidence. Most outages are caused by changes; decoupling lets you deploy to production, verify, and only then route traffic. If something breaks, you roll back the release, not the deployment.

Key insights: zero-downtime deployment is non-negotiable — rolling, blue-green, or canary. Feature flags dark-launch code independent of deployment. Database migrations must be backward-compatible (expand-contract) since old and new code run simultaneously during deploys. Immutable infrastructure: never patch a running server — build a new image, deploy, destroy the old. Rollback must be faster than roll-forward; if rollback takes 30 minutes, deploys get avoided.

Apply: blue-green with a health-check gate before swapping the router. Canary rollout with automated rollback — 5% traffic to canary, auto-rollback if error rate exceeds 1%. Feature launch: ship behind a flag, enable for 10%, monitor, ramp, with an emergency off switch. Schema changes: add column, write both, backfill, drop old.

## 5. Health Checks and Observability

You cannot operate what you cannot observe. Health checks, metrics, logs, and traces are the sensory organs of a system in production — a first-class design concern, not an afterthought. Emit high-cardinality, structured events (not just pre-aggregated counters) so you can ask new questions of past incidents without shipping new instrumentation first.

Key insights: health checks come in two flavors — shallow (process alive) and deep (dependencies reachable, resources available). Three pillars: structured logs (what happened), metrics (how much), distributed traces (where and how long). RED method for services (Rate, Errors, Duration); USE method for resources (Utilization, Saturation, Errors). Define SLIs (measure user experience) then SLOs (targets) then SLAs (contracts), in that order. Alert on symptoms users feel (error rate, latency), not causes (CPU); a dashboard should answer "is the system healthy?" within 5 seconds.

Apply: `/health` reports DB, cache, queue, and disk status. Instrument RED per endpoint: rate, error rate, p50/p95/p99 latency. Propagate trace context (trace ID in headers) to correlate logs across services. Alert on SLO burn rate ("error budget burning 10x") rather than raw thresholds ("CPU > 80%").

## 6. Adaptation and Chaos Engineering

Safety note: chaos engineering here is a design-time planning activity — what to test and what to verify — not something for an AI agent to execute autonomously. All failure injection must be performed by authorized engineers using dedicated tooling (Gremlin, Litmus, AWS FIS) with approvals, rollback plans, and blast-radius controls.

Confidence in resilience comes from testing under realistic failure conditions. You cannot know how a system handles failure until it actually fails; controlled injection turns unknown-unknowns into known-knowns before they cause real outages.

Key insights: define steady state first — you need a measurable baseline to detect deviation. Every experiment has a hypothesis ("we believe that when X fails, the system will Y"). Start small in non-production (kill one process, add latency to one call), then escalate gradually with approvals. Minimize blast radius with canary populations, feature flags, and an emergency stop. Automate recurring experiments; GameDay exercises test both the system and the team. Build a culture where finding weaknesses is celebrated, not punished.

Apply: kill one pod via chaos tooling and verify recovery within SLO. Inject +500ms latency on DB calls and verify the circuit breaker trips. Simulate a downstream outage (return 503 from a payment API) and verify graceful degradation. Run scheduled GameDay exercises (e.g. "primary DB goes read-only at 2pm") to practice team response.

## Common Mistakes

No timeouts on outbound calls → one slow dependency freezes the system; add connect and read timeouts everywhere. Unbounded retries → retry storms amplify failures; use exponential backoff, jitter, and fleet-wide retry budgets. Shared thread/connection pools → one failing dependency drains everything; bulkhead per dependency. Shallow health checks only → traffic routed to instances with broken dependencies; use deep health checks. Testing only the happy path → works until the first real failure; load, soak, and chaos test before major releases. Coupling deploy and release → every deployment is all-or-nothing high risk; use feature flags, canary, blue-green. Alerting on causes not symptoms → CPU alerts fire while users suffer silently; alert on user-facing SLIs. No capacity model → system falls over at 2x load; model bottlenecks and load test to 3x expected peak.

## Quick Diagnostic

Does every outbound call have a timeout? If not, calls hang and block threads — add connect and read timeouts everywhere. Are circuit breakers on critical dependencies? If not, one failure takes down the system — add breakers with tuned thresholds. Are pools isolated per dependency? If not, failures cross-contaminate — implement bulkheads with dedicated pools. Is deployment decoupled from release? Are health checks deep? Has the system been load-tested past peak and chaos-tested?