observability-designer

---
name: observability-designer
description: Use when adding observability to a new service, fixing noisy alerting, or designing an SLO program — covers SLI/SLO frameworks, golden-signals monitoring, and alert optimization.
category: Engineering
version: 1.0.0
tools: []
---

# Observability Designer

Design comprehensive observability strategies for production systems: SLI/SLO frameworks, alerting optimization, and dashboard specifications across the three pillars — metrics, logs, and traces.

**Scope note**: for SLO/SLI design with full error-budget math, multi-window burn-rate alerting thresholds, and SLO review gates, @mention the teammate covering SLO architecture — this skill covers the quick scaffold plus dashboards and alert-noise reduction.

**Verification loop**: after deploying optimized alerts, track the actionable-alert ratio for one on-call rotation; if it hasn't improved, re-analyze the live alert config and iterate. Confirm every golden-signal panel on a generated dashboard renders with live data before closing the task.

## SLI/SLO/SLA Framework Design

- **Service Level Indicators (SLI):** measurable signals of service health
- **Service Level Objectives (SLO):** reliability targets based on user experience
- **Service Level Agreements (SLA):** customer-facing commitments with consequences
- **Error Budget Management:** calculate and track error budget consumption
- **Burn Rate Alerting:** multi-window burn rate alerts for proactive SLO protection

## Three Pillars of Observability

**Metrics** — Golden Signals (latency, traffic, errors, saturation); **RED Method:** Rate, Errors, and Duration for request-driven services; **USE Method:** Utilization, Saturation, and Errors for resource monitoring; plus business and infrastructure metrics.

**Logs** — structured (JSON) logging with consistent fields; centralized aggregation and indexing; appropriate DEBUG/INFO/WARN/ERROR/FATAL levels; correlation IDs for distributed request tracing; sampling to manage volume on high-throughput systems.

**Traces** — end-to-end distributed request flow visualization; meaningful span boundaries and metadata; intelligent trace sampling for cost/performance; automatic service-map discovery; trace-driven root-cause debugging.

## Dashboard Design Principles

**Information architecture**: Overview → Service → Component → Instance drill-down. **Golden Ratio:** 80% operational metrics, 20% exploratory metrics. Cognitive load cap: 7±2 panels per screen. Role-based personas (SRE, Developer, Executive).

**Visualization**: time series for trends, heatmaps for distributions, gauges for status. **Color Theory:** red for critical, amber for warning, green for healthy states. Overlay SLO targets, capacity thresholds, and historical baselines as reference lines. Default time ranges: 4h for incidents, 7d for trends.

**Panels**: efficient Prometheus/InfluxDB queries with proper aggregation; visual alert-state indicators on relevant panels; template variables and drill-down links; sub-second render times through query optimization.

## Alert Design and Optimization

**Severity**: Critical (service down, high SLO burn rate) / Warning (approaching thresholds, non-user-facing) / Info (deployments, capacity planning). Every alert must have a clear response action, with escalation routed by severity and team ownership.

**Fatigue prevention**: favor precision (few false positives) over recall; use hysteresis (different firing vs. resolving thresholds); suppress dependent alerts during known outages; group related alerts into single notifications.

**Rule design**: choose thresholds with statistical methods; use appropriate averaging windows and percentile calculations; define clear firing and auto-resolution conditions; validate alert rules against historical data.

## Runbook Generation and Incident Response

**Runbook structure**: alert context (what the alert means and why it fired), impact assessment (user-facing vs. internal), ordered investigation steps with time estimates, resolution actions (common fixes and escalation procedures), and post-incident follow-up and prevention tasks.

**Incident detection patterns**: statistical anomaly detection for unusual patterns; composite (multi-signal) alerts for complex failure modes; predictive, trend-based forward-looking alerts on capacity; canary monitoring for early detection during progressive deployment.

## Golden Signals Framework

- **Latency:** P50/P95/P99 response time, queue latency, network latency, database latency
- **Traffic:** requests per second with burst detection, bandwidth usage, active sessions, feature/endpoint usage
- **Errors:** 4xx/5xx rate, error-budget consumption, error-type distribution, detection of silent failures (no HTTP error)
- **Saturation:** CPU/memory/disk/network utilization, queue depth, connection-pool saturation, rate-limit/quota exhaustion

## Distributed Tracing and Log Aggregation

Tracing: head-based, tail-based, and adaptive sampling; context propagation across service boundaries; parent-child span correlation; retention-tuned trace storage; auto- and manual instrumentation, with overhead measured.

Logs: push vs. pull shipping agents; topic-based routing/filtering; structured vs. unstructured parsing; schema versioning; indexed fields for common query patterns; time/volume retention policies; compression and archival; query and cache optimization.

## Cost Optimization

Tiered metric retention by importance; intelligent log and trace sampling to cut ingestion cost; cold storage for historical data; query efficiency; storage-tier matching to data type; ingestion rate limiting; cardinality management for high-cardinality metrics.

## Integration Patterns

**Monitoring stack**: Prometheus for metric collection and alerting rules; Grafana for dashboards; Elasticsearch/Kibana for log analysis; Jaeger/Zipkin for distributed tracing.

**CI/CD**: instrument build/test/deploy pipelines; correlate releases with impact and rollback triggers; monitor feature-flag rollouts and A/B tests; catch performance regressions automatically.

**Incident management**: route alerts through PagerDuty/VictorOps; notify and collaborate via Slack/Teams; track incidents in JIRA/ServiceNow; automate post-mortem analysis and improvement tracking.

## Advanced Patterns

**Multi-cloud**: unified metrics across AWS/GCP/Azure; inter-cloud connectivity monitoring; cost attribution; compliance posture tracking.

**Microservices**: service-mesh observability (Istio/Linkerd); API-gateway request/rate-limit monitoring; Kubernetes cluster and workload monitoring; dynamic service discovery and health checks.

**Machine learning**: model accuracy/drift/bias monitoring; feature-store quality and freshness; ML pipeline execution monitoring; statistical significance and business-impact measurement for A/B tests.

## Scripted Capabilities (methodology)

Three deterministic capabilities back this skill's recommendations: an SLO scaffolder that turns a service's type/criticality/dependencies into SLI definitions, SLO targets, error budgets, and burn-rate alert rules; an alert-config analyzer that flags noise, duplicate rules, and coverage gaps before emitting an optimized config; and a dashboard generator that produces a Grafana-compatible spec with golden-signal coverage, RED/USE method panels, drill-down paths, and role-based views.

## Best Practices

**Organizational**: set SLOs collaboratively between product and engineering; give every alert a clear escalation path and team owner; govern dashboards centrally with shared standards; invest in team training on observability tooling.

**Technical**: version-control observability configuration as code; test alert rules and validate dashboards before rollout; monitor the observability system's own performance; enforce access control and data privacy on telemetry.

**Continuous improvement**: review SLI/SLO effectiveness regularly; keep tuning alert thresholds and routing; evolve dashboards from user feedback; periodically reassess tool effectiveness.