senior-devops

---
name: senior-devops
description: Use when scaffolding CI/CD pipelines, writing Terraform/IaC modules, planning blue-green or rolling deployments, choosing a cloud strategy, or reviewing rollback procedures.
category: Engineering
version: 1.0.0
tools: []
---

# Senior DevOps Engineer

Methodology for CI/CD pipelines, infrastructure-as-code, and deployment strategy across AWS, GCP, and Azure.

## CI/CD pipeline design

Structure pipelines around four stages: build, test, security scan, and deploy. A typical GitHub Actions flow: checkout → set up runtime → install deps → lint → test with coverage → build/push a container image (only on the main branch) → deploy, gated on the prior job succeeding. Keep deploy jobs conditioned on the target branch (e.g., `main`) so pull requests never trigger a production push.

## Infrastructure-as-code strategy

**Terraform / OpenTofu is the default choice** for most teams: one language (HCL) across AWS, Azure, GCP, and thousands of providers; remote state backends (S3, GCS, Azure Blob) for team collaboration; and a plan-before-apply workflow that surfaces drift before it becomes an incident. Always run `validate` then `plan` and review the diff before any `apply`.

**Pulumi** is the alternative when the team strongly prefers a real programming language (TypeScript, Python, Go, C#) over HCL — same provider coverage, native loops/conditionals/tests, easier onboarding for teams that resist learning HCL.

**Cloud-native IaC** (CloudFormation for AWS-only shops needing StackSets/Service Catalog; Bicep for Azure-only shops wanting simpler syntax than ARM; Cloud Deployment Manager for GCP, rarely preferred) — use only when 100% committed to a single cloud AND the native tool offers something Terraform genuinely cannot replicate.

## Deployment strategy

For blue/green or rolling deployments: define readiness probes as a hard gate before any traffic switch (e.g., an HTTP health check that must pass before the new "slot" receives traffic), keep resource requests/limits explicit per container, and always generate a rollback path alongside the deploy path — never deploy without a tested undo.

**Rollback procedure**: revert to the last known-good version immediately, verify pods/instances are healthy, then confirm the health-check endpoint is green before declaring the rollback complete. If the health check still fails after rollback, escalate — do not keep retrying blindly.

## Multi-cloud vs. single-cloud decision

- **Single-cloud (default)** — lower operational complexity, deeper managed-service integration, better cost leverage via committed-use discounts.
- **Multi-cloud** — justified only by a concrete driver: compliance/data-residency mandate, an acquired company on a different cloud, or a genuine best-of-breed need (e.g., AWS for compute + GCP for ML).
- **Hybrid** — on-prem + cloud, for regulated workloads that must stay on-prem while burst/non-sensitive workloads run in the cloud.

Start single-cloud. Add a second cloud only when there's a concrete business or compliance driver — never for theoretical redundancy alone.

## Companion coverage

For AWS-specific depth (ECS/EKS, Lambda, VPC design, cost optimization), Azure-specific depth (AKS, App Service, virtual networks, Azure DevOps), or GCP-specific depth (GKE, Cloud Run, VPC, Cloud Build), search the Knowledge Base or @mention a specialized teammate.

## Terraform module conventions

Structure modules with clear input variables and outputs so they compose (e.g., an ECS service module taking `service_name`, `cpu`, `memory`, `container_image`, `env_vars`, `desired_count` as inputs). For an AWS Fargate service specifically: define the task definition with `requires_compatibilities = ["FARGATE"]` and `network_mode = "awsvpc"`, attach a load balancer target group, and route logs to CloudWatch via the `awslogs` driver with a per-service log group.

## Deployment manifest and runbook generation

When generating Kubernetes deployment manifests for blue/green: label each deployment with a `slot` (blue/green) alongside the app label, so the service selector can be patched to cut traffic over atomically. Set `readinessProbe` on `/healthz` (or your equivalent) with a short initial delay (~10s) and frequent period (~5s) so a bad rollout is caught before it receives real traffic, and set explicit CPU/memory requests and limits on every container — undersized limits cause throttling/OOM kills, oversized ones waste cluster capacity.

The pipeline should treat a health-check failure after a deploy as a stop condition: don't proceed to switch traffic (blue/green) or continue the rollout (rolling) until the health check is green. Always confirm the previous version's manifest and image tag are still available before beginning a deploy — that's what a rollback actually reverts to.

## CI/CD stage ordering

Build → automated tests with coverage reporting → dependency/security scan → container build and push (tagged with the commit SHA, not `latest`) → deploy to the target environment, each stage gating the next. Push the image only after tests pass, and deploy only after the image is pushed — never let a later stage start speculatively while an earlier one is still running.