secrets-vault-manager

---
name: secrets-vault-manager
description: Use when setting up secret management infrastructure, integrating HashiCorp Vault, configuring cloud secret stores (AWS/Azure/GCP), implementing secret rotation, or auditing secret access.
category: Engineering
version: 1.0.0
tools: []
---

# Secrets Vault Manager

Production secret infrastructure management for teams running HashiCorp Vault, cloud-native secret stores, or hybrid architectures — policy authoring, auth method configuration, automated rotation, dynamic secrets, audit logging, and incident response.

Operates at the infrastructure layer (Vault clusters, cloud KMS, certificate authorities, CI/CD secret injection) — distinct from local `.env` file hygiene and leak detection, which a teammate covering environment-secrets hygiene handles.

## When to Use

- Standing up a new Vault cluster or migrating to a managed secret store
- Designing auth methods for services, CI runners, and human operators
- Implementing automated credential rotation (database, API keys, certificates)
- Auditing secret access patterns for compliance (SOC 2, ISO 27001, HIPAA)
- Responding to a secret leak requiring mass revocation
- Integrating secrets into Kubernetes workloads or CI/CD pipelines

## HashiCorp Vault Patterns

**Architecture decisions**: HA deployment with Raft storage (no external dependency, built-in leader election); auto-unseal via cloud KMS (eliminates manual unseal, enables automated restarts); one namespace per environment for blast-radius isolation; dual audit devices (file + syslog) — Vault refuses requests if all audit devices fail, so dual configuration prevents outages.

**Auth methods**: AppRole for machine-to-machine service/batch-job authentication (short token TTL, single-use secret IDs); Kubernetes auth for pod-native authentication via service-account tokens; OIDC for human operator access through an SSO provider (Okta, Azure AD, Google Workspace).

**Secret engines** — pick TTL strategy by type:

| Engine | Use Case | TTL Strategy |
|--------|----------|-------------|
| KV v2 | Static secrets (API keys, config) | Versioned, manual rotation |
| Database | Dynamic DB credentials | 1h default, 24h max |
| PKI | TLS certificates | 90d leaf certs, 5y intermediate CA |
| Transit | Encryption-as-a-service | Key rotation every 90d |
| SSH | Signed SSH certificates | 30m interactive, 8h automation |

**Policy design**: least-privilege, path-based granularity, explicit deny on admin paths (`sys/*`); naming convention `{service}-{access-level}` (e.g., `payment-service-read`).

## Cloud Secret Store Integration

### Comparison Matrix

| Feature | AWS Secrets Manager | Azure Key Vault | GCP Secret Manager |
|---------|--------------------|-----------------|--------------------|
| Rotation | Built-in Lambda | Custom via Functions | Cloud Functions |
| Versioning | Automatic | Manual or automatic | Automatic |
| Encryption | AWS KMS (default or CMK) | HSM-backed | Google-managed or CMEK |
| Access control | IAM + resource policy | RBAC + Access Policies | IAM bindings |
| Cross-region | Replication supported | Geo-redundant by default | Replication supported |
| Audit | CloudTrail | Azure Monitor + Diagnostic Logs | Cloud Audit Logs |

**When to use which**: AWS Secrets Manager for out-of-the-box RDS/Aurora rotation on AWS; Azure Key Vault for certificate-management strength and Azure AD-integrated workloads; GCP Secret Manager for the simplest API surface on GKE with Workload Identity; HashiCorp Vault for multi-cloud, dynamic secrets, PKI, and transit encryption in complex or hybrid environments.

**SDK access principle**: always fetch secrets at startup or via sidecar — never bake them into images or config files. Each cloud SDK (boto3, google-cloud-secretmanager, azure-keyvault-secrets) exposes a simple get-by-name/version call for this.

## Secret Rotation Workflows

| Secret Type | Frequency | Method | Downtime Risk |
|-------------|-----------|--------|---------------|
| Database passwords | 30 days | Dual-account swap | Zero (A/B rotation) |
| API keys | 90 days | Generate new, deprecate old | Zero (overlap window) |
| TLS certificates | 60 days before expiry | ACME or Vault PKI | Zero (graceful reload) |
| SSH keys | 90 days | Vault-signed certificates | Zero (CA-based) |
| Service tokens | 24 hours | Dynamic generation | Zero (short-lived) |
| Encryption keys | 90 days | Key versioning (rewrap) | Zero (version coexistence) |

**Dual-account database rotation**: two accounts exist (`app_user_a`/`app_user_b`); rotate the inactive account's password and update the secret store; the application picks it up on next fetch; after a grace period, rotate the now-inactive account; repeat.

**API key overlap window**: generate the new key, store it as `current` while the old moves to `previous`; deploy so applications read `current`; after all instances restart (or TTL expires) and monitoring confirms zero usage of the old key, revoke `previous`.

## Dynamic Secrets

Prefer dynamic, on-demand, auto-expiring secrets over static credentials wherever possible: Vault-issued database credentials scoped to a role and TTL; short-lived cloud IAM credentials (AWS/Azure/GCP) instead of long-lived keys; a Vault-backed SSH certificate authority issuing short-TTL (30-minute) signed certificates instead of distributing `authorized_keys`.

## Audit Logging

| Event | Priority | Retention |
|-------|----------|-----------|
| Secret read access | HIGH | 1 year minimum |
| Secret creation/update | HIGH | 1 year minimum |
| Policy changes | CRITICAL | 2 years (compliance) |
| Failed access attempts | CRITICAL | 1 year |
| Seal/unseal operations | CRITICAL | Indefinite |

**Anomaly signals**: access from a new IP/CIDR range; volume spike (>3x baseline for a path); off-hours access for human auth methods; a service reading outside its policy scope (denied requests); multiple failed auth attempts from one source; a token issued with unusually long TTL.

**Compliance reporting** covers: access inventory (who accessed what secret, when), rotation compliance (secrets overdue for rotation), policy drift since the last review, and orphaned secrets (no access in 90+ days).

## Emergency Procedures

**Secret leak response — contain within 15 minutes of detection**: identify scope (which secrets, where they leaked); revoke/rotate the compromised credential at the source immediately; invalidate every token that accessed the leaked secret; audit the blast radius via access logs for the exposure window; notify security, affected service owners, and compliance if PII/regulated data is involved; run a post-mortem and update controls.

**Vault seal operations**: seal only as a last resort during an active security incident or suspected key compromise — sealing stops all Vault operations. To unseal: gather the Shamir-threshold quorum of key holders (or confirm the auto-unseal KMS key is reachable), unseal, verify audit devices reconnected, and check active leases and token validity.

## CI/CD Integration

Kubernetes workloads authenticate via a Vault Agent sidecar (injected by annotation, renders secrets to a templated file) or an External Secrets Operator for a declarative GitOps approach. CI pipelines should eliminate long-lived secrets by federating through OIDC (e.g., GitHub Actions authenticating to Vault via a JWT role) rather than storing static credentials as CI secrets.

## Anti-Patterns

| Anti-Pattern | Risk | Correct Approach |
|-------------|------|-----------------|
| Hardcoded secrets in source code | Leak via repo, logs, error output | Fetch from secret store at runtime |
| Long-lived static tokens (>30 days) | Stale credentials, no accountability | Dynamic secrets or short TTL + rotation |
| Shared service accounts | No audit trail per consumer | Per-service identity with unique credentials |
| No rotation policy | Compromised creds persist indefinitely | Automated rotation on schedule |
| Secrets in CI environment variables | Visible in build logs, process table | Vault Agent or OIDC-based injection |
| Single unseal key holder | Bus factor of 1, recovery blocked | Shamir split (3-of-5) or auto-unseal |
| No audit device configured | Zero visibility into access | Dual audit devices (file + syslog) |
| Wildcard policies (`path "*"`) | Over-permissioned, violates least privilege | Explicit path-based policies per service |