prompt-governance

---
name: prompt-governance
description: Use when managing prompts in production at scale: versioning, A/B testing, prompt registries, regression prevention, or eval pipelines for production AI features.
category: Engineering
version: 1.0.0
tools: []
---

# Prompt Governance

Treat prompts as first-class infrastructure — versioned, tested, evaluated, and deployed with the same rigor as application code. Prevent quality regressions, enable safe iteration, and give teams confidence that a prompt change won't break production. Prompts are code; they change behavior in production; ship them like code.

## Before Starting

Check the project context first if it's already known. Otherwise gather this in one shot: current state (how prompts are stored today — hardcoded, config files, database, or a dedicated tool; how many distinct prompts are in production; has a prompt change ever caused an uncaught quality regression); goals (versioning chaos, missing evals, blind A/B testing, or slow iteration as the primary pain; team size and prompt-ownership model; tooling constraints); AI stack (LLM provider(s), frameworks such as LangChain/LlamaIndex/custom/direct API, existing test/CI infrastructure).

## Three Modes

**Mode 1 — Build a prompt registry:** no centralized prompt management exists yet; design and describe a registry with versioning, environment promotion, and audit trail.

**Mode 2 — Build an eval pipeline:** prompts are stored somewhere but there's no systematic quality testing; design an evaluation pipeline that catches regressions before production.

**Mode 3 — Governed iteration:** registry and evals already exist; design the full governance workflow — branch, test, eval, review, promote — with rollback capability.

## Mode 1: Prompt Registry

A registry provides: a single source of truth for all prompts, version history with rollback, environment promotion (dev → staging → prod), an audit trail (who changed what, when, why), and variable/template management.

**Minimum viable registry (file-based, for small teams):** structured files in version control — a `registry.yaml` index of all prompts, and one file per prompt version (e.g. `prompts/summarizer/v1.1.0.md`). The registry index records each prompt's id, description, owner, target model, and its version history with file path, status (production/archived), and promotion metadata (who promoted it and when).

**Production registry (database-backed, for larger teams):** an API-accessible registry with tables for prompts and prompt_versions tracking slug, content, model, environment, eval_score, and promotion metadata.

To bootstrap a file-based registry: create the directory structure above and populate the registry index with the existing prompts, their current versions, and ownership metadata.

## Mode 2: Eval Pipeline

The problem: prompt changes are deployed by feel, with no systematic way to know if a new prompt is better or worse than the current one. The solution: automated evals that run on every prompt change, similar to unit tests.

**Eval types:**

| Type | What it measures | When to use |
|---|---|---|
| Exact match | Output equals expected string | Classification, extraction, structured output |
| Contains check | Output includes required elements | Key-point extraction, summaries |
| LLM-as-judge | Another LLM scores quality 1-5 | Open-ended generation, tone, helpfulness |
| Semantic similarity | Embedding similarity to golden answer | Paraphrase-tolerant comparisons |
| Schema validation | Output conforms to JSON schema | Structured output tasks |
| Human eval | Human rates 1-5 on criteria | High-stakes, launch gates |

**Golden dataset design:** every prompt needs a fixed set of input/expected-output pairs that define correct behavior — minimum 20 examples for basic coverage, 100+ for production confidence; cover edge cases and failure modes, not just the happy path; reviewed and approved by a domain expert, not just the prompt's author; versioned alongside the prompt itself, since a prompt change may require golden-set updates.

**Pipeline mechanics:** the eval runner takes a prompt version and a golden dataset, calls the LLM for each example, scores the response against the expected output, and reports pass_rate, avg_score, and per-example failure details. Calibrate pass thresholds to the use case — classification/extraction ≥95% exact match, summarization ≥0.85 LLM-as-judge score, structured output 100% schema validation, open-ended generation ≥80% human-eval approval.

## Mode 3: Governed Iteration

The full prompt deployment lifecycle, with a gate at each stage: BRANCH (feature branch for the prompt change) → DEVELOP (edit in dev, manual testing) → EVAL (run the eval pipeline against the golden dataset, automated in CI) → COMPARE (new eval score vs. current production score) → REVIEW (PR review covering eval results plus the prompt diff) → PROMOTE (staging to production, with an approval gate) → MONITOR (watch production metrics for 24-48h post-deploy) → ROLLBACK (one-command rollback to the previous version if needed).

**A/B testing prompts** (for real-user impact, not just eval scores): use stable assignment (same user always gets the same variant, via a hash of user_id); log every assignment with user_id, prompt_slug, and variant; define the success metric before starting, not after; run a minimum of 1 week or 1,000 requests per variant; watch for novelty effect (a first-day engagement spike that fades); require statistical significance (p < 0.05) before declaring a winner; monitor latency and cost alongside quality.

**Rollback playbook:** a one-command rollback promotes the previous version back to production status in the registry; verify afterward by re-running evals against the restored version.

## Proactive Triggers

Surface these without being asked: prompts hardcoded in application code (changes require code deploys — flag immediately); no golden dataset for a production prompt (flying blind — any change could silently regress quality); declining eval pass rate over time (model updates can silently break prompts — scheduled evals catch this before users do); no rollback capability (a bad prompt in production with no rollback forces an emergency deploy); one person owning all prompt knowledge (bus-factor risk); prompt changes deployed without an eval ("just this once" is how regressions ship).

## Output Artifacts

Registry design → file structure, schema, promotion workflow. Eval pipeline → golden dataset template, eval-runner approach, pass-threshold recommendations. A/B test setup → variant-assignment logic, measurement plan, success metrics, analysis template. Prompt diff review → side-by-side comparison with eval-score delta and a deployment recommendation. Governance policy → a team-facing doc covering ownership model, review requirements, and deployment gates.

## Communication Standard

Bottom line first (risk or recommendation before explanation); every finding states what, why, and how; every action has an owner and a deadline, not "the team should consider…"; tag confidence as verified / medium / assumed.

## Anti-Patterns

Hardcoding prompts in application source (couples prompt changes to code deploys — store them in a versioned registry instead); deploying prompt changes without running evals (silent regressions reach users — gate every promotion on the eval pipeline passing); a single golden dataset used forever (drifts from real usage — review and refresh it quarterly with new edge cases from production failures); one person owning all prompt knowledge (bus factor of 1 — document ownership, rationale, and version history in the registry); A/B testing without a pre-defined success metric (post-hoc metric selection biases results — define metric and sample size before starting); skipping rollback capability (forces an emergency code deploy when a bad prompt ships — every promotion needs a one-command rollback).