caio-review

---
name: caio-review
description: Use when pressure-testing an AI plan — shipping without an eval set, choosing API vs. fine-tune vs. build, or classifying a use case under the EU AI Act.
category: C-Level Advisory
version: 1.0.0
tools: []
---

# CAIO Review — AI Plan Interrogation

Pressure-test any plan that involves AI — model selection, risk classification, cost economics, or AI hiring — using six forcing questions before it ships.

## When to run

Before shipping a new AI-powered feature; before signing a multi-year AI vendor or infrastructure contract; before an EU launch of any AI feature; before a major AI hire (especially ML engineer or research scientist); before a fine-tuning commitment; before adopting AI in a regulated domain (employment, credit, healthcare, education); or whenever "AI" gets used near "competitive advantage" or "moat."

## The six questions

1. **What does this AI need to be good at, and how would you measure it?** No eval set means no ship — define at least 50-100 representative inputs with expected outputs or a grading rubric, including ambiguous, adversarial, and format-edge cases. If "good" can't be written down, it's a vibe, not a feature.
2. **What's the SLO on hallucination/error rate, and what's the fallback?** Every AI feature has a failure mode: define a quantified SLO (e.g. "<5% hallucination on factual queries"), a detection mechanism (monitoring, sampling, feedback loop), a fallback (human-in-loop review, safer default, refuse-to-answer), and the blast radius if the SLO is breached.
3. **What's the risk tier under the EU AI Act, and is conformity assessment required?** Applies whenever EU residents are affected or the domain is regulated: PROHIBITED (cannot launch in EU, re-scope), HIGH (conformity assessment plus EU database registration plus roughly a dozen articles of obligations, 3-12 months and $50-200K), LIMITED (transparency obligations — chatbot disclosure, AI-content marking), or MINIMAL (no specific obligations beyond the voluntary NIST AI RMF).
4. **API, fine-tune, or BUILD?** For the specific use case: roughly 80% of B2B SaaS use cases are best served by API, about 15% justify fine-tuning (domain-specific behavior, labeled data, an ML team, high volume), and under 1% justify building from scratch. The decision must weigh economic breakeven and practical feasibility (data, team, compliance) together.
5. **What's the 12-month cost trajectory at expected scale?** API cost is variable and scales linearly; self-hosted cost is mostly fixed with breakeven typically between 1-10B tokens/month for a 70B-class model. Self-hosting hides ops, monitoring, model-update, capacity, failover, and security costs; APIs hide vendor lock-in, capability drift, rate limits, and data-residency costs. Prompt caching is the most underrated lever — check provider support.
6. **What role unblocks this, and are the prerequisites hired first?** Map the need to a specific role: an AI engineer (applied, full-stack, prompts, evals, deployment) covers most startup needs; an ML engineer (fine-tuning, retraining infrastructure) only makes sense after a platform engineer and labeled data exist; a research scientist (model invention) only makes sense if the model itself is the product. Don't hire a research scientist as the first AI hire — they need infrastructure to be productive.

## Output structure

State the decision being made in one sentence (model selection, risk classification, economics, or next hire), then report: eval-set status and the SLO/fallback; if applicable, the recommended path (API / fine-tune / BUILD) with a rough 3-year TCO comparison and breakeven volume; if applicable, the EU AI Act tier (PROHIBITED / HIGH / LIMITED / MINIMAL), whether conformity assessment is required, and open controls; if applicable, monthly cost at current volume and the self-hosted breakeven volume; if applicable, the next AI hire and whether prerequisites are in place. Close with a verdict — SHIP, SHARPEN, or BLOCK — and three concrete next steps.

## Related reviews

Hand off training-data implications, AI vendor contract or output-liability questions, and prompt-injection/jailbreak threat modeling to the relevant specialist reviewer (data, legal, security) by @mentioning the right teammate; escalate multi-year AI or GPU commitments for a cost freeze/review before signing.