senior-ml-engineer

---
name: senior-ml-engineer
description: Use when deploying ML models to production, setting up MLOps pipelines, integrating LLM APIs, building RAG systems, or monitoring models for drift.
category: Engineering
version: 1.0.0
tools: []
---

# Senior ML Engineer

Production ML engineering methodology: model deployment, MLOps infrastructure, LLM integration, RAG, and monitoring.

## Model deployment workflow

1. Export the model to a standardized format (ONNX, TorchScript, SavedModel).
2. Package it with dependencies in a container with a health-check endpoint.
3. Deploy to staging, run integration tests.
4. Canary at 5% of production traffic.
5. Monitor latency and error rate for at least 1 hour before promoting.
6. Promote to full production only if p95 latency < 100ms and error rate < 0.1%.

Serving options: FastAPI+Uvicorn for REST APIs and small models (low latency, medium throughput); Triton Inference Server for GPU inference with batching (very low latency, very high throughput); TensorFlow Serving for TF models; TorchServe for PyTorch models; Ray Serve for complex multi-model pipelines.

## MLOps pipeline

1. Configure a feature store (Feast, Tecton) for training data, keyed by entity (e.g., `user_id`) with a defined TTL per feature.
2. Set up experiment tracking (MLflow, Weights & Biases).
3. Build the training pipeline with hyperparameter logging.
4. Register models in a model registry with version metadata.
5. Trigger staging deployment from registry events.
6. Set up A/B infrastructure for model comparison.
7. Enable drift monitoring with alerting.

Retraining triggers: scheduled (weekly/monthly cron) → full retrain; performance drop below threshold → immediate retrain; data drift (PSI > 0.2) → evaluate then retrain; new-data volume threshold reached → incremental update.

## LLM integration workflow

1. Build a provider abstraction layer so you can swap vendors without touching call sites.
2. Add retry logic with exponential backoff (e.g., up to 3 attempts).
3. Configure a fallback to a secondary provider.
4. Add token counting and context truncation before every call.
5. Cache responses for repeated queries.
6. Track cost per request.
7. Validate structured output against a schema (e.g., with Pydantic) rather than trusting free-text parsing.

Representative per-1K-token pricing to reason about cost trade-offs (verify current rates before quoting): GPT-4 ($0.03 in / $0.06 out), GPT-3.5 ($0.0005 in / $0.0015 out), Claude 3 Opus ($0.015 in / $0.075 out), Claude 3 Haiku ($0.00025 in / $0.00125 out).

## RAG system implementation

1. Choose a vector database: Pinecone (managed, high scale, low latency, production-ready), Qdrant (self- or managed-hosted, very low latency, performance-critical), Weaviate (hybrid search), Chroma (self-hosted, prototyping), or pgvector (self-hosted, reuse an existing Postgres instance).
2. Select an embedding model on a quality/cost trade-off.
3. Chunk documents: fixed (500–1000 tokens, 50–100 overlap) for general text; sentence-based (3–5 sentences, 1-sentence overlap) for structured text; semantic (variable, boundary at meaning shifts) for research papers; recursive/hierarchical (parent-child) for long documents.
4. Build ingestion with metadata extraction, then retrieval with query embedding and reranking for relevance.
5. Validate: does the response actually reference retrieved context, with no hallucinated claims not grounded in it?

## Model monitoring

Track latency (p50/p95/p99), error rate, input drift, and prediction-distribution shift; log ground truth when it becomes available and compare model versions with A/B metrics. Detect drift statistically with a Kolmogorov-Smirnov two-sample test between a reference distribution and the current one — flag drift when p-value falls below 0.05.

Alert thresholds: p95 latency warning >100ms / critical >200ms; error rate warning >0.1% / critical >1%; PSI (drift) warning >0.1 / critical >0.2; accuracy drop warning >2% / critical >5%. Alerts should fire before user-visible degradation, not after.

## Tech stack

ML frameworks: PyTorch, TensorFlow, Scikit-learn, XGBoost. LLM frameworks: LangChain, LlamaIndex, DSPy. MLOps: MLflow, Weights & Biases, Kubeflow. Data: Spark, Airflow, dbt, Kafka. Deployment: Docker, Kubernetes, Triton. Databases: PostgreSQL, BigQuery, Pinecone, Redis.