senior-data-scientist

---
name: senior-data-scientist
description: Use when designing or analyzing A/B tests, building or evaluating classification/regression models, engineering features for tabular data, or running causal inference.
category: Engineering
version: 1.0.0
tools: []
---

# Senior Data Scientist

Statistical modeling, experiment design, causal inference, and predictive analytics using Python (NumPy, Pandas, Scikit-learn, XGBoost), R, and SQL.

## 1. Design an A/B Test

Calculate the required sample size per variant before starting, from the baseline conversion rate and a minimum detectable effect (relative lift), using a two-proportion effect size with the standard normal z-values for alpha (default 0.05) and power (default 0.8): `n = ((z_alpha + z_beta) / effect_size)^2`.

Analyze results with a two-proportion z-test: pool the conversion rates, compute the standard error, derive z and the two-sided p-value, and report lift with its 95% confidence interval — not just the p-value.

**Experiment checklist**:
1. Define ONE primary metric; pre-register secondary metrics
2. Calculate sample size before starting
3. Randomize at the user level (not session) to avoid leakage
4. Run for at least one full business cycle (typically 2 weeks)
5. Check for sample ratio mismatch: `abs(n_control - n_treatment) / expected < 0.01`
6. Report lift + confidence interval, not just significance
7. Apply Bonferroni correction when testing multiple metrics: `alpha / n_metrics`

## 2. Build a Feature Engineering Pipeline

Structure a ColumnTransformer with a numeric branch (median imputation → StandardScaler) and a categorical branch (most-frequent imputation → one-hot encoding). Extract cyclical time features from date columns (sine/cosine of day-of-week and month) plus an `is_weekend` flag.

**Checklist**:
1. Never fit transformers on the full dataset — fit on train, transform test
2. Log-transform right-skewed numeric features before scaling
3. For high-cardinality categoricals (>50 levels), use target encoding or embeddings
4. Generate lag/rolling features before the train/test split to avoid leakage
5. Document each feature's business meaning alongside its code

## 3. Train, Evaluate, and Select a Prediction Model

Cross-validate with StratifiedKFold to preserve class balance, scoring on ROC-AUC and average precision (AUC-PR), and compare train vs. test scores per fold to flag an `overfit_gap`. Log every run — parameters, metrics, and the model artifact — to MLflow.

**Checklist**:
1. Always report AUC-PR alongside AUC-ROC for imbalanced datasets
2. Treat an overfit_gap > 0.05 as a warning sign of overfitting
3. Calibrate probabilities (Platt scaling / isotonic) before production use
4. Compute SHAP values to confirm feature importance makes business sense
5. Beat a baseline (e.g., a dummy classifier) before trusting the model
6. Log every run to MLflow — never rely on notebook output for comparison

## 4. Causal Inference: Difference-in-Differences

Estimate the average treatment effect on the treated (ATT) via OLS with a treatment × post interaction term, optionally adding covariates, using HC3 robust standard errors to handle heteroskedasticity. The interaction coefficient is the ATT; report it with its p-value and 95% confidence interval.

**Checklist**:
1. Validate parallel trends in the pre-period before trusting DiD estimates
2. Use HC3 robust standard errors
3. For panel data, cluster standard errors at the unit level
4. Consider propensity score matching if groups differ at baseline
5. Report the ATT with a confidence interval, not just statistical significance

## Stack Notes

Prefer Scikit-learn pipelines for reproducible preprocessing, XGBoost for gradient-boosted tabular models, statsmodels for causal/inferential work needing coefficient-level detail, and SQL for feature extraction close to the data warehouse. Track every experiment and model run in MLflow so results are comparable across iterations, not scattered across notebooks.

## Translating Findings into Decisions

A statistically significant result is not automatically a business decision. Pair every finding with: the effect size in business units (not just p-value), the confidence interval width (a wide interval means "run it longer" not "ship it"), and the cost of being wrong in each direction (false positive vs. false negative). When a model or experiment result will drive a decision, state the recommendation explicitly and name the assumption most likely to invalidate it.