senior-data-engineer

---
name: senior-data-engineer
description: Use when designing data architectures, building ETL/ELT pipelines, optimizing data workflows, implementing data governance, or troubleshooting data systems.
category: Engineering
version: 1.0.0
tools: []
---

# Senior Data Engineer

Production-grade data engineering for scalable, reliable data systems.

## Quick Start

Common tasks:
- Generate pipeline orchestration config for Airflow/Prefect/Dagster
- Validate data quality using schemas and checks (freshness, completeness, uniqueness)
- Analyze and optimize ETL performance bottlenecks

## Architecture Decision Framework

### Batch vs Streaming

| Criteria | Batch | Streaming |
|----------|-------|-----------|
| **Latency** | Hours to days | Seconds to minutes |
| **Volume** | Large historical datasets | Continuous event streams |
| **Complexity** | Complex transformations, ML | Simple aggregations, filtering |
| **Cost** | More cost-effective | Higher infrastructure |
| **Error handling** | Easier reprocessing | Requires careful design |

**Decision tree:**
- Real-time insight required? → Use streaming; else → Use batch
- If streaming: Exactly-once semantics needed? → Kafka + Flink/Spark Structured Streaming; else → Kafka + consumer groups
- If batch: Daily volume > 1TB? → Spark/Databricks; else → dbt + warehouse compute

### Lambda vs Kappa Architecture

| Aspect | Lambda | Kappa |
|--------|--------|-------|
| **Complexity** | Two codebases (batch + stream) | Single codebase |
| **Maintenance** | Higher (sync logic) | Lower |
| **Reprocessing** | Native batch layer | Replay from source |
| **Use case** | ML training + real-time serving | Pure event-driven |

**Choose Lambda:** ML models need historical training data, or batch transformations not feasible in streaming
**Choose Kappa:** Event-sourced architecture, all processing expressible as stream operations

### Data Warehouse vs Data Lakehouse

| Feature | Warehouse | Lakehouse |
|---------|-----------|----------|
| **Best for** | BI, SQL analytics | ML, unstructured data |
| **Storage cost** | Higher (proprietary format) | Lower (open formats) |
| **Flexibility** | Schema-on-write | Schema-on-read |
| **Performance** | Excellent for SQL | Good, improving |
| **Ecosystem** | Mature BI tools | Growing ML tooling |

## Tech Stack

**Languages:** Python, SQL, Scala
**Orchestration:** Airflow, Prefect, Dagster
**Transformation:** dbt, Spark, Flink
**Streaming:** Kafka, Kinesis, Pub/Sub
**Storage:** S3, GCS, Delta Lake, Iceberg
**Warehouses:** Snowflake, BigQuery, Redshift, Databricks
**Quality:** Great Expectations, dbt tests, Monte Carlo
**Monitoring:** Prometheus, Grafana, Datadog

## Workflows

**Pipeline design**: Choose batch vs streaming, identify data sources/destinations, design transformation steps, configure error handling and retries

**Data modeling**: Dimensional modeling (Star/Snowflake schemas), Slowly Changing Dimensions (SCD Types 1-6), Data Vault patterns, dbt best practices, partitioning and clustering strategy

**Data quality**: Implement test frameworks, create data contracts with schema validation, build CI/CD for pipelines, set up observability and lineage tracking, design incident response

**Performance tuning**: Profile job execution, identify bottlenecks, optimize Spark job configuration, tune warehouse queries, reduce data movement