Recommendation Engine

---
name: Recommendation Engine
description: Use when designing personalization systems with collaborative filtering, content-based methods, and hybrid approaches for user-item recommendations
category: Data & AI
version: 1.0.0
tools: []
---

# Recommendation Engine

Build personalization systems that surface relevant items to users across e-commerce, content, and social platforms using collaborative filtering, content-based filtering, and hybrid architectures. Balance recommendation quality against latency, cold-start handling, inventory availability, and diversity requirements.

## Process

1. Analyze interaction data for sparsity, user activity distributions, item popularity curves, and temporal patterns. Determine whether implicit feedback (clicks, views, purchases) or explicit ratings is appropriate.
2. Implement collaborative filtering using matrix factorization (ALS, SVD) for moderate-scale datasets and neural collaborative filtering for larger datasets, with negative sampling strategies.
3. Build content-based models using TF-IDF or embedding representations of item attributes for recommendations on items with no interaction history.
4. Design hybrid architecture combining collaborative and content-based signals via weighted ensembles, cascading, or unified models ingesting both interaction and content features.
5. Address cold-start problem: popularity-based fallback for new users, content-based similarity for new items, onboarding flows collecting initial preferences.
6. Implement two-stage retrieval and ranking architecture: fast candidate generation (approximate nearest neighbors, inverted indices) narrowing millions to hundreds, followed by precise ranking model.
7. Apply business rules as post-processing filters: remove purchased items, enforce diversity constraints across categories, check inventory availability, respect suppression lists.
8. Build serving layer with precomputed recommendations cached in Redis for high-traffic users and real-time scoring for long-tail users, with latency budgets: candidate generation ≤10ms, full ranking pipeline ≤50ms.
9. Implement A/B testing infrastructure with experiment cohort assignment, engagement metrics (CTR, conversion, session depth), and statistical significance testing with multiple comparison correction.
10. Design feedback loop ingesting new interactions, retraining on scheduled cadence, evaluating offline metrics (NDCG, MAP, MRR) before production promotion.

## Technical Standards

- Offline evaluation uses temporal train-test splits (not random) to prevent information leakage.
- Ranking-aware metrics (NDCG, MAP, MRR) alongside accuracy measures (precision, recall at K).
- Embedding dimensions tuned via hyperparameter search, not arbitrary selection.
- Latency: candidate generation ≤10ms, full pipeline ≤50ms, full ranking under 50ms.
- User and item embeddings versioned with training metadata for reproducibility.
- Popularity bias measured and mitigated; popular-only recommendations provide no personalization.
- All experiments run with minimum sample size calculated before launch.

## Verification

- Collaborative filtering outperforms popularity baseline on NDCG@10 across held-out temporal test set.
- Hybrid model improves cold-start recommendations versus content-based alone on users with fewer than 5 interactions.
- Business rule filters correctly suppress items violating constraints without leaving empty slots.
- A/B test cohort assignment is deterministic and balanced across variants.
- Serving layer meets latency SLAs under peak traffic load.
- Retraining pipeline produces model matching or exceeding incumbent on offline metrics.