Streaming Recommendation Engine
Design a streaming recommendation engine for a large scale music or video platform (think Spotify, Netflix, or YouTube). The system must deliver personalized content recommendations to 100 million users using collabor...
Design a streaming recommendation engine for a large-scale music or video platform (think Spotify, Netflix, or YouTube). The system must deliver personalized content recommendations to 100 million users using collaborative filtering, content-based signals, and real-time user behavior. User interaction events, plays, likes, skips, searches, and playlist additions, arrive at 500K events per second. Recommendations must refresh in near-real-time as users interact, while batch model training runs daily. How would you design the end-to-end data pipeline, from event ingestion through embedding generation to real-time serving?
How to Approach This Problem
How to Approach This Problem Recommendation engines have three hard problems that separate generic system design answers from ones that show genuine depth. Knowing these upfront shapes every architectural decision you make. Three Hard Problems Unique to Recommendation Engines 1. The cold start problem requires a completely different strategy than collaborative filtering. A new user has no interaction history. A new item has no ratings. Matrix factorization fails silently, either returning generic recommendations or errors. Strong answers have a dedicated cold start path: content based features for new items (genre, tags, metadata), popularity/trending fallback for new users, and a graduated
Clarifying Questions to Ask the Interviewer
Functional Requirements What types of user interactions do we capture? (plays, likes, skips, searches, playlist adds, shares, scroll impressions?) What is the expected scale? (100M users? 50M content items? 500K events/sec?) What recommendation paradigms should we support? (collaborative filtering, content based, hybrid?) How fresh must recommendations be? (Real time after each interaction? Near real time within 1 minute? Batch hourly/daily?) Do we need to handle the cold start problem? (new users with no history, new content with no interactions) Should the system support A/B testing of ranking models? (How many concurrent experiments?) Do we need diversity/serendipity constraints? (Avoid s
Envelope Estimation & Capacity Planning
Throughput Math Metric Value Calculation Monthly active users 100,000,000 Given requirement Content catalog size 50,000,000 Tracks/videos in catalog Events per second 500,000 Given (plays, likes, skips, searches, playlist adds) Events per day ~43.2 billion 500K x 86,400 sec Average event size ~500 bytes user id, item id, event type, timestamp, context Daily raw ingestion ~21.6 TB/day 43.2B x 500 bytes After Parquet compression ~2.2 TB/day ~10:1 compression ratio Monthly storage (compressed) ~66 TB 2.2 TB x 30 days Embedding Storage Parameter Value Calculation User embedding dimensions 128 Balance of quality vs storage (256 for premium models) Item embedding dimensions 128 Same dimensionality
Architecture Walkthrough
End to End Flow The recommendation engine has two parallel data paths, real time (milliseconds) and batch (hours), that converge at the Ranking Service when a user requests a page. Event Ingestion Every user interaction is an immutable event. We never discard raw events, they flow to both paths simultaneously. Real Time Path (Latency: milliseconds) Step Component What Happens 1 Kafka Delivers events partitioned by user id to Flink consumer group 2 Flink Updates user feature vectors in sliding windows (last 30 min of activity), computes session context (genre distribution, skip rate, time of day pattern), updates item popularity counters 3 Feature Store (Redis) Stores real time features per u
Component Deep Dive
Candidate Generation, Approximate Nearest Neighbor (ANN) "We need to find the top 500 most relevant items out of 50 million in under 10ms. Exact nearest neighbor search is O(n), we use ANN with HNSW indexing for O(log n) approximate search." How it works: 1. User embedding (128 dim vector) is fetched from the Feature Store 2. ANN index (FAISS or Milvus) finds the top 500 items whose embeddings are closest (cosine similarity) 3. Results are approximate but achieve 95%+ recall compared to exact search Technology choices: Option Pros Cons Best For FAISS (Facebook) Fastest in memory, GPU support No built in distribution, manual sharding Single node or GPU serving Milvus Distributed, cloud native
Data Modeling
Core Tables User Interactions (Event Stream, Source of Truth) In the data lake, this table is stored as Parquet partitioned by DATE(created at) and clustered by user id . The OLTP version above is for the operational API; analytics queries hit the lake. Item Catalog User Profiles Recommendations Served (Request Log) Recommendation Feedback (Engagement Tracking) This table is critical for measuring recommendation quality and training the next model iteration. Join with recommendations served to compute CTR by position, model version, and candidate count. Experiment Assignments Model Embeddings Metadata Entity Relationship Summary Storage strategy: Operational tables (user profiles, item catal