Clickstream Data Pipeline for Targeted Ads
Design a real time clickstream data pipeline for a large scale e commerce platform. The system needs to capture user behavior, page views, clicks, add to cart events, and purchases, across web and mobile, and process...
Design a real-time clickstream data pipeline for a large-scale e-commerce platform. The system needs to capture user behavior, page views, clicks, add-to-cart events, and purchases, across web and mobile, and process this data to power targeted ad campaigns. Expect around 500K events per second at peak. How would you design this end to end?
How to Approach This Problem
What the Interviewer Is Actually Testing The interviewer isn't checking if you know what Kafka is. They're evaluating: Judgment under ambiguity : Can you define scope from a vague problem statement? Architectural reasoning : Do you explain why you picked each component, not just what ? Awareness of failure modes : Do you proactively call out what breaks before being asked? Cross cutting ownership : Can you speak to privacy, cost, and operational complexity, not just the happy path? Trade off fluency : Every architectural decision is a trade off. Can you name the cost of your choice? Interview Approach (45–60 min) Phase Focus What a Strong Answer Looks Like Scoping (5 min) Define functional +
Clarifying Questions to Ask the Interviewer
Functional Requirements What events are we capturing? (page views, clicks, scroll depth, add to cart, impressions, hovers?) What is the expected data volume? (e.g., 100K events/sec? 1M events/sec?) What is the acceptable latency for ad targeting? (real time < 1 sec? near real time < 5 min? batch daily?) Do we need to join clickstream with user profile data, purchase history, or product catalog? Are users authenticated or do we also track anonymous sessions? What downstream consumers exist? (ML models, dashboards, ad serving APIs, A/B testing?) Non Functional Requirements Data retention policy? (7 days hot, 90 days warm, years cold?) Compliance requirements? (GDPR, CCPA right to deletion, con
Envelope Estimation & Capacity Planning
Throughput Math Metric Value Calculation Events per second 500,000 Given requirement Events per day ~43.2 billion 500K × 86,400 sec Average event size ~1 KB (JSON) Typical click event payload Daily raw ingestion ~43 TB/day 43.2B × 1 KB After Parquet compression ~4.3 TB/day ~10:1 compression ratio Monthly storage (raw) ~130 TB 4.3 TB × 30 days Kafka Sizing Parameter Value Reasoning Partitions per topic 100 200 500K/sec ÷ ~5K msgs/sec per partition Replication factor 3 Standard fault tolerance Brokers 15 20 Handle partition distribution + headroom Retention 7 days Allows replay for reprocessing Throughput per broker ~50 MB/s Conservative estimate with replication Total Kafka throughput ~500 MB
Architecture Walkthrough, End-to-End Data Flow
Why Lambda Architecture? I chose a Lambda Architecture : a dual path design with both a real time streaming path and a batch processing path. The reasoning is that ad targeting demands sub second user profiles (real time path), but we also need heavier computations like audience segmentation, backfills, and late data reconciliation that are better handled in batch. Having both paths means the real time layer delivers 95% of the data within seconds, and the batch layer fills in the remaining 5% within an hour. This is the right trade off for this domain, we'd rather show a slightly incomplete user profile immediately than wait 30 minutes for a perfect one. Technology Selection Rationale Desig
Component Deep Dive
Event Collection Layer "On the client side, we use a lightweight SDK that captures events and batches them before sending to our collection endpoint." Key Design Decisions: Client SDK : Batches events (every 5 seconds or 10 events), compresses payload (gzip), sends via POST Collection Endpoint : Stateless HTTP service behind load balancer Validates schema (reject malformed events fast) Enriches with server side data (IP geolocation, user agent parsing) Publishes to Kafka immediately, no heavy processing here Returns 202 Accepted (async processing) Event Schema (standardized): Kafka, The Central Nervous System "I'd use Apache Kafka as the central nervous system. It decouples producers from co
Identity Resolution & The Identity Graph
This is the hardest unsolved problem in clickstream systems. If you don't raise it proactively, the interviewer will. The Core Problem When a user visits your site, they're initially anonymous, tracked only via a cookie based anonymous id . When they log in, you can link that session to a known user id . But the same real person may have dozens of anonymous IDs: multiple browsers, devices, incognito sessions, and cross app visits. Without identity resolution, your ad targeting sees 50 anonymous strangers where there's really one high value customer. Two Stitching Strategies Deterministic Resolution (High Precision, Lower Recall) Link anonymous id → user id only when a hard identifier is avai