1M Requests/Sec Pipeline to Data Warehouse
Design a high throughput data pipeline that ingests 1 million events per second from distributed API servers, processes them with exactly once semantics, and loads the data into a data warehouse with less than 5 minut...
Design a high-throughput data pipeline that ingests 1 million events per second from distributed API servers, processes them with exactly-once semantics, and loads the data into a data warehouse with less than 5-minute freshness. The system must handle 10x traffic spikes gracefully using backpressure mechanisms at every layer, support schema evolution without downtime, and guarantee no data loss or duplication in the warehouse. How would you architect this end to end?
How to Approach This Problem
How to Approach This Problem "Before you draw a single box, tell the interviewer: this problem has three hard subproblems that most candidates miss. Naming them upfront signals you have been burned by them in production." The Three Hard Problems Unique to High Throughput Pipelines 1. Backpressure without data loss is harder than it looks. When consumers are slower than producers, you must buffer, drop, or slow the producer. Naive implementations drop silently or cascade failures upstream. The right answer uses Kafka as the decoupling buffer between producers, which write at their rate, and consumers, which read at their rate. But naming Kafka is not enough. You need to explain how consumer l
Clarifying Questions to Ask the Interviewer
Functional Requirements What is the event source? (API servers, IoT devices, mobile clients, internal microservices?) What does "1M requests/sec" mean exactly? Sustained average or peak? Is 1M the P50 or P99? What is the acceptable warehouse freshness? (Sub minute? 5 minutes? 15 minutes?) What does "exactly once" mean in this context? Exactly once into Kafka? Into the warehouse? End to end? What is the event schema? Fixed schema or evolving? How often do fields get added/changed? What kind of warehouse? (Snowflake, Redshift, BigQuery, Synapse?), this affects the loading strategy Do we need real time transformations (dedup, enrichment, aggregation) or just raw landing? Are there downstream co
Envelope Estimation & Capacity Planning
Throughput Math Metric Value Calculation Events per second (sustained) 1,000,000 Given requirement Events per second (10x spike) 10,000,000 Peak burst scenario Events per day ~86.4 billion 1M x 86,400 sec Average event size ~1 KB (Avro) Typical structured event payload Raw throughput (sustained) ~1 GB/sec 1M x 1 KB Raw throughput (10x spike) ~10 GB/sec 10M x 1 KB Daily raw ingestion ~86 TB/day 86.4B x 1 KB After Parquet compression (~10:1) ~8.6 TB/day Columnar + Snappy compression Monthly storage (compressed) ~258 TB 8.6 TB x 30 days Yearly storage (compressed) ~3.1 PB Petabyte scale archive Kafka Sizing Parameter Value Reasoning Partitions per topic 500 1000 1M/sec / ~2K msgs/sec per partit
Architecture Walkthrough
End to End Data Flow The pipeline has three major stages: Ingestion , Processing , and Loading . Each stage is decoupled by a durable buffer so that failures in one stage do not cascade to the others. Stage 1: Ingestion (Client to Kafka) 1. Client applications send events over HTTPS to an L7 load balancer 2. Load balancer (Envoy or NGINX) distributes traffic across API server instances using round robin with health checks. L7 gives us request level routing, header inspection, and TLS termination 3. Stateless API servers perform three tasks on each event: Validate the payload against the Avro schema (via Schema Registry lookup) Assign an event id (UUID v7, time sortable) for idempotency if th
Component Deep Dive
Kafka at 1M Events/Second "Kafka is not just a message queue here, it is the shock absorber that decouples ingestion speed from processing speed. Getting the partition count and producer config right is critical." Partition count calculation: Replication and durability: Setting Value Why replication.factor 3 Survive loss of any single AZ min.insync.replicas 2 Guarantee at least 2 copies before ACK unclean.leader.election false Never elect an out of sync replica as leader Producer tuning for throughput: acks=all vs acks=1 tradeoff: Setting Latency Durability When to use acks=1 ~2ms Leader only; data loss if leader dies before replication Metrics, logs, loss tolerant acks=all ~10 15ms All ISR
Data Modeling
Core Tables Raw Events (Landing Zone) "The raw events table is the immutable source of truth. Everything downstream can be rebuilt from this table." Why JSONB for payload? At the raw layer, we preserve the full payload without schema enforcement. This allows reprocessing even if schema evolution changes field names. The typed fact tables downstream extract specific fields. Event Processing Log (Lineage & Debugging) Use case: When a data analyst reports "event X is missing from the warehouse," query this table to trace exactly where it got stuck. Was it produced to Kafka? Did Flink process it? Did the COPY succeed? Warehouse Load Batches (Operational Metadata) Monitoring query, batch load hea