Real-Time Fraud Detection Pipeline
Design a real time fraud detection pipeline for a payment processing platform operating at Stripe/PayPal scale. The system must score every transaction in under 100ms using a two layer approach: a fast rule engine for...
Design a real-time fraud detection pipeline for a payment processing platform operating at Stripe/PayPal scale. The system must score every transaction in under 100ms using a two-layer approach: a fast rule engine for velocity checks and blocklists, plus an ML model (gradient-boosted trees) for behavioral analysis. The pipeline processes 10,000 transactions per second, computes real-time features (transaction velocity, geo-distance) and historical features (spending patterns, device fingerprints), and produces a three-way decision: approve, decline, or send to manual review. Fraud investigators label flagged cases, and their feedback feeds back into model retraining. How would you design this end to end?
How to Approach This Problem
How to Approach This Problem Fraud detection interviews test something different from generic system design. The interviewer is not just checking whether you know Kafka and Redis. They want to see whether you understand the three hard problems that make fraud detection architecturally distinct from other real time pipelines, and whether your design decisions are grounded in those constraints. The Three Hard Problems Unique to Fraud Detection 1. Sub 100ms scoring with velocity features requires a pre computed feature store, not on demand computation. Fraud detection fires on every transaction. The model needs velocity features: transactions in the last 1 minute, 10 minutes, 1 hour, and 24 hou
Clarifying Questions to Ask the Interviewer
Functional Requirements What is the transaction throughput? (10K TPS? 100K TPS?) What is the latency SLA for scoring? (Sub 100ms? Sub 200ms? Is this p50 or p99?) What scoring layers are required? (Rule engine only? ML model only? Both in series?) What is the decision output? (Binary approve/decline? Three way approve/decline/review?) What types of fraud are we detecting? (Card not present, account takeover, friendly fraud, merchant fraud?) Do we need real time features (transaction velocity, geo distance) or can we rely on pre computed features only? Is there a manual review queue? How many investigators? What is their SLA for reviewing flagged cases? Do we need explainability for declined t
Envelope Estimation & Capacity Planning
Throughput Math Metric Value Calculation Transactions per second 10,000 Given requirement Transactions per day ~864 million 10K x 86,400 sec Average transaction payload ~2 KB Payment details, device info, geo, merchant data Daily raw ingestion ~1.7 TB/day 864M x 2 KB Fraud rate ~0.1% Industry average for card not present Fraudulent transactions per day ~864,000 864M x 0.001 Manual review rate ~1 2% of total Transactions in the "gray zone" Review queue per day ~8.6 17.2 million 864M x 0.01 0.02 Latency Budget Breakdown (p99 < 100ms) Stage Budget What Happens Network (gateway to scoring) 10 ms TLS handshake amortized, keep alive connections Feature Store lookup (Redis) 5 10 ms Fetch velocity c
Architecture Walkthrough
End to End Flow: Payment Event to Decision in < 100ms The architecture follows a two layer scoring pattern : a fast deterministic rule engine runs in parallel with an ML scorer, and a Decision Service merges both signals into a final approve/decline/review verdict. Critical Path (Synchronous, Must Complete in < 100ms) Step by step: Step Component What Happens Latency 1 Payment Service Publishes a transaction event (amount, card hash, device, geo, merchant) to the API Gateway ~2 ms 2 API Gateway Authenticates, rate limits, and forwards to Kafka transactions.raw topic ~5 ms 3 Kafka Persists the event; the Scoring Service consumer picks it up ~10 ms 4a Rule Engine Evaluates deterministic rules:
Component Deep Dive
Rule Engine Design "The rule engine is the first line of defense, it catches obvious fraud in sub 5 ms before the ML model even runs." Rule DSL (YAML defined rules): Rule engine internals: Feature Implementation Priority ordering Rules sorted by priority; lowest number evaluated first Short circuit evaluation If a HARD DECLINE rule fires, skip all remaining rules Hot reload without downtime Rules loaded from Redis; RULE VERSION key incremented on update; engine polls every 5 sec Rule versioning Each rule has a version; old versions archived for audit trail A/B testing rules Shadow mode: new rules evaluated but not enforced; results logged for comparison Performance ~50 rules evaluated in < 5
Data Modeling
Core Tables Transactions Table (Source of Truth) Note: For the real time scoring path, the ML Scorer reads from the Feature Store (Redis), NOT from this table. This table is the write ahead log and the source of truth for batch processing, audits, and analytics. Fraud Decisions Table Fraud Rules Table Fraud Labels Table (Feedback Loop) Key insight: The label delay days column is critical for model training. Chargebacks arrive 30 45 days after the transaction, creating a label delay problem . The model must be trained on data where enough time has passed for chargebacks to arrive, or use semi supervised techniques for recent transactions. User Risk Profiles Table Case Queue Table (Manual Revi