Question 1

Explain how Spark Catalyst optimizer works.

Accepted Answer

Catalyst is Spark SQL's query optimizer — it transforms logical plans into efficient physical execution plans. Pipeline : 1. Parsing : SQL/DataFrame code → unresolved logical plan (AST) 2. Analysis : resolve column names, data types using catalog → analyzed logical plan 3. Logical optimization : apply rule based transformations: Predicate pushdown (move filters close to data source) Column pruning (remove unused columns) Constant folding ( 1 + 1 → 2 ) Join reordering 4. Physical planning : generate multiple physical plans (e.g., SortMergeJoin vs BroadcastHashJoin), pick the lowest cost one using statistics 5. Code generation (Tungsten) : generate bytecode for CPU efficient execution — avoids Java object overhead Why it matters : you write simple DataFrame operations; Catalyst figures out the optimal execution order — most manual repartition calls and join hints are only needed when Catalyst lacks statistics or makes wrong cost estimates.

Question 2

What is Adaptive Query Execution (AQE) in Spark?

Accepted Answer

AQE (Spark 3.0+) re optimizes query plans at runtime using statistics collected during execution — unlike Catalyst which plans everything upfront. Enable : spark.sql.adaptive.enabled=true (default true in Spark 3.2+) Key features : 1. Dynamic coalescing of shuffle partitions : after a shuffle, if many partitions are small, AQE merges them — reduces the number of tasks for the next stage 2. Dynamic switch of join strategies : if Catalyst planned a SortMergeJoin but runtime statistics show one side is small, AQE switches to BroadcastHashJoin automatically 3. Skew join optimization : detects skewed partitions at runtime and splits them into smaller sub partitions to avoid one task processing 90% of the data Impact : reduces need for manual tuning of spark.sql.shuffle.partitions and broadcast hints.

Question 3

How do you handle skewed data in joins?

Accepted Answer

Data skew occurs when one join key value has disproportionately many rows — one task processes most of the data while others sit idle. Detection : Spark UI → Stages → look for max task time median task time. Solutions : 1. AQE skew handling (automatic in Spark 3): set spark.sql.adaptive.skewJoin.enabled=true — AQE splits oversized partitions and replicates the matching small side partition 2. Salting (manual): 3. Broadcast join : if the skewed side is small enough, broadcast it — no partitioning needed at all 4. Pre aggregate : reduce cardinality before joining with groupBy + agg

Question 4

What is the difference between DataFrame and Dataset API?

Accepted Answer

DataFrame Dataset Type safety Runtime (untyped) Compile time (typed) Language Python, Scala, Java, R Scala, Java only API Column based operations Typed case class operations Performance Same (both use Catalyst/Tungsten) Same Errors Caught at runtime Caught at compile time In Python (PySpark) : only DataFrames exist — Python has no compile time type checking so Dataset has no value there. Rule of thumb : use DataFrames in Python/production pipelines; use Datasets in Scala when you want compile time safety for complex transformations.

Question 5

How do you handle large data ingestion from APIs?

Accepted Answer

Key challenges : rate limiting, pagination, varying payload sizes, partial failures. Approach : At scale : use Airflow to orchestrate parallel API calls (one task per entity type), store raw responses in Bronze Delta, transform in Silver layer. Checkpoint last ingested id to support incremental runs.

Question 6

How do you handle out-of-order events in streaming systems?

Accepted Answer

Out of order events arrive at the pipeline after events with later timestamps — due to network delays, mobile offline buffering, or multi region delivery. Watermarking (Spark Structured Streaming): Events arriving more than 30 minutes after the current watermark are dropped . The watermark advances as the max observed event ts allowed lateness. Flink : BoundedOutOfOrdernessWatermarks — same concept. For batch processing : partition by event date , but run the previous day's partition again the next morning to catch stragglers:

Question 7

Explain how Azure Event Hub integrates with Spark Streaming.

Accepted Answer

Azure Event Hub is a managed Kafka compatible event streaming service. Spark can consume from it using the Kafka connector. Setup : Key difference from Kafka : Event Hub uses consumer groups (called Consumer Groups) and has a retention period (default 1 7 days). Scale by increasing partitions (called partition count in Event Hub). Checkpoint : store at ADLS path to resume after failure.

Question 8

Explain the concept of snapshot isolation in Delta tables.

Accepted Answer

Snapshot isolation means each read transaction sees a consistent snapshot of the data as of a specific point in time — even if concurrent writes are happening. How Delta Lake implements it : Every write creates a new version in the transaction log ( delta log/00000000000000000001.json ) Reads use the latest committed version's file list — they never see partial writes Writers use optimistic concurrency : read the current version, write changes, then attempt to commit. If another writer committed first, retry with conflict resolution Time travel is a byproduct of this: VACUUM : removes old files after retention period (default 7 days). After VACUUM , time travel to older versions is no longer possible.

Wipro Interview Questions