Q: What is the difference between repartition and coalesce in Spark?

Both repartition and coalesce change the number of partitions in a DataFrame, but they work differently: repartition(n) : Performs a full shuffle — data is redistributed evenly across all n partitions Can both increase and decrease the number of partitions Produces evenly sized partitions (good for balancing skew) More expensive due to the shuffle, but necessary when you need balanced partitions coalesce(n) : Avoids a full shuffle by merging existing partitions on the same executor — moves data locally Can only decrease the number of partitions (attempting to increase returns the original count) May produce uneven partition sizes since it simply combines existing partitions without redistribution Much cheaper than repartition for reducing partition count Aspect repartition coalesce Shuffle Full shuffle Minimal/no shuffle Direction Increase or decrease Decrease only Partition balance Even Potentially uneven Cost Higher Lower

Q: When should you use repartition vs. coalesce?

Choose between these operations based on your specific need: Use repartition when: You need to increase the number of partitions (e.g., before a large join or shuffle heavy aggregation) Your data is skewed and you need evenly distributed partitions for downstream processing You want to repartition by a specific column for efficient downstream joins or writes: df.repartition(200, 'customer id') Writing to storage and you want a controlled, uniform number of output files Use coalesce when: You need to reduce partition count before writing small output files (e.g., writing a small final result) After filtering a large DataFrame down to a small result set and you want to avoid writing hundreds of tiny files You want to reduce overhead with a minimal cost operation Practical example: As a rule of thumb: use coalesce to write smaller results; use repartition to scale up or fix skew.

Q: In what scenario would you use repartition to reduce the number of partitions?

Although coalesce is the typical choice for reducing partitions, repartition to reduce is preferred when data is heavily skewed . coalesce simply merges adjacent partitions on the same executor — if some partitions are very large (due to data skew), those large partitions remain large. This causes stragglers during downstream processing. Scenario: skewed data after a filter or aggregation Also use repartition to reduce partitions when you simultaneously want to repartition by a column for downstream co partitioning — coalesce doesn't support column based repartitioning. The trade off is the cost of the full shuffle, which is acceptable when the skew problem is significant enough to cause task failures or timeouts.

Q: Which version of Spark are you using? Are you working with Spark 3.4?

This is a context setting question the interviewer asks to gauge your familiarity with recent Spark features. A strong answer acknowledges both version specifics and notable improvements. Spark 3.x major improvements over Spark 2.x: Adaptive Query Execution (AQE) — introduced in 3.0, stable in 3.2+. Dynamically re plans queries at runtime based on shuffle statistics Dynamic Partition Pruning (DPP) — pushes partition filters from dimension tables into fact table scans at runtime Improved Pandas UDFs — Spark 3.0 introduced pandas udf with type hints and scalar iterator UDFs for better performance Delta Lake 2.x — better integration with Spark 3.x, Change Data Feed, MERGE performance improvements Spark 3.4 specifics : Enhanced OFFSET and LIMIT pushdown, Spark Connect (remote client/server separation), improved Python UDF vectorization Typical answer structure for interviews: State the version used in your project (e.g., Spark 3.2 on Databricks Runtime 11.x or Spark 3.3 on EMR) Mention one or two features you actively used (e.g., "We relied heavily on AQE for optimizing our star schema joins and enabled DPP to reduce fact table scan times by ~40%")

Q: What is Adaptive Query Execution (AQE) in Spark, and how does it improve performance?

Adaptive Query Execution (AQE) is a Spark optimization framework (stable since Spark 3.2) that re optimizes query plans at runtime using statistics collected from completed shuffle stages, rather than relying solely on compile time estimates. Enabled via: spark.conf.set('spark.sql.adaptive.enabled', 'true') Three core features of AQE: 1. Dynamically coalescing shuffle partitions : After a shuffle, Spark inspects actual partition sizes and merges small partitions into larger ones. This prevents the overhead of processing thousands of tiny tasks when data volume is lower than estimated. 2. Dynamically switching join strategies : If a build side of a sort merge join turns out to be small enough at runtime (after filtering), AQE converts it to a broadcast hash join , avoiding the expensive sort merge shuffle. 3. Dynamically optimizing skew joins : AQE detects skewed partitions (much larger than the median) and splits them into smaller sub partitions, distributing the work more evenly across executors. Practical impact: Reduces need for manual tuning of spark.sql.shuffle.partitions Can cut join execution time significantly for skewed data Makes query plans more resilient to stale or inaccurate table statistics AQE is particularly valuable in production pipelines where data volumes fluctuate.

Question 1

What are the Key differences between DataFrame and RDD.

Accepted Answer

RDD (Resilient Distributed Dataset) is Spark's low level abstraction — a distributed collection of JVM objects with no schema. DataFrame is a higher level abstraction built on top of RDD with a named column schema, similar to a relational table. Key differences: Schema : RDDs are untyped; DataFrames have a defined schema with column names and data types Optimization : DataFrames go through the Catalyst optimizer and Tungsten execution engine for automatic query optimization; RDDs do not Performance : DataFrames are generally faster due to whole stage code generation and columnar processing; RDDs incur Java serialization overhead API style : RDDs use functional transformations ( map , filter , flatMap ); DataFrames use SQL like operations ( select , groupBy , join ) Language support : DataFrames offer consistent performance across Python, Scala, Java, and R; RDD performance varies by language (Python is slower due to serialization) Use cases : Use RDDs when you need fine grained control over data layout or when working with unstructured data; prefer DataFrames for structured/semi structured data and analytics workloads

Question 2

Narrow vs wide transformations; shuffling impact and optimization.

Accepted Answer

Narrow transformations are operations where each output partition depends on exactly one input partition — no data movement across executors is required. Examples: map , filter , flatMap , union . Wide transformations require data from multiple input partitions to produce each output partition, triggering a shuffle — data is serialized, written to disk, and transferred over the network. Examples: groupByKey , reduceByKey , join , distinct , repartition . Shuffle impact: Significant network I/O and disk I/O overhead Triggers a new stage boundary in the DAG Can be a major bottleneck for large datasets Increases memory pressure (shuffle spills to disk when memory is exhausted) Optimization strategies: Use reduceByKey instead of groupByKey — pre aggregates locally before shuffle Apply filters as early as possible (predicate pushdown) to reduce shuffle data volume Use broadcast joins for small lookup tables to avoid shuffle joins Tune spark.sql.shuffle.partitions (default 200) to match data size Enable AQE ( spark.sql.adaptive.enabled=true ) to dynamically adjust partition counts post shuffle

Question 3

What strategies can be used to minimize shuffling in Spark?

Accepted Answer

Minimizing shuffle is one of the most impactful Spark performance optimizations: Broadcast joins : When one table is small (< spark.sql.autoBroadcastJoinThreshold , default 10MB), Spark broadcasts it to all executors, eliminating the shuffle entirely Pre partition data : Store data pre partitioned by join/group keys in Parquet/Delta so Spark can exploit partition pruning and avoid full shuffles Use reduceByKey over groupByKey : reduceByKey performs a map side combine (partial aggregation) before shuffling, drastically reducing data transferred Colocate related data : Write joined tables with the same partitioning scheme so joins become narrow transformations Filter early : Apply where / filter before joins or aggregations to reduce the dataset size going into the shuffle Bucketing : Pre bucket tables on join keys using bucketBy in Spark SQL so join operations can skip the shuffle stage entirely Adaptive Query Execution (AQE) : Enable AQE to automatically coalesce small shuffle partitions and switch to broadcast join at runtime when a partition is determined to be small

Question 4

What is the difference between repartition and coalesce in Spark?

Accepted Answer

Both repartition and coalesce change the number of partitions in a DataFrame, but they work differently: repartition(n) : Performs a full shuffle — data is redistributed evenly across all n partitions Can both increase and decrease the number of partitions Produces evenly sized partitions (good for balancing skew) More expensive due to the shuffle, but necessary when you need balanced partitions coalesce(n) : Avoids a full shuffle by merging existing partitions on the same executor — moves data locally Can only decrease the number of partitions (attempting to increase returns the original count) May produce uneven partition sizes since it simply combines existing partitions without redistribution Much cheaper than repartition for reducing partition count Aspect repartition coalesce Shuffle Full shuffle Minimal/no shuffle Direction Increase or decrease Decrease only Partition balance Even Potentially uneven Cost Higher Lower

Question 5

When should you use repartition vs. coalesce?

Accepted Answer

Choose between these operations based on your specific need: Use repartition when: You need to increase the number of partitions (e.g., before a large join or shuffle heavy aggregation) Your data is skewed and you need evenly distributed partitions for downstream processing You want to repartition by a specific column for efficient downstream joins or writes: df.repartition(200, 'customer id') Writing to storage and you want a controlled, uniform number of output files Use coalesce when: You need to reduce partition count before writing small output files (e.g., writing a small final result) After filtering a large DataFrame down to a small result set and you want to avoid writing hundreds of tiny files You want to reduce overhead with a minimal cost operation Practical example: As a rule of thumb: use coalesce to write smaller results; use repartition to scale up or fix skew.

Question 6

In what scenario would you use repartition to reduce the number of partitions?

Accepted Answer

Although coalesce is the typical choice for reducing partitions, repartition to reduce is preferred when data is heavily skewed . coalesce simply merges adjacent partitions on the same executor — if some partitions are very large (due to data skew), those large partitions remain large. This causes stragglers during downstream processing. Scenario: skewed data after a filter or aggregation Also use repartition to reduce partitions when you simultaneously want to repartition by a column for downstream co partitioning — coalesce doesn't support column based repartitioning. The trade off is the cost of the full shuffle, which is acceptable when the skew problem is significant enough to cause task failures or timeouts.

Question 7

Which version of Spark are you using? Are you working with Spark 3.4?

Accepted Answer

This is a context setting question the interviewer asks to gauge your familiarity with recent Spark features. A strong answer acknowledges both version specifics and notable improvements. Spark 3.x major improvements over Spark 2.x: Adaptive Query Execution (AQE) — introduced in 3.0, stable in 3.2+. Dynamically re plans queries at runtime based on shuffle statistics Dynamic Partition Pruning (DPP) — pushes partition filters from dimension tables into fact table scans at runtime Improved Pandas UDFs — Spark 3.0 introduced pandas udf with type hints and scalar iterator UDFs for better performance Delta Lake 2.x — better integration with Spark 3.x, Change Data Feed, MERGE performance improvements Spark 3.4 specifics : Enhanced OFFSET and LIMIT pushdown, Spark Connect (remote client/server separation), improved Python UDF vectorization Typical answer structure for interviews: State the version used in your project (e.g., Spark 3.2 on Databricks Runtime 11.x or Spark 3.3 on EMR) Mention one or two features you actively used (e.g., "We relied heavily on AQE for optimizing our star schema joins and enabled DPP to reduce fact table scan times by ~40%")

Question 8

What is Adaptive Query Execution (AQE) in Spark, and how does it improve performance?

Accepted Answer

Adaptive Query Execution (AQE) is a Spark optimization framework (stable since Spark 3.2) that re optimizes query plans at runtime using statistics collected from completed shuffle stages, rather than relying solely on compile time estimates. Enabled via: spark.conf.set('spark.sql.adaptive.enabled', 'true') Three core features of AQE: 1. Dynamically coalescing shuffle partitions : After a shuffle, Spark inspects actual partition sizes and merges small partitions into larger ones. This prevents the overhead of processing thousands of tiny tasks when data volume is lower than estimated. 2. Dynamically switching join strategies : If a build side of a sort merge join turns out to be small enough at runtime (after filtering), AQE converts it to a broadcast hash join , avoiding the expensive sort merge shuffle. 3. Dynamically optimizing skew joins : AQE detects skewed partitions (much larger than the median) and splits them into smaller sub partitions, distributing the work more evenly across executors. Practical impact: Reduces need for manual tuning of spark.sql.shuffle.partitions Can cut join execution time significantly for skewed data Makes query plans more resilient to stale or inaccurate table statistics AQE is particularly valuable in production pipelines where data volumes fluctuate.

EPAM Interview Questions