JP Morgan Chase Interview Questions

Spark follows a master worker model: Driver : runs the user program, builds the DAG, coordinates with the cluster manager, collects results Cluster Mana...

• What is Spark's architecture?

  Spark follows a master worker model: Driver : runs the user program, builds the DAG, coordinates with the cluster manager, collects results Cluster Manager : allocates resources (YARN, Kubernetes, Standalone) Executors : JVM processes on worker nodes that run tasks and store cached data Execution flow: action triggers → DAG Scheduler splits into stages at shuffle boundaries → Task Scheduler sends tasks (one per partition) to executors → results returned to driver.

• Explain the difference between bucketing and partitioning in Spark.

  Partitioning : splits data into subdirectories by column value ( year=2024/month=01 ). Enables partition pruning — Spark skips non matching directories entirely. Bucketing : hashes rows into a fixed number of buckets by a key column. Eliminates shuffle when joining two tables bucketed on the same key with the same bucket count. Partitioning helps with filtering; bucketing helps with joins. They can be combined.

• Write an SQL query to find the second-highest salary from the Employee table.

• How would you handle skewed data in Spark?

  Broadcast join : if one side is small enough, replicate it to all executors — no shuffle at all AQE skew handling : spark.sql.adaptive.skewJoin.enabled=true — Spark 3+ automatically splits large skewed partitions Salting : add a random suffix to the skewed join key; explode the other side to match each salt value Repartition : df.repartition(n, col("key")) to redistribute more evenly before joins

• What is the difference between map() and flatMap() in Spark?

  map(f) : applies f to each element and returns one output per input — 1:1 transformation flatMap(f) : applies f to each element and flattens the result — f returns a collection, resulting in 1:N transformation Common use: flatMap for tokenizing sentences, exploding arrays.

• Explain the concept of DataFrames in Spark.

  A DataFrame is a distributed table with named, typed columns — conceptually like a SQL table or Pandas DataFrame, but partitioned across a cluster. Key properties: Processed by the Catalyst optimizer for performance Lazy evaluation — transformations build a plan; actions trigger execution Interoperable with SQL: df.createOrReplaceTempView("t"); spark.sql("SELECT...") Available in Python, Scala, Java, R Preferred over RDDs for all structured/semi structured workloads due to 2 5x better performanc

• How would you implement a one-to-many relationship in a database?

  Add a foreign key on the "many" side pointing to the primary key of the "one" side: One customer → many orders. Query with JOIN ON orders.customer id = customers.customer id .

• Write a SQL query to find the top 3 highest-paid employees in each department.

  DENSE RANK ensures ties at rank 3 are both included.