HCL Interview Questions

Output: Key concepts: Window.partitionBy('Name').orderBy(col('Timestamp').desc()) creates a window per unique name, ordered newest first row number() as...

• Given a DataFrame with columns Name and Age, remove duplicate rows based on the Name column, keeping only the latest entry based on a given timestamp column Timestamp.

  Output: Key concepts: Window.partitionBy('Name').orderBy(col('Timestamp').desc()) creates a window per unique name, ordered newest first row number() assigns 1 to the most recent record per name Filtering on rn == 1 retains only the latest row Avoid dropDuplicates('Name') here — it does not guarantee which duplicate is kept

• Given a DataFrame with columns Department and Salary, calculate the average salary for each department.

  Output: Alternative using SQL: Key points: groupBy followed by agg(avg(...)) is the idiomatic PySpark approach spark round avoids floating point precision issues in the output Both DataFrame API and Spark SQL produce identical results — choose based on team preference

• Given a data frame with columns Item and Sales, calculate the percentage contribution of each item to the total sales.

  Output: Approach 2: Using a single aggregation + join (alternative) Key points: The window function approach is preferred — it avoids a cross join and runs in a single pass Window.rowsBetween(unboundedPreceding, unboundedFollowing) without PARTITION BY makes the entire DataFrame one window, effectively computing a global sum Multiplying by 100 and rounding gives a clean percentage