Virtusa Interview Questions

A Data Lakehouse combines: Data Lake : cheap object storage (S3, ADLS), any data format, flexible schema Data Warehouse : ACID transactions, schema enfo...

• Explain Data Lakehouse architecture.

  A Data Lakehouse combines: Data Lake : cheap object storage (S3, ADLS), any data format, flexible schema Data Warehouse : ACID transactions, schema enforcement, fast SQL Implemented by adding a table format layer (Delta Lake, Apache Iceberg, Apache Hudi) on top of object storage. Medallion architecture (typical Lakehouse pattern): Key capabilities : ACID transactions on Parquet files Schema enforcement and evolution Time travel / versioning DML operations (UPDATE, DELETE, MERGE) Performance: Z o

• How do you perform CDC (Change Data Capture) in Databricks?

  Approach 1 — Autoloader + MERGE (most common): Approach 2 — Delta Live Tables (DLT) with APPLY CHANGES INTO : DLT handles out of order events, deduplication, and SCD Type 1/2 automatically.

• What is the difference between structured, semi-structured, and unstructured data?

  Type Schema Examples Storage Structured Fixed, predefined Relational DB tables, CSV with schema RDBMS, data warehouse Semi structured Flexible, self describing JSON, XML, Avro, Parquet Data lake, MongoDB Unstructured No schema Images, videos, PDFs, audio, free text Object storage (S3, ADLS) In data engineering : Structured : OLTP databases (orders, customers) → ETL to warehouse Semi structured : API responses (JSON), clickstream events, IoT telemetry → ingest to Bronze, parse in Silver Unstructu

• Explain Data Skew. How to handle it?

  Data skew is when data is unevenly distributed across partitions — one partition has significantly more data than others. Symptoms : Spark UI shows one task taking 10x longer than others; overall job waits for the single slow task. Causes : high frequency join key (e.g., NULL or unknown country), hot partition in time series data. Solutions : 1. AQE skew handling (automatic): 2. Salting (for join skew): 3. Filter and separate : process the skewed key separately with custom logic, union with the

• How do you implement CI/CD for ADF and Databricks?

  ADF CI/CD : Store ADF pipeline JSON definitions in Git (ARM templates or ADFV2 JSON) Dev → Git branch → PR → merge to main → Azure DevOps pipeline publishes to Test → manual approval → Production Databricks CI/CD : Notebooks/Python files in Git GitHub Actions (or Azure DevOps) triggers on PR: 1. pytest unit tests on local[1] Spark 2. Deploy notebooks to Databricks workspace via databricks workspace import 3. Run integration tests on staging cluster 4. Deploy Databricks Job to production on merge

• Explain Window functions in PySpark.

  Window functions compute a value for each row using a set of rows defined by a window specification. Note : window functions require a shuffle (similar to groupBy ) — ensure the partition column has good cardinality.

• What are best practices for handling large files in Data Lake?

  Target file size : 128 MB – 1 GB per file. Smaller files hurt because: Each file = one ADLS/S3 API call overhead Query engines create one task per file → too many tiny tasks Avoid small files from streaming : Avoid very large files : Files 1 GB limit parallelism — all readers use one task for that file Use repartition() or coalesce() before writing to control file count Partitioning : Partition by date for time series data — each partition should be ≥ 1 GB total Don't partition by high cardinali

• Explain columnar storage benefits.

  In columnar storage (Parquet, ORC), data is stored column by column rather than row by row. Benefits for analytics : 1. Column pruning : reading only the columns in your SELECT — if a table has 100 columns and you query 5, columnar reads only those 5 columns from disk (vs row based which reads all 100) 2. Better compression : each column contains the same data type and often similar values (all integers, or strings from a small set) → high compression ratio with dictionary encoding, run length e