Q: Explain difference between managed vs external tables in Databricks.

Managed Table External Table Data location Databricks controlled storage User specified path (ADLS, S3) Drop behavior DROP TABLE deletes data AND metadata DROP TABLE deletes only metadata; data stays Use case Fully managed, no sharing with other tools Data shared with Spark, Presto, ADF, etc. Unity Catalog Data governed by UC Data at external location, registered in UC Best practice for Amdocs : use external tables for production data — Databricks workspace can be recreated without losing data. The data lives safely in ADLS regardless of Databricks cluster/workspace lifecycle.

Question 1

What is a Data Pipeline, and what are its key components?

Accepted Answer

A data pipeline is a series of automated steps that move and transform data from source systems to destinations. Key components : 1. Ingestion : read data from sources (databases, APIs, files, streams) 2. Transformation : clean, enrich, aggregate, join — business logic applied 3. Storage : write to target (data lake, warehouse, downstream DB) 4. Orchestration : schedule and manage dependencies between steps (Airflow, ADF, Databricks Workflows) 5. Monitoring : track job status, row counts, data freshness, latency — alert on failures 6. Error handling : dead letter queues, retry logic, alerting At Amdocs (telecom context) : pipelines ingest CDRs (Call Detail Records), subscriber events, billing data from BSS/OSS systems → transform into analytics ready tables for churn prediction, revenue assurance, and network analytics.

Question 2

How do you implement CDC (Change Data Capture) in Azure?

Accepted Answer

Option 1 — Azure Data Factory + SQL Change Tracking : ADF reads changes using the CHANGETABLE(CHANGES subscribers, @last sync version) function — captures INSERTs, UPDATEs, DELETEs. Option 2 — Debezium + Event Hub : Debezium connector reads PostgreSQL/MySQL WAL Publishes change events to Azure Event Hub Spark Structured Streaming processes events → MERGE INTO Delta Lake Option 3 — ADF Incremental Copy with watermark : Store last loaded ts in a control table ADF Lookup → filter source with WHERE updated at last loaded ts → Copy → update watermark Telecom use case : capture real time changes in subscriber plans and propagate to the billing analytics pipeline within minutes.

Question 3

What are the advantages of Delta Lake in telecom analytics?

Accepted Answer

Telecom generates billions of CDRs (call records), network events, and billing records daily — Delta Lake's features directly address the challenges: ACID transactions : concurrent billing and analytics jobs don't corrupt each other MERGE INTO (upsert) : subscriber records are frequently updated (plan changes, activations) — MERGE handles inserts and updates atomically Time travel : audit historical subscriber states for dispute resolution without maintaining separate archive tables Schema evolution : mergeSchema=true handles new network event types (5G metrics) without breaking existing pipelines Scalability : Delta on ADLS handles petabytes — entire CDR history for millions of subscribers Z ordering on msisdn (subscriber ID) : co locates subscriber data for fast per subscriber queries Streaming + batch unified : same Delta table receives real time CDR streams and nightly billing batch loads

Question 4

How do you optimize Spark jobs for telecom call data (billions of records)?

Accepted Answer

Partitioning : partition CDR tables by date — daily query patterns only scan relevant partitions: Broadcast small dimensions : subscriber plan details are small relative to CDR volume: Repartition before joins : cdr df.repartition(400, "msisdn") before joining on subscriber ID — avoids shuffle during the join. Avoid UDFs : use built in functions for CDR parsing (substring, regexp extract) rather than Python UDFs. AQE : spark.sql.adaptive.enabled=true — handles CDR skew automatically (some subscribers have 100x more calls). Predicate pushdown : WHERE call date = '2024 01 15' on partitioned table — Spark skips all other date partitions. Right size shuffle partitions : billions of CDRs × many columns → set spark.sql.shuffle.partitions=2000 (not default 200).

Question 5

Explain partitioning vs bucketing with an example.

Accepted Answer

Partitioning : creates directory hierarchy by column value. Queries filtering on the partition column skip unmatched directories entirely. Bucketing (Hive/Spark) : hash distributes rows into a fixed number of files based on a column. Eliminates shuffle when joining two tables bucketed on the same column with the same count. Key difference : Partitioning: reduces data scanned (partition pruning) Bucketing: reduces shuffle in joins (when both sides are bucketed on join key with same bucket count) Don't partition on high cardinality columns (msisdn) — creates millions of tiny directories → use bucketing instead

Question 6

How do you implement error handling in ADF pipelines?

Accepted Answer

Activity level error handling (if/else branching): Retry configuration : set retryCount: 3, retryIntervalInSeconds: 30 on individual activities. Dead letter pattern : in Copy Activity, set enableSkipIncompatibleRow: true and logStorageSettings to write rejected rows to a separate ADLS path. Pipeline level : wrap in a Try Catch using ADF's IfCondition + error variable @activity('Copy').output.errors . Monitoring : ADF Monitor tab shows all failed runs with error messages. Send to Log Analytics for persistent alerting.

Question 7

How do you ensure data quality in pipelines?

Accepted Answer

Layer by layer checks : At ingestion (Bronze) : Schema validation: reject messages missing required fields Null checks on mandatory keys (subscriber id, event ts) Row count comparison vs source system At transformation (Silver) : Referential integrity: every plan id exists in dim plans Business rules: call duration sec = 0 , charge amount = 0 Deduplication: no duplicate cdr id At serving (Gold) : Aggregate totals match Silver layer (sum reconciliation) Data freshness: MAX(load ts) < 2 hours old Tools : Response : quarantine bad records to dead letter Delta table, continue pipeline with valid data, alert data owners.

Question 8

Explain difference between managed vs external tables in Databricks.

Accepted Answer

Managed Table External Table Data location Databricks controlled storage User specified path (ADLS, S3) Drop behavior DROP TABLE deletes data AND metadata DROP TABLE deletes only metadata; data stays Use case Fully managed, no sharing with other tools Data shared with Spark, Presto, ADF, etc. Unity Catalog Data governed by UC Data at external location, registered in UC Best practice for Amdocs : use external tables for production data — Databricks workspace can be recreated without losing data. The data lives safely in ADLS regardless of Databricks cluster/workspace lifecycle.

Amdocs Interview Questions