Question 1

How do you implement parallel copies in Azure Data Factory (ADF) using partitioning?

Accepted Answer

ADF's parallel copy feature in the Copy Activity enables reading from a source using multiple concurrent threads or connections, significantly improving throughput for large datasets. 1. Physical Partitioning (for relational sources) For Azure SQL, Oracle, SAP HANA, and similar sources, ADF can partition the source query by: Physical partitions : Use existing table partitions (fastest — no extra scanning) Dynamic range : ADF splits a numeric or date column into equal sized ranges Mod partitioning : ADF distributes rows using modulo on a column Configuration in Copy Activity (Source tab): ADF spawns multiple readers in parallel, each querying a slice of the range: 2. Degree of Copy Parallelism (DIU and parallel copy count) 3. File based Parallel Copy For ADLS/Blob sources with multiple files, ADF automatically parallelizes reading across files: Set wildcardFolderPath or prefix to select multiple files Each file is read in a separate thread up to the parallelCopies limit 4. ForEach based Parallelism For table level parallelism, use ForEach Activity with batchCount (max 50): Performance tips: Match parallelCopies to source system's concurrent connection limit Higher DIUs increase network bandwidth between source and sink Use staging (Azure Blob as intermediate) for complex copy scenarios to unlock higher throughput

Question 2

Write Python code to replace vowels in a string with spaces.

Accepted Answer

Here are multiple Python approaches to replace vowels in a string with spaces: 1. Using str.translate() — most Pythonic and fast 2. Using regex substitution 3. Using list comprehension 4. Using a loop (explicit, beginner friendly) Performance comparison: str.translate() is fastest — C level implementation, single pass re.sub() is fast and most flexible (supports complex patterns) List comprehension is readable and efficient Loop is clearest for beginners but slowest Testing all approaches:

Question 3

How do you implement data encryption at rest and in transit in ADLS?

Accepted Answer

ADLS Gen2 provides multiple layers of encryption to protect data both at rest and during transmission. Encryption at Rest: 1. Azure Storage Service Encryption (SSE) — enabled by default All data written to ADLS Gen2 is automatically encrypted using AES 256 Transparent to applications — no code changes required Covers all stored data: blobs, metadata, snapshots, and backups 2. Service Managed Keys (default) Microsoft manages the encryption keys Automatic key rotation No configuration required 3. Customer Managed Keys (CMK) via Azure Key Vault You control and own the encryption keys Keys stored in Azure Key Vault (HSM protected if using Key Vault Premium) Required for compliance scenarios (HIPAA, GDPR, FedRAMP) 4. Infrastructure Encryption (Double Encryption) Enable at account creation time for two layers of encryption (platform managed + service managed) Cannot be enabled after account creation Encryption in Transit: 1. HTTPS only (Secure Transfer Required) Enforce HTTPS on all requests to the storage account 2. TLS Version Enforcement Enforce minimum TLS version (1.2 recommended) 3. Private Endpoints All traffic flows through Azure backbone (VNet) — never over public internet Eliminates data in transit exposure to public network entirely 4. ABFS Driver (Azure Blob File System) Always uses HTTPS by default when accessing ADLS Gen2 Compliance verification: Azure Policy can enforce Secure transfer required and minimum TLS version across subscriptions Microsoft Defender for Storage detects unencrypted HTTP requests

Question 4

Describe the use of Azure Synapse Analytics and how it integrates with other Azure services.

Accepted Answer

Azure Synapse Analytics is a unified analytics platform that combines big data analytics, data warehousing, data integration, and visualization in a single workspace. Core components: 1. Dedicated SQL Pool (formerly Azure SQL DW) Massively Parallel Processing (MPP) architecture Optimized for large scale analytical queries (petabyte scale) Columnstore indexes, result set caching, workload management Cost scales with DWU (Data Warehouse Units) 2. Serverless SQL Pool Query data directly in ADLS Gen2 using T SQL without provisioning any cluster Pay per query (TB of data processed) Perfect for ad hoc exploration of raw/parquet/delta files 3. Apache Spark Pool Managed Spark clusters within Synapse workspace Supports PySpark, Scala, .NET Spark Integrates with Delta Lake (via Synapse Link for Delta tables) 4. Synapse Pipelines ADF compatible data integration service embedded in Synapse Same connectors, activities, and triggers as ADF 5. Synapse Link Near real time analytical access to operational data without ETL Supported sources: Azure Cosmos DB, Dataverse, SQL Server 2022 Integration with Azure services: Service Integration ADLS Gen2 Native storage for all Synapse assets; read/write Parquet, Delta, CSV Azure Data Factory Shared pipeline technology; ADF can trigger Synapse notebooks Power BI Direct publish to Power BI workspace; live connection from Synapse Studio Azure ML Train models on Synapse Spark data; score data in pipelines Azure Purview Data lineage scanning and classification across Synapse assets Azure Key Vault Secrets for linked services and credentials Event Hub / Kafka Stream ingestion into Spark Structured Streaming Azure Monitor Diagnostics, metrics, alerting Typical architecture:

Question 5

How do you implement Continuous Integration and Continuous Deployment (CI/CD) in Azure DevOps?

Accepted Answer

CI/CD in Azure DevOps automates the testing and deployment of data engineering artifacts — ADF pipelines, Databricks notebooks, dbt models, SQL scripts — through controlled stages. 1. Repository Setup (Azure Repos or GitHub) Store all code in Git: ADF JSON templates, Databricks notebooks, Python scripts, Terraform Use feature branches for development; merge to main via pull requests Enforce branch policies: require at least 1 reviewer, pass CI build before merge 2. Continuous Integration (CI) Pipeline 3. Multi Stage Deployment Pipeline 4. Environment Gates and Approvals In Azure DevOps, create Environments (dev, staging, prod) Add Approval checks on the prod environment — require manual sign off before deployment 5. Variable Groups and Secrets 6. Databricks CI/CD (notebooks + jobs) Data engineering CI/CD flow: PR created → CI: lint + unit tests → Merge to main → Deploy to dev → Integration tests → Approval → Deploy to prod

Question 6

Explain the role of metadata in data modeling and data architecture.

Accepted Answer

Metadata is data about data — it describes structure, origin, meaning, lineage, and usage of data assets. In data engineering, metadata is the backbone of data governance, discoverability, and trust. Types of Metadata: 1. Technical Metadata Schema definitions: column names, data types, constraints, primary/foreign keys File metadata: format (Parquet/CSV), size, location, partition keys, row count Table statistics: cardinality, null counts, min/max values (used by query optimizers) 2. Operational Metadata Pipeline execution logs: run times, row counts processed, error messages Data freshness: last updated ts , watermarks, load frequencies Data quality metrics: completeness %, duplicate rates, schema drift history 3. Business Metadata Business definitions: what does customer lifetime value mean to the organization? Data ownership: which team owns the orders table? Sensitivity classification: PII, PHI, confidential, public Business glossary: standardized terminology across departments 4. Lineage Metadata Source to target mappings: where did each column come from? Transformation history: which ADF pipeline, Databricks notebook, or dbt model created this table? Impact analysis: which downstream reports are affected if sales.orders changes? Role in Data Modeling: Role in Data Architecture: Data Catalog (Purview): Auto scan ADLS, Synapse, SQL databases to build metadata inventory Unity Catalog (Databricks): Centralized metadata store with column level governance dbt : Generates schema documentation and column level lineage automatically Hive Metastore : Stores partition metadata and table locations for Spark/Hive query planning Why metadata matters: Enables self service analytics — users find the right data without asking the data team Powers query optimization — statistics guide join ordering, partition pruning Supports data trust — lineage shows where data came from; quality metrics show reliability Drives compliance — know what PII exists, where it flows, and who can access it Best practice: Treat metadata as a first class product. Document schemas in dbt YAML, scan assets into Purview, and embed ownership + sensitivity tags at creation time.