Question 1

How have you orchestrated workflows in Apache Airflow?

Accepted Answer

Apache Airflow orchestrates pipelines as DAGs (Directed Acyclic Graphs) — Python files defining tasks and their execution order. Key patterns I use: Advanced patterns: ExternalTaskSensor — wait for upstream DAG to complete BranchPythonOperator — conditional branching based on data conditions TaskGroup — logical grouping for complex DAGs Dynamic task mapping with expand() for variable fan out XComs for passing metadata between tasks Production best practices: Set catchup=False , configure on failure callback for Slack alerts, use Airflow Connections for credential management, set SLAs per task.

Question 2

What is the difference between HDFS (Hadoop Distributed File System) and Google Cloud Storage (GCS)?

Accepted Answer

Both store large scale distributed data, but they differ fundamentally in architecture and operational model: Feature HDFS GCS Architecture Tightly coupled compute+storage Decoupled (object storage) Scalability Scale by adding nodes Infinitely scalable, managed by Google Compute dependency Jobs must run on HDFS cluster Any compute can access GCS Consistency Strong consistency Strong consistency (since 2021) Cost model CapEx (hardware) + OpEx (ops) Pay per GB stored + per request fees Metadata NameNode (SPOF historically) Google managed, no SPOF Replication 3x default block replication Multi region durability built in Performance High throughput for co located compute Slightly higher latency for small files Operations Requires Hadoop admin expertise Fully managed, no cluster management Key architectural difference: HDFS is compute storage coupled — data lives on the same nodes running MapReduce/Spark. GCS is object storage — compute (Dataproc, Dataflow) spins up separately, reads from GCS, then shuts down. When HDFS is still used: Legacy on prem Hadoop deployments, workloads requiring extremely low latency local disk access. Why GCS wins for new cloud native workloads: No cluster management, cost effective cold storage tiers, IAM based security, versioning, and lifecycle policies.

Question 3

Explain Google Cloud Dataproc — why it is used, when to use it, and how it works.

Accepted Answer

Google Cloud Dataproc is a fully managed service for running Apache Spark, Hadoop, Hive, and Pig workloads on GCP. Why use Dataproc: Managed cluster lifecycle — create, scale, and delete clusters via API/CLI Pay only for cluster uptime — spin up for a job, then delete (no idle costs) Integrates natively with GCS, BigQuery, Bigtable, Pub/Sub Pre installed Spark, Hadoop, and ecosystem tools How it works: 1. You create a cluster specifying master + worker node types and count 2. Dataproc provisions GCE VMs, installs Hadoop/Spark, and configures YARN 3. Data is read from GCS (not HDFS) for ephemeral cluster pattern 4. Submit jobs via gcloud dataproc jobs submit or Airflow DataprocSubmitJobOperator 5. Cluster can be deleted after jobs complete Cluster types: Standard: Fixed size cluster for long running workloads Single node: Development/testing Autoscaling: Scales workers based on YARN metrics Dataproc Serverless: No cluster management — submit job directly, GCP handles resources When to use Dataproc vs other GCP services: Use Dataproc for existing Spark/Hadoop workloads being migrated from on prem Use Dataflow for streaming pipelines and Apache Beam Use BigQuery for SQL analytics without cluster management Use Dataproc Serverless for ad hoc Spark jobs

Question 4

What is the architecture of Pub/Sub and how does message flow between publisher and subscriber?

Accepted Answer

Google Cloud Pub/Sub is a fully managed asynchronous messaging service for event driven architectures. Core concepts: Topic: Named resource to which publishers send messages Subscription: Named resource attached to a topic; subscribers pull from it Publisher: Sends messages to a topic Subscriber: Pulls messages from a subscription (or receives push deliveries) Message: Data (up to 10 MB) + optional attributes (key value pairs) Message flow: 1. Publisher sends a message to a topic 2. Pub/Sub durably stores the message (replicated across 3+ zones) 3. Each subscription receives its own copy of every message 4. Subscribers acknowledge messages after processing; unacknowledged messages are redelivered 5. Messages are retained until acknowledged or until the retention period expires (default 7 days) Delivery modes: Pull: Subscriber explicitly calls pull() — used for batch/streaming processing (Dataflow, Spark) Push: Pub/Sub delivers to a webhook URL — used for Cloud Functions, Cloud Run At least once delivery: Pub/Sub guarantees messages are delivered at least once but may deliver duplicates. Make consumers idempotent . Ordering: Enable message ordering on subscriptions where sequence matters. Use case: Decouple microservices, fan out events to multiple consumers, buffer for Dataflow/BigQuery streaming inserts.

Question 5

What is a Materialized View in Business Analytics, and how is it different from a normal view?

Accepted Answer

Normal View: A stored SQL query — no data is physically stored. Every time you query the view, the underlying query re executes against the base tables. Materialized View: The query result is physically stored (cached on disk). The stored result is served on query, avoiding re execution. Feature Normal View Materialized View Data storage None — query only Physical table like storage Query performance Slow for complex queries Fast — reads pre computed result Data freshness Always current Stale until refreshed Refresh needed No Yes (scheduled or on demand) Storage cost Minimal Proportional to result set Write operations Not directly possible Can be incrementally updated Refresh strategies: Full refresh: Truncate and recompute entire result (simple, expensive) Incremental refresh: Only update changed rows (efficient, complex) On demand: Manually triggered when freshness is acceptable BigQuery Materialized Views: BigQuery auto refreshes within 5 minutes and uses smart rewriting to serve queries from the materialized view when possible. Use in analytics: Pre aggregate expensive joins and aggregations for dashboards that need sub second query performance.

Question 6

What are External Tables and how do they work internally?

Accepted Answer

An External Table (also called Foreign Table) is a table definition that points to data stored outside the database — in cloud storage, another database, or a file system — without loading the data into the database itself. How they work: The database stores only the table metadata (schema, location, format) When you query the external table, the database's query engine reads data directly from the external location at query time No data is copied or stored in the database BigQuery External Tables: Athena (AWS) External Tables via Glue: Advantages: Query data in place without ETL into the warehouse No storage duplication Always reflects the latest files in the storage location Disadvantages: Slower than native tables (no caching, no statistics) No DML (INSERT/UPDATE/DELETE) Performance depends on file format and layout Use case: Ad hoc analysis of raw data lake files before deciding to load them into the warehouse.

Question 7

How is a data pipeline created and managed in GCP?

Accepted Answer

GCP offers multiple services for building and managing data pipelines: Batch pipelines: 1. Cloud Composer (managed Airflow) — orchestrate multi step batch ETL 2. Dataproc — run PySpark/Hadoop batch jobs 3. BigQuery Scheduled Queries / dbt — SQL based transformations 4. Dataflow (Apache Beam) — batch or streaming unified model Streaming pipelines: 1. Pub/Sub → Dataflow → BigQuery : standard real time ingestion 2. Pub/Sub → Spark Structured Streaming on Dataproc → GCS/BigQuery 3. BigQuery Streaming Inserts for low latency writes Example: End to end batch pipeline on GCP: Management and monitoring: Cloud Logging — capture job logs, set log based alerts Cloud Monitoring — metrics dashboards for Dataflow/Dataproc job health Pub/Sub dead letter topics — capture unprocessable messages Cloud Composer UI — DAG run history, task logs CI/CD for pipelines: Store DAGs/scripts in GitHub GitHub Actions → run tests → deploy to GCS bucket → Composer picks up new DAGs Use Terraform for infrastructure (Dataproc cluster configs, IAM)

Question 8

How is a data pipeline created and managed in Apache Airflow?

Accepted Answer

Apache Airflow manages pipelines through DAG files — Python scripts placed in the dags/ folder. Creating a pipeline: Managing pipelines: Monitoring: Airflow UI shows DAG run history, task durations, retry counts Alerts: on failure callback for Slack/PagerDuty notifications Scaling: CeleryExecutor or KubernetesExecutor for parallel task execution Variables/Connections: Store credentials and config in Airflow's encrypted store Versioning: DAG files in Git; CI/CD deploys to dags/ folder Testing: Use airflow dags test dag id execution date for local DAG testing

Nielsen Interview Questions