Question 1

Outline a data lake architecture using S3, Glue, and Athena.

Accepted Answer

Key components : S3 : cheap, durable object storage — organize as s3://bucket/layer/table/year=2024/month=01/ Glue Catalog : Hive compatible metastore — stores table schemas and partition metadata; Athena uses it to know where data lives Glue ETL : PySpark based serverless jobs for transformation; auto scales without cluster management Athena : serverless SQL on S3 — charges per TB scanned; use Parquet + partitioning to minimize cost Lake Formation : permission layer on top — fine grained access control at table/column level Best practices : partition by date, use Parquet + Snappy, run MSCK REPAIR TABLE or Glue crawler after adding new partitions.

Question 2

Design real-time analytics using Kinesis or Kafka streaming.

Accepted Answer

Kinesis architecture : Kafka alternative (MSK — Managed Streaming for Kafka): When to use which : Kinesis : fully managed, tight AWS integration, easier ops — good for AWS native shops Kafka (MSK) : higher throughput, richer ecosystem (Kafka Connect, Kafka Streams), better for multi cloud Key concerns : shard count in Kinesis (1 MB/s per shard), consumer lag monitoring via CloudWatch, checkpoint management for exactly once delivery.

Question 3

Handle schema evolution in Glue catalog datasets.

Accepted Answer

Additive changes (new columns): safe — Parquet is self describing. Athena queries return NULL for the new column in older files. Breaking changes (renamed/dropped columns): 1. Write new data to a versioned path ( v2/orders/ ) 2. Create a new Glue table pointing to v2/ path 3. Migrate Athena views and downstream jobs to new table 4. Deprecate old table after consumers migrate Schema drift handling : Glue DynamicFrame's resolveChoice handles type conflicts (e.g., a column with mixed int/string values):

Question 4

Optimize Redshift queries for large datasets.

Accepted Answer

Distribution style : choose based on query patterns DISTKEY(customer id) : hash distribute on join key — collocates matching rows on same node DISTSTYLE ALL : replicate small dimension tables to every node DISTSTYLE EVEN : default; even distribution but requires shuffle on joins Sort keys : SORTKEY(date) : range queries on date skip unsorted blocks INTERLEAVED SORTKEY(date, region) : multi column filters benefit equally from any column Maintenance : VACUUM FULL table — reclaims space from deleted rows, re sorts data ANALYZE table — updates statistics for query planner Query tips : Use EXPLAIN to check join types and distribution steps Avoid SELECT — Redshift is columnar; scan only needed columns Use result caching (automatic for identical queries) Increase WLM queue slots for concurrent heavy queries

Question 5

Design a CDC system using Lambda and Kinesis.

Accepted Answer

Event format (DMS → Kinesis): Lambda processing : Parse operation (insert/update/delete) For delete : soft delete or remove from target For insert/update : upsert into target (DynamoDB put item , or batch to S3 for Redshift MERGE ) Alternatives : Debezium (self managed): reads PostgreSQL WAL → Kafka → more flexibility AWS Glue Streaming : micro batch CDC from Kinesis into S3 Key concern : ordering — Kinesis is ordered per partition key; use order id as partition key to guarantee ordered updates per record.

Question 6

Use S3 partitioning and file formats for performance.

Accepted Answer

Partitioning strategy : Athena/Glue uses Hive style partitions — queries with WHERE year=2024 AND month=01 skip all other partitions. Partition granularity : daily is usually right. Hourly creates too many small files; monthly causes too much data per scan on date range queries. File format recommendations : Parquet + Snappy : best for analytics — columnar, splittable, ~5x compression Parquet + ZSTD : better compression ratio when storage cost is priority Avro : use for Kinesis/Kafka ingestion (row based, schema embedded) File size : target 128 MB–1 GB per file. Many small files hurt performance: S3 lifecycle policies : auto transition data older than 90 days to S3 Glacier for cost savings.

Question 7

Manage scheduler and orchestration via Step Functions or Airflow.

Accepted Answer

Step Functions (AWS native): State machine for orchestrating AWS services (Glue, Lambda, ECS, EMR) Visual workflow; built in retry + error handling per state Good for event driven pipelines triggered by S3 events or API calls Limited Python logic — best for AWS service orchestration Airflow (MWAA — Managed Workflows for Apache Airflow) : Python defined DAGs — full programming flexibility Hundreds of operators (S3, Glue, Redshift, BigQuery, Spark, etc.) Backfill support, SLA monitoring, rich UI Better for complex multi step pipelines with branching logic When to use which : Step Functions: lightweight event driven AWS pipelines, microservice orchestration Airflow (MWAA): complex scheduled data pipelines, data engineering workflows with many operators Combined approach : use Step Functions for event driven triggers → call Airflow API to kick off DAGs for heavy processing.

Question 8

Build cost-cutting strategies for Spark on EMR.

Accepted Answer

Spot instances : use for task nodes (not master/core) — 60 80% cost savings. Configure with Spot interruption handling (checkpoint before spot reclaim) Instance fleet : mix instance types to improve Spot availability Auto scaling : scale task nodes based on YARN memory pressure — scale down when idle Right size : profile jobs first (Spark History Server) — don't over provision Transient clusters : spin up cluster per job, terminate after completion — avoid idle master node cost Storage optimized formats : Parquet + compression reduces data scanned and shuffled Partition tuning : spark.sql.shuffle.partitions should match data size (~128MB per partition) — too many small partitions wastes scheduling overhead EMR Serverless : for bursty workloads — pay only for vCPU hours used, no cluster management

Amazon Interview Questions