GCP FAQ & Resources
A recommended architecture includes: Cloud Pub/Sub for ingesting streaming IoT data Cloud Dataflow (Apache Beam) to process and transform the data strea...
Scenario: Real-Time IoT Data Ingestion on GCP Question: Your organization needs to design a data pipeline that ingests real-time streaming data from IoT devices into GCP for analytics. Which GCP services would you use, and how would you architect the pipeline?
A recommended architecture includes: Cloud Pub/Sub for ingesting streaming IoT data Cloud Dataflow (Apache Beam) to process and transform the data stream with windowing strategies BigQuery to store processed and aggregated data for analytics Cloud Functions (optional) for lightweight event based processing Key considerations: ensure idempotent processing , configure dead letter topics for failed messages, and monitor latency and throughput using Cloud Monitoring .
Coding: Write a Python script to upload a file to a specific Cloud Storage bucket.
This uses the Google Cloud Storage client library to create a client, select the bucket, and upload the file to the specified destination path.
Scenario: Optimizing BigQuery Query Performance Question: You have a large dataset in BigQuery and need to improve query performance. What strategies would you use?
Use partitioned tables (e.g., by date) to limit data scanned Apply clustering on frequently filtered columns Avoid SELECT ; project only required columns Use materialized views for recurring queries Monitor expensive queries via INFORMATION SCHEMA.JOBS BY PROJECT
Scenario: Implementing a Data Lake on GCP Question: How would you design a data lake architecture on GCP?
A typical data lake consists of: Bronze (Raw Layer): Cloud Storage for raw ingested data Silver (Processed Layer): Dataflow or Dataproc for cleaning and transformation Gold (Curated Layer): BigQuery or Cloud SQL for analytics ready data Orchestration: Cloud Composer or Workflows Security: IAM roles and VPC Service Controls for access management
Coding: Write a Python function that executes a BigQuery SQL query and returns results as a Pandas DataFrame.
Use the BigQuery Python client to execute the query and convert the results using to dataframe() . Ensure pandas and db dtypes are installed.
Scenario: Orchestrating Complex GCP Data Workflows Question: How would you automate workflows involving multiple GCP services?
Use Cloud Composer (managed Apache Airflow ) to: Define DAGs for orchestrating BigQuery , Dataflow , and Cloud Functions tasks Manage dependencies, retries, and SLAs declaratively Share metadata using XComs or Cloud Storage between tasks
Scenario: Row-Level Security in BigQuery Question: How do you restrict data access based on user roles in BigQuery?
Use Row Access Policies ( CREATE ROW ACCESS POLICY ) to filter rows dynamically based on the querying user Implement authorized views for role based access to specific columns or rows Combine IAM permissions with session based filters for fine grained control
Coding: How would you build a Dataflow pipeline in Java for streaming data from Pub/Sub into BigQuery?
High level approach: Read messages using PubsubIO.readStrings() Transform data using ParDo with custom DoFn logic Write results using BigQueryIO.writeTableRows() Enable streaming mode and autoscaling Handle schema validation and route failures to dead letter tables