AWS Kinesis
Amazon Kinesis is a fully managed AWS platform for collecting, processing, and analyzing real time streaming data at any scale. It enables you to ingest...
What is Amazon Kinesis, and what are its primary components?
Amazon Kinesis is a fully managed AWS platform for collecting, processing, and analyzing real time streaming data at any scale. It enables you to ingest millions of records per second from sources like clickstreams, IoT devices, and application logs. Its primary components are: Kinesis Data Streams captures and stores data streams in real time for custom processing by consumer applications Kinesis Data Firehose a fully managed delivery service that loads streaming data into destinations such as
How does Amazon Kinesis Data Streams ensure data durability and availability?
Kinesis Data Streams synchronously replicates data across three Availability Zones within an AWS region. This ensures both high availability and data durability if one AZ experiences an outage, the data remains accessible from the other two. This replication happens automatically and transparently, requiring no configuration from the user.
Explain the concept of a shard in Kinesis Data Streams.
A shard is the fundamental unit of capacity in a Kinesis Data Stream . Each shard provides a fixed throughput of 1 MB/sec (or 1,000 records/sec) for writes and 2 MB/sec for reads. The total capacity of a stream is the sum of the capacities of its shards. For example, a stream with 4 shards can ingest up to 4 MB/sec and emit up to 8 MB/sec . You choose the number of shards based on your expected data throughput, and each record is routed to a specific shard based on its partition key .
How can you scale a Kinesis Data Stream?
You scale a Kinesis Data Stream by adjusting the number of shards through a process called resharding . You can increase capacity by splitting shards or decrease it by merging shards. Scaling can be done via: The UpdateShardCount API call The AWS Management Console Automatically, if the stream is in on demand capacity mode
What is the difference between Kinesis Data Streams and Kinesis Data Firehose?
Kinesis Data Streams is designed for real time processing where you build custom consumer applications using the KCL , AWS Lambda , or the SDK. You manage shard capacity and have sub second processing latency. Kinesis Data Firehose is a fully managed, zero administration delivery service that automatically batches, compresses, and loads data into destinations like S3 , Redshift , or OpenSearch . Key differences: Custom processing Data Streams (yes) vs. Firehose (limited transforms only) Latency
Describe the role of partition keys in Kinesis Data Streams.
A partition key is a string specified by the producer when putting a record into a stream. Kinesis uses an MD5 hash function on the partition key to determine which shard receives the record. Records with the same partition key always go to the same shard, which guarantees ordering for that key. Choosing a good partition key is critical: High cardinality keys (e.g., user ID, device ID) distribute data evenly across shards Low cardinality keys (e.g., region with only 3 values) can create hot shar
How does Kinesis Data Analytics process streaming data?
Kinesis Data Analytics allows you to process and analyze streaming data in real time using SQL queries or Apache Flink applications. You define an input source (a Kinesis Data Stream or Firehose), write your transformation logic, and specify an output destination. With SQL based analytics, you can write standard SQL against an in application stream. With Apache Flink , you get a full featured stream processing framework supporting windowed aggregations, pattern detection, and stateful computatio
What is the maximum retention period for data in Kinesis Data Streams?
The default retention period for Kinesis Data Streams is 24 hours . You can extend this up to 7 days (168 hours) using the IncreaseStreamRetentionPeriod API. With long term data retention , the maximum extends to 365 days . Longer retention is useful when consumers need to reprocess data or when downstream systems experience outages. Note that extended retention beyond 24 hours incurs additional charges per shard hour.