In the previous subchapter, we built a data lake to analyze stored data. But much data arrives in real time, continuously: clicks on a website as users browse, sensor readings every second, transactions as they happen... How do you capture and process that continuous stream of data in the moment, without losing anything? That's what Amazon Kinesis is for, AWS's real-time (streaming) data service. We'll look at its two main components: Kinesis Data Streams and Kinesis Data Firehose.

Some data doesn't arrive "every now and then" in files, but as a continuous stream that never stops:

Processing this presents challenges: it arrives constantly and in large volume, you can't lose data, and often you want to react instantly (detect fraud while it's happening, not the next day). You need something capable of capturing and moving that continuous stream reliably and at scale.

Real-time data processing (streaming) consists of capturing and processing data as it is generated, continuously, instead of waiting to have a large batch and then processing it (which would be "batch" processing).

Analogy: the difference is like a dam and a river. Batch processing is like a dam: you accumulate water and release it all at once every so often. Streaming is like a river that flows non-stop: the water (the data) passes by continuously and you use it as it flows. Kinesis is the "channel" prepared to manage that river of data without overflowing or losing anything.

Amazon Kinesis is the AWS family of services to capture, process, and analyze real-time (streaming) data at scale. It allows you to reliably ingest huge streams of continuous data. It has several components; we'll look at the two main ones.

Kinesis Data Streams captures a continuous stream of real-time data and makes it available for your applications to process instantly. Data enters the "stream" and your consumers (for example, Lambdas—remember that Kinesis can be a Lambda trigger, subchapter 14.2) read and process it in the moment.

• Use case: when you need to react in real time to data (detect fraud instantly, alert on a sensor anomaly, update a live dashboard).
• Key: data is available to be processed immediately, with minimal latency.

Analogy: Kinesis Data Streams is like a live conveyor belt where data passes by, and your workers (applications) pick it up and process it as it passes, without waiting. Ideal when every piece of data matters now.

Kinesis Data Firehose focuses on something different: collecting a stream of data and automatically delivering it to a storage or analytics destination (like S3—your data lake from subchapter 29.1—, Redshift, etc.), without you having to program anything to manage it. It's the simplest way to load streaming data into a place to store or analyze it.

• Use case: when you want to send a stream of data to your data lake (S3) or another destination automatically and easily, without needing to process it instantly.
• Key: it's fully managed and very easy: you configure the source and destination, and Firehose takes care of moving the data (it can even transform or batch it along the way).

Analogy: if Data Streams is a live conveyor belt, Firehose (whose name means "fire hose") is like a hose that channels the stream of data directly into the tank (S3). You don't worry about managing the belt or the workers: you just connect the hose to the tank and the data flows there automatically.

💡 Rule of thumb: if you need to react instantly to the data, use Data Streams. If you just want to send streaming data to a place (like your data lake in S3) easily and automatically, use Firehose. They're often used together: Streams to react live and Firehose to archive the same stream in S3.

Kinesis is often the entry point for real-time data into the data lake from subchapter 29.1:

Thus, data that arrives continuously ends up in your data lake, ready to be analyzed along with the rest. Streaming (Kinesis) and data lake (S3+Glue+Athena) combine into a complete data platform.

Real-world example: an online gaming platform wants to analyze player behavior in real time and also store it for later analysis. They use Kinesis Data Streams to capture every player action (millions per minute) and process them instantly with Lambdas that, for example, detect cheating or adjust difficulty live. At the same time, they use Kinesis Data Firehose to dump that same stream of events into S3 (their data lake), where they later analyze it with Athena to understand long-term trends. Streaming to react now, data lake to understand the historical: the best of both worlds.

• Much data arrives in real time, continuously (clicks, sensors, transactions); processing it requires capturing that continuous stream without losing anything, often to react instantly.
• Streaming processing handles data as it is generated (like a flowing river), versus batch processing (like a dam that accumulates and releases).
• Amazon Kinesis captures, processes, and analyzes real-time data at scale. Two main components:
  • Kinesis Data Streams: captures a live stream to process it instantly (react in real time: fraud, alerts). Like a live conveyor belt.
  • Kinesis Data Firehose: collects a stream and automatically delivers it to a destination (S3, Redshift...), fully managed. Like a hose to the tank.
• 💡 Data Streams to react instantly; Firehose to send data to a destination easily. They're often used together.
• Kinesis is the entry point for real-time data into the data lake (S3), combining streaming and historical.

In the next subchapter, we'll look at the other major pillar of analytics: the data warehouse optimized for large-scale queries, Redshift.