Exam AWS Certified Data Engineer - Associate DEA-C01 topic 1 question 27 discussion

The answer is C. It can provide near real-time insight analysis. Refer the article from AWS - https://aws.amazon.com/blogs/big-data/real-time-analytics-with-amazon-redshift-streaming-ingestion/

Key word here is near real-time. If it's involve S3 and COPY, it's not gonna be near real-time

Creating an external schema and using materialized views directly on top of Kinesis Data Streams is also not an ideal choice because this approach can add complexity and doesn't leverage fully managed solutions like Kinesis Data Firehose. The manual management of data refresh rates adds operational overhead.

Refer to the article from AWS - https://aws.amazon.com/blogs/big-data/real-time-analytics-with-amazon-redshift-streaming-ingestion/

option D provides a streamlined, efficient, and low-overhead approach to achieving real-time analytics with the specified technologies.

A: Kinesis Data Streams to stage data in Amazon S3. not really easy, B: sql directly to Kinesis Data Streams : functionality not exist C : external schema from redshift to Kinesis Data Streams : functionality not exist D : near real-time = Kinesis Data Firehose

https://aws.amazon.com/blogs/big-data/real-time-analytics-with-amazon-redshift-streaming-ingestion/

https://aws.amazon.com/blogs/big-data/real-time-analytics-with-amazon-redshift-streaming-ingestion/

A for me C - Redshift does not natively support direct mapping to Kinesis Data Streams. Some extra configs are needed. D - There will be a 60s latency when using Firehose, so it's "Near" real time not real time.

Redshift does not natively support direct mapping to Kinesis Data Streams. Materialized views cannot directly query streaming data from Kinesis.

See https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion-getting-started.html

D could be the most standard way to handle this case. How to use C to implement it is questionable for me.

Amazon Redshift can automatically refresh materialized views with up-to-date data from its base tables when materialized views are created with or altered to have the autorefresh option. Amazon Redshift autorefreshes materialized views as soon as possible after base tables changes. https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh.html

Firehose is Near-Real time, you can set your buffer size and stream to either Redshift or S3 directly. Since Redshift is not in the option, use s3...

Kinesis Data Streams , option D using Kinesis Data Firehose is a fully managed service that automatically handles the ingestion of data

Here’s why D is the best choice: Kinesis Data Firehose is a fully managed service that automatically handles the ingestion of data from Kinesis Data Streams and stages it in S3, which significantly reduces operational overhead compared to managing custom data ingestion pipelines. S3 as a staging area: Using Amazon S3 as a staging location allows for flexible data management, high durability, and direct loading into Redshift without needing to manage complex buffering or data handling processes. COPY command: The COPY command in Amazon Redshift is highly optimized for loading large datasets efficiently, making it a common and effective method to load bulk data from S3 into Redshift for near real-time analysis. Firehose to Redshift: Firehose can automatically buffer, batch, and transform data before loading it into Redshift, reducing manual intervention and ensuring data is readily available for real-time analytics.

Option C has an issue: Redshift does not natively support direct querying or mapping of Kinesis Data Streams. D is the only correct option.