Exam AWS Certified Big Data - Specialty topic 1 question 63 discussion

Not A - Because Spark Streams or any Spark component for that matter cannot read directly from KFH. Also not real-time solution. Not C - Not a real-time solution although doable if real-time is not an issue. Not D- It doesn't make sense to write the original row data one at a time to S3 when its possible to configure Kinesis stream destination as S3 and split data into multiple files organized by date as prefix. B- Is the right answer. Spark Streams can read directly from Kinesis streams and so can a Lambda function which will then insert each record into KFH to be delivered to S3.

A is wrong - Spark Streaming can only read from Kinesis Data Streams B doesn't make sense - Kinesis Data Firehose has direct integration with Kinesis Data Streams https://docs.aws.amazon.com/firehose/latest/dev/writing-with-kinesis-streams.html C doesn't make sense - Lambda to integrate Firehose with Data Streams. I don't belive people design such crazy things D is correct and looks best compared to the others

D. Real-time message ruled out A and C. For B, my concern is that streaming data in parallel to both Lambda and Spark, will that reduce the performance of KDS, or requires more shards? Hence, I pick D.

Option D. "The data engineer needs to fulfill a corporate mandate to keep ALL raw messages as they are received as a top priority" It does mention anything regarding batching, so its ok to write single rows to S3

It has to be B.

It could have been B, but we don't need a lambda to push messages, send the stream straight to FH. Because of this, going with D to keep the solution simple.

It could have been B, but we don't need a lambda to push messages, send the stream straight to FH. Because of this, going with D to keep the solution simple.

my selection B

I vote for D. It looks simple and working solution

Answer C Spark Streaming can't pull messages from Firehose or Streams, so options A, B and D invalid. https://docs.aws.amazon.com/solutions/latest/real-time-analytics-spark-streaming/architecture.html

, Apache Spark can be over-burdened with file operations if it is processing a large number of small files versus fewer larger files. Each of these files has its own overhead of a few milliseconds for opening, reading metadata information, and closing. This overhead of file operations on these large numbers of files results in slow processing. This blog post shows how to use Amazon Kinesis Data Firehose to merge many small messages into larger messages for delivery to Amazon S3. This results in faster processing with Amazon EMR running Spark. Option C.

Nobody has thoughts on A? can spark stream pull out message from kFH directly. A lambda function is necessary for pulling data from KFH to feed spark stream?

I think D is correct. B looks overkill.

In context, this case doesn't need managed service (FH + fee) in gathering & processing. Cz FH needs buffering time for efficient bulk transfer.. For real time processing.. (not real-time gathering..) it doesn't need FH yet (in gathering & processing.) But in back up to S3.. FH is better.

I choose D, kinesis stream can be used to first store the raw data in s3 and then analyze the data in real time.

funny, nobody bore to search KCL checkpoint table

FH cannot be an answer it is not a realtime. B is correct