Exam AWS Certified Data Analytics - Specialty topic 1 question 72 discussion

Agree with A.

A. https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html "Applications that need strict guarantees should embed a primary key within the record to remove duplicates later when processing."

Ans D Option A doesn't talk about "Obtain exactly once semantics for the entire processing pipeline

D Apache Flink provides a powerful API to transform, aggregate, and enrich events, and supports exactly-once semantics. Apache Flink is therefore a good foundation for the core of your streaming architecture. https://aws.amazon.com/blogs/big-data/streaming-etl-with-apache-flink-and-amazon-kinesis-data-analytics/

A: I passed the test

A - exactly what is requested in the description

Correct answer is A as producer retries can result in duplicates in Kinesis Data Streams and must be handled by the producer by using a unique key for each message. There are two primary reasons why records may be delivered more than one time to your Amazon Kinesis Data Streams application: producer retries and consumer retries. Your application must anticipate and appropriately handle processing individual records multiple times. Options B & C are wrong as they would not handle the exactly-once processing semantics. Option D is wrong as although Apache Flink and Spark Streaming would work, it would need a complete change in the current application.

Selected Answer: A

application should be idempotent. Can be achieved by including a primary key in the record. Hence, A is correct answer

The problem is the question is, the producer may put the records several times. Donell explained this very well. KDA, Flink or Spark can only make sure 'Exactly once' for every record in stream. If the record is duplicated by producer, they will process exactly once for every record in stream. So the answer should be A.

Answer : A

A is right

Answer A. Design the application so it can remove duplicates during processing by embedding a unique ID in each record. Producer Retries Consider a producer that experiences a network-related timeout after it makes a call to PutRecord, but before it can receive an acknowledgement from Amazon Kinesis Data Streams. The producer cannot be sure if the record was delivered to Kinesis Data Streams. Assuming that every record is important to the application, the producer would have been written to retry the call with the same data. If both PutRecord calls on that same data were successfully committed to Kinesis Data Streams, then there will be two Kinesis Data Streams records. Although the two records have identical data, they also have unique sequence numbers. Applications that need strict guarantees should embed a primary key within the record to remove duplicates later when processing. Note that the number of duplicates due to producer retries is usually low compared to the number of duplicates due to consumer retries. Reference: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html

ANSWER:D KDS and KDF has 'exactly once' semantics .Option A is a fail safe mechanism when there is a choppy network while sending data to KDS with PutRecord. Apache Flink and Apache Spark both guarantees 'Exactly once' semantics and which is what is the requirement as per the question .

A is the right answer

A was a good choice until i search EMR Flink...: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html Apache Flink is a streaming dataflow engine that you can use to run real-time stream processing on high-throughput data sources. Flink supports event time semantics for out-of-order events, exactly-once semantics, backpressure control, and APIs optimized for writing both streaming and batch applications. Can be connected to KDS. So i will pick up D

Link provided supports A as the answer