Exam AWS Certified Data Analytics - Specialty topic 1 question 152 discussion

I would pick D over B. 1) The question is asking for the LEAST latency. It doesn't matter if we will have high costs, setup or complexity. 2) What we will be the benefit of having 1 lambda (aka 1 consumer) with EFO? The EFO exists exatly to be possible to have multiple consumers without one affecting each other. 3) The parallelism factor will increase throughput, but we will still have 1 consumer getting data in parallel from the same shard, it is not like we will have multiple lambda executions writting to different HTTP endpoints. 4) With 1 lambda, we will need to write to the HTTP sequentially, like http1.write, http2.write, etc. 5) To finalize, I think that EFO with one consumer is irrelevant, because you will get 2MB/s even without EFO. I would go for D all the way. Please, if there is people going around this, just share your thoughts.

Enhanced fan-out is an Amazon Kinesis Data Streams feature that enables consumers to receive records from a data stream with dedicated throughput of up to 2 MB of data per second per shard. When using Lambda as an enhanced fan-out consumer, you can use the Event Source Mapping Parallelization Factor to have one Lambda pull from one shard concurrently with up to 10 parallel invocations per consumer. A consumer that uses enhanced fan-out doesn't have to contend with other consumers that are receiving data from the stream. our data cap is up to 500kb, meaning we have the ability to pull at least 4 messages per shard per second. The 10 parralel invocations would mean that we can call up to 10 shards in parallel and thus consume a minimum of 40 messages at the maximum message size as defined in the question. Thus, one lambda enhanced fanout consumer should be enough to route to 20 destinations concurrently and 20 lambdas would be waaay too many, hence B

Bing said answer is D :)

I've heard that the enhanced fan out feature only works for http/2.

I am confused with lambda payload limit that is 256kb thhat is greater than message size

In my opinion this two requirements make me select D "The performance of a single destination HTTP endpoint should not impact the performance of the delivery for other destinations." This requirement underscores the need for a design that isolates the message delivery paths to different endpoints. If a single Lambda function were responsible for all destinations, any slow response or failure at one endpoint could cause delays or retries that would affect the entire function, impacting the delivery performance for other endpoints. This is undesirable because the behavior or performance of one endpoint should not influence the others and Message Ordering, Each function maintains the order for its own endpoint, reducing complexity and the risk of out-of-order delivery that might arise when one function handles multiple endpoints.

B does not address the requirement: The performance of a single destination HTTP endpoint should not impact the performance of the delivery for other destinations. As one single lambda would send http put requests one after the other. 20 independent destinations = 20 consumers so I say D

B: I passed the test

The key requirement is "The performance of a single destination HTTP endpoint should not impact the performance of the delivery for other destinations."

Answer: D I think it's D because https://aws.amazon.com/about-aws/whats-new/2018/11/aws-lambda-supports-kinesis-data-streams-enhanced-fan-out-and-http2/ enhanced-fan-out with one lambda doesn't makes sense and logically incorrect statement.

KDS for near real time ? for low size .. KDS is for real time and min 10GB of data streaming ?? considering its near real time - shouldnt the answer be firehose ?

The question is: does 20 HTTP Endpoint destinations considered as one consumer or 20 consumers? Fan-out lambda can support up to 5 consumers so 20 HTTPS endpoints consider 20 consumers, you will need to establish at least 4 Fan-out lambda functions, which rule out B. However, these 20 HTTP endpoints should consider as one consumer, they are going to the same place and this place has 20 endpoints. So one Fan-out should be good enough instead of 20. Link: https://aws.amazon.com/blogs/compute/increasing-real-time-stream-processing-performance-with-amazon-kinesis-data-streams-enhanced-fan-out-and-aws-lambda/

Ans- B Lambda enhanced fan-out consumers Enhanced fan-out consumers can increase the per shard read consumption throughput through event-based consumption, reduce latency with parallelization, and support error handling. The enhanced fan-out consumer increases the read capacity of consumers from a shared 2 MB per second, to a dedicated 2 MB per second for each consumer. When using Lambda as an enhanced fan-out consumer, you can use the Event Source Mapping Parallelization Factor to have one Lambda pull from one shard concurrently with up to 10 parallel invocations per consumer. Each parallelized invocation contains messages with the same partition key (HotelCode) and maintains order. The invocations complete each message before processing with the next parallel invocation. Figure 8: Lambda fan-out consumers with parallel invocations, maintaining order

Answer B - https://aws.amazon.com/blogs/compute/increasing-real-time-stream-processing-performance-with-amazon-kinesis-data-streams-enhanced-fan-out-and-aws-lambda/ refer "Enhanced fan-out with Lambda functions"

I have concerns regarding B, 1 lamda function processing data and sending to 20 HTTP endpoints seems bit complicated and fan out for 1 lambda seems none necessary I think. From the other side in D haveing 20 lambdas that send to each point puts unnecesary load on kinesis, I think good case would be to have severl lambdas , for example 5 , each for 4 endpoints

Answer-B

Why would you want to use enhanced fan out consumer in answer B? there is only a single consumer and there are no benefits in using this feature. In answer D, using 20 lambda functions is a waste, but with enhanced consumer you can be sure that you will have not throttling problems.