Exam AWS Certified Solutions Architect - Associate SAA-C03 topic 1 question 603 discussion

AWS Step Functions allows you to orchestrate and scale distributed processing using the Map state. The Map state can process items in a large dataset in parallel by distributing the work across multiple resources. Using the Map state in Distributed mode will automatically handle the parallel processing and scaling. Step Functions will add more workers to process the data as needed. Step Functions is serverless so there are no servers to manage. It will scale up and down automatically based on demand.

https://docs.aws.amazon.com/step-functions/latest/dg/use-dist-map-orchestrate-large-scale-parallel-workloads.html

Using step functions will be overwill from my point of view. I would use Glue, it’s serverless and purposely designed for such use case

Simple - user Lambda / Complex - user Step Functions

A Map in Inline mode can support concurrency of 40 parallel branches and execution history limits of 25,000 events or approximately 6,500 state transitions in a workflow. With the Distributed mode, you can run at concurrency of up to 10,000 parallel branches. So I believe if it has to process thousands of items in parallel Distributed Mode is more appropriate

https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/ https://docs.aws.amazon.com/step-functions/latest/dg/sample-dist-map-s3data-process.html

The Distributed Map has been optimized for Amazon S3.,helping you more easily iterate over objects in an S3 bucket. With the Distributed mode, you can run at concurrency of up to 10,000 parallel branches. https://aws.amazon.com/step-functions/faqs/#:~:text=A%20Map%20in%20Inline%20mode,up%20to%2010%2C000%20parallel%20branches.

https://docs.aws.amazon.com/step-functions/latest/dg/concepts-orchestrate-large-scale-parallel-workloads.html

With Step Functions, you can orchestrate large-scale parallel workloads to perform tasks, such as on-demand processing of semi-structured data. These parallel workloads let you concurrently process large-scale data sources stored in Amazon S3. https://docs.aws.amazon.com/step-functions/latest/dg/concepts-orchestrate-large-scale-parallel-workloads.html

Large Scale + Parallel = Distributed Step Function https://docs.aws.amazon.com/step-functions/latest/dg/concepts-inline-vs-distributed-map.html