Exam AWS Certified Data Engineer - Associate DEA-C01 topic 1 question 19 discussion

AWS Lambda can be effectively used to trigger Athena queries. By using the start_query_execution API from the Athena Boto3 client, you can programmatically start Athena queries. Lambda functions are cost-effective as they charge based on the compute time used, and there's no charge when the code is not running. However, Lambda has a maximum execution timeout of 15 minutes, which means it's not suitable for long-running operations but can be used to trigger or start queries. AWS Step Functions can orchestrate multiple AWS services in workflows. By using a Wait state, the workflow can periodically check the status of the Athena query, and proceed to the next step once the query is complete. This approach is more scalable and reliable compared to continuously running a Lambda function, as Step Functions can handle long-running processes better and can maintain the state of each step in the workflow.

B - because https://docs.aws.amazon.com/step-functions/latest/dg/sample-athena-query.html E - because https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-spark-jobs-with-amazon-mwaa-and-data-validation-using-amazon-athena/

After real-world testing, A is a valid answer. This is because the Lambda only sends the API request to Athena, which runs the query. Even if the Lambda times out, the query result is still stored in the designated S3 bucket.

Why? B (Step Functions): Step Functions are ideal for orchestrating long-running workflows, including polling the Athena query status and invoking the next query when ready. A (Lambda): Lambda is used to programmatically trigger Athena queries within Step Functions, despite its 15-minute limitation, because Step Functions can manage the long runtime using Wait states. Why Not C, D, or E? C and D involve Glue, which is better suited for ETL jobs than orchestration, making them less efficient and cost-effective. E (Amazon MWAA) introduces unnecessary cost and complexity for a straightforward workflow.

BC for me A - lambda function will stop at 900s, so it will stop before query finishes(more than 15mins) E - Airflow is way more complex and expensive than step function

AB - Because of the Lambda timeout CE—is correct. The query will be executed by a glue job, which will be orchestrated by Airflow. The job will be scheduled using AWS Batch.

Not E because "You should use Step Functions if you prioritize cost and performance" https://aws.amazon.com/managed-workflows-for-apache-airflow/faqs/ And also the fact that the queries take longer than 15 min can be handled with step functions, therefore AB

A.Why it's correct: AWS Lambda is a cost-effective, serverless option for invoking Athena queries using the Boto3 API. Lambda charges are based on execution time and memory usage, making it an efficient solution for periodic query execution. B. Why it's correct: Step Functions provide a serverless orchestration option with a pay-per-use pricing model. Adding a Wait state prevents excessive API calls and ensures queries are executed in sequence, making it a cost-effective and scalable solution. Why the other options are less optimal: -- E. Use Amazon Managed Workflows for Apache Airflow (MWAA): MWAA is powerful for complex workflows, but its pricing includes environment uptime costs, which can be higher than Lambda and Step Functions for simple tasks like orchestrating Athena queries. By choosing A and B, you balance cost-effectiveness and simplicity for orchestrating daily Athena queries.

Selected Answer: BD Lambda maximum timeout is 15 minutes. So the query takes more than Lambda could manage. So you cannot use lambda. Use Step-Function (answer B) or glue python (answer D) Airflow is more expensive than Glue/Step-Functions, so E is discarted also.

It should be AB

Since the Athena API supports async/await, users are able to separate the steps into trigger queries and get results after 15 minutes.

tricky, A is valid. Still, cost effective: B no one doubt on it. then why E? MWAA offers a managed Apache Airflow environment for orchestrating complex workflows. It can handle long-running tasks like Athena queries efficiently. Batch Processing: Leveraging AWS Batch within the Airflow workflow allows for distributed and scalable execution of the Athena queries, improving overall processing efficiency.

my opinian

I would prefer AB

Lambda for kick start Athena Step Functions for orchestration

Option C and D involve using an AWS Glue Python shell script to run a sleep timer and periodically check whether the current Athena query has finished running. While this approach might seem cost-effective in terms of using AWS Glue, it's not the most efficient way to manage the execution of Athena queries. AWS Glue is primarily designed for ETL (Extract, Transform, Load) tasks rather than orchestrating long-running query execution. Therefore, while both options B, C and D could technically work, they might not be the most cost-effective or efficient solutions for orchestrating long-running Athena queries. Instead, options A and E would likely be more cost-effective and suitable for this scenario.

Option B, utilizing AWS Step Functions, can be a cost-effective solution for orchestrating the execution of Athena queries, but it might not be the most cost-effective in this scenario because Step Functions are billed based on state transitions and the duration of state execution. Since each query can run for more than 15 minutes, using Step Functions to wait and periodically check the status of the queries could potentially result in higher costs, especially if the queries frequently take a long time to complete.