Exam AWS Certified Solutions Architect - Associate SAA-C03 topic 1 question 214 discussion

It looks like AWS Glue allows fully managed CSV to Parquet conversion jobs: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/three-aws-glue-etl-job-types-for-converting-data-to-apache-parquet.html

AWS Glue is a fully managed ETL service that simplifies the process of preparing and transforming data for analytics. Using AWS Glue requires minimal development effort compared to the other options. Option A requires more development effort as it involves writing a Spark application to transform the data. It also introduces additional infrastructure management with the EMR cluster. Option C requires writing and managing custom Bash scripts for data transformation. It requires more manual effort and does not provide the built-in capabilities of AWS Glue for data transformation. Option D requires developing and managing a custom Lambda for data transformation. While Lambda can handle the transformation, it requires more effort compared to AWS Glue, which is specifically designed for ETL operations. Therefore, option B provides the easiest and least development effort by leveraging AWS Glue's capabilities for data discovery, transformation, and output to the transformed data bucket.

AWS Glue offers a serverless, automated, and cost-effective solution with minimal development and operational effort, making it the best choice for this use case. Why not the other options? A. Amazon EMR cluster with Apache Spark: While EMR and Spark can handle this task, it requires more setup, maintenance, and development effort compared to AWS Glue. Managing the cluster introduces operational overhead. C. AWS Batch with Bash job definition: Using AWS Batch for this would require creating custom Bash scripts for the transformation and managing jobs. This introduces more complexity and development effort than AWS Glue D. AWS Lambda with S3 event notifications: Lambda is suitable for lightweight, real-time processing. However, converting hundreds of .csv files into Parquet format could exceed Lambda's execution time and resource limits, leading to scalability challenges.

AWS Glue and parquet go hand in hand

i will go with answer B cause: You can use AWS Glue to write ETL jobs in a Python shell environment. You can also create both batch and streaming ETL jobs by using Python (PySpark) or Scala in a managed Apache Spark environment. Apache Parquet is built to support efficient compression and encoding schemes. It can speed up your analytics workloads because it stores data in a columnar fashion. Converting data to Parquet can save you storage space, cost, and time in the longer run

D I think people are forgetting the question says "Low Overhead".

Parquet format ========> Amazon Glue

B. Create an AWS Glue crawler to discover the data. Create an AWS Glue extract, transform, and load (ETL) job to transform the data. Specify the transformed data bucket in the output step.

Least development effort means lambda. Glue also works but more overhead and cost. A simple lambda like this https://github.com/ayshaysha/aws-csv-to-parquet-converter/blob/main/csv-parquet-converter.py can be used to convert as soon as you see files in s3 bucket.

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/three-aws-glue-etl-job-types-for-converting-data-to-apache-parquet.html

S3 provides a single control to automatically encrypt all new objects in a bucket with SSE-S3 or SSE-KMS. Unfortunately, these controls only affect new objects. If your bucket already contains millions of unencrypted objects, then turning on automatic encryption does not make your bucket secure as the unencrypted objects remain. For S3 buckets with a large number of objects (millions to billions), use Amazon S3 Inventory to get a list of the unencrypted objects, and Amazon S3 Batch Operations to encrypt the large number of old, unencrypted files.

ETL = Glue

B is the correct answer

AWS Glue Crawler is for ETL

The correct answer is B

B is the answer

ıt should be b