Exam Certified Data Engineer Associate topic 1 question 34 discussion

Auto Loader is a feature in Databricks that is specifically designed to efficiently ingest new files incrementally from cloud storage directories. It can handle the scenario where files accumulate in a shared directory, and you want to ingest only the new files since the previous run without reprocessing the entire dataset. Auto Loader uses file notification services to track new files that appear in the directory, enabling incremental processing of those files as they are added. It also supports schema inference and automatically manages the state of the files that have been processed, so you don't need to manually track which files have been ingested.

E. Auto Loader Auto Loader is designed to incrementally ingest new data files as they appear in a directory, making it ideal for scenarios where files accumulate and need to be ingested without reprocessing previously ingested files. It automatically tracks which files have already been processed, ensuring that only new files are ingested with each pipeline run.

E is correct

E is correct

the data engineer needs to identify which files are new since the previous run. This seems to be an analysis effort. If that is the case, and I might be wrong, then DB SQL is the correct answer.

Autoloader can help if you want to ingest data incrementally.

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. https://docs.databricks.com/en/ingestion/auto-loader/index.html

option E

E is the correct answer.