Exam Certified Data Engineer Professional topic 1 question 131 discussion

If its exact duplicate of Question #13 then we are missing a good choice of : "E. Ingest all log information into a bronze table; use MERGE INTO to insert, update, or delete the most recent entry for each pk_id into a silver table to recreate the current table state."

Delta CDF is used to read changes from Delta tables, not ingest changes into them and history of chnages are required

Sorry the correct one was E : Ingest all log information into a bronze table; use merge into to insert, update, or delete the most recent entry for each pk_id into a silver table to recreate the current table state

This is the correct answer because it meets the requirements of maintaining a full record of all values that have ever been valid in the source system and recreating the current table state with only the most recent value for each record. The code ingests all log information into a bronze table, which preserves the raw CDC data as it is. Then, it uses merge into to perform an upsert operation on a silver table, which means it will insert new records or update or delete existing records based on the change type and the pk_id columns. This way, the silver table will always reflect the current state of the source table, while the bronze table will keep the history of all changes. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Upsert into a table using merge" section.

Delta Lake's Change Data Feed (CDF) only works on Delta tables, as it is a feature explicitly tied to Delta Lake's architecture. In the scenario described in the question, the source is an external system writing CDC logs to cloud object storage, and there is no mention of the data being stored in Delta tables initially.

The right answer is B because CDF only works on delta tables and here the source is external

There is no right answer. Closest is B only after the CDC logs are ingested to a Bronze table and then use merge into a silver table. Why not D? Because CDF only works on delta tables and not on external CDC logs.

The MERGE INTO statement in Delta Lake is a powerful feature designed to handle Change Data Capture (CDC) data efficiently. This approach meets both the auditing and analytical requirements. CDF is not enabled by default. So these data is not generated by it to handel them.

agree with D

Delta Lake provides built-in change data feed functionality. It captures changes (inserts, updates, deletes) and propagates them to dependent tables. By using Delta Lake, you can maintain historical records and propagate changes efficiently.

Correct Answer: D Delta Lake’s change data feed feature is specifically designed to handle CDC scenarios. It processes data from external systems, tracking all changes (inserts, updates, deletes) and maintaining a detailed history of these changes. This feature allows for keeping a comprehensive log while also ensuring the most recent state is correctly reflected in analytical tables.