Exam Certified Data Engineer Professional topic 1 question 8 discussion

B. Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table. Explanation: In the provided code, the .dropDuplicates(["customer_id","order_id"]) operation is performed on the data loaded from the Parquet files. This operation ensures that only unique records, based on the composite key of "customer_id" and "order_id," are retained in the DataFrame before writing to the "orders" table. However, this operation does not consider duplicates that may already exist in the "orders" table. It only filters duplicates from the current batch of data. If there are duplicates in the "orders" table from previous batches, they will remain in the table. So, newly written records will not have duplicates within the batch being written, but duplicates from previous batches may still exist in the target table.

Append method does not take in consideration any key in the target table, it simply add all rows of the input table to the target table.

Yes it should be B

B. Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table. Using merge this problem would not happen

B is the right answer. The above code only remove duplicates from the batch that is processed, no logic is applied to already saved records.

B is correct

Answer B

B is correct

correct B

Correct

Correct. B

Correct