Exam Certified Data Engineer Professional topic 1 question 8 discussion

B. Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table. Explanation: In the provided code, the .dropDuplicates(["customer_id","order_id"]) operation is performed on the data loaded from the Parquet files. This operation ensures that only unique records, based on the composite key of "customer_id" and "order_id," are retained in the DataFrame before writing to the "orders" table. However, this operation does not consider duplicates that may already exist in the "orders" table. It only filters duplicates from the current batch of data. If there are duplicates in the "orders" table from previous batches, they will remain in the table. So, newly written records will not have duplicates within the batch being written, but duplicates from previous batches may still exist in the target table.

No duplicates in the current batch - that is obvious. The duplicates may happen since the source occasionally produces duplicates hours apart. This means that one record can be generated by the source and processed on day 1, the duplicate on day 2. Since there is no logic checking if the corresponding record exists in the target - you get the duplicates there given we use append mode.

yeah B is the correct answer cause in the current code it will look for duplicates in the currentDF based on composite keys and not for the duplicates which are already in the target table. if we want to insert for the rows which are not there in target table then we can make use of Merge Into statement of databricks.

Append method does not take in consideration any key in the target table, it simply add all rows of the input table to the target table.

Yes it should be B

B. Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table. Using merge this problem would not happen

B is the right answer. The above code only remove duplicates from the batch that is processed, no logic is applied to already saved records.

B is correct

Answer B

B is correct

correct B

Correct

Correct. B

Correct