ExamTopics Logo
Mail Us[email protected]
Menu
Databricks Discussions
exam questions

Exam Certified Data Engineer Professional All Questions

View all questions & answers for the Certified Data Engineer Professional exam
Go to Exam

Exam Certified Data Engineer Professional topic 1 question 95 discussion

Actual exam question from Databricks's Certified Data Engineer Professional
Question #: 95
Topic #: 1
[All Certified Data Engineer Professional Questions]

A task orchestrator has been configured to run two hourly tasks. First, an outside system writes Parquet data to a directory mounted at /mnt/raw_orders/. After this data is written, a Databricks job containing the following code is executed:



Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order, and that the time field indicates when the record was queued in the source system.

If the upstream system is known to occasionally enqueue duplicate entries for a single order hours apart, which statement is correct?

  • A. Duplicate records enqueued more than 2 hours apart may be retained and the orders table may contain duplicate records with the same customer_id and order_id.
  • B. All records will be held in the state store for 2 hours before being deduplicated and committed to the orders table.
  • C. The orders table will contain only the most recent 2 hours of records and no duplicates will be present.
  • D. Duplicate records arriving more than 2 hours apart will be dropped, but duplicates that arrive in the same batch may both be written to the orders table.
  • E. The orders table will not contain duplicates, but records arriving more than 2 hours late will be ignored and missing from the table.
Show Suggested Answer Hide Answer
Suggested Answer: A 🗳️
by alexvno at Dec. 19, 2023, 8:28 a.m.

Comments

Chosen Answer:
This is a voting comment (?). It is better to Upvote an existing comment if you don't have anything to add.
Switch to a voting comment New
Submit Cancel
alexvno
Highly Voted 1 year ago
Selected Answer: A
Only A seems logical
upvoted 8 times
...
arekm
Most Recent 11 hours, 5 minutes ago
Selected Answer: A
A - since the same [customer_id, order_id] record can be emitted twice with the "time" value set apart by more than 2 hours. The target table will then contain duplicates (duplicates from the perspective of the composite key).
upvoted 1 times
...
benni_ale
1 month, 1 week ago
Per me è la cipolla
upvoted 1 times
...
cf56faf
1 month, 3 weeks ago
Selected Answer: E
Seems that E should be the correct answer. As time is the time it was queued in the *source_system*. And withWatermark ignores records that have a "time" more than 2 hours old.
upvoted 1 times
...
Ananth4Sap
2 months, 1 week ago
When watermarking is set to 2 hours, the system will wait for up to 2 hours for late data to arrive. Any data that arrives within this 2-hour window will be considered for processing and deduplication. However, data that arrives later than 2 hours after the event time will be considered too late and will be discarded. This ensures that the state store does not grow indefinitely, but it also means that any records arriving more than 2 hours late will not be included in the orders table.
upvoted 1 times
...
m79590530
2 months, 2 weeks ago
Selected Answer: E
Every Stream micro-batch is executed on all of the new data that arrived after the last run 2 hours ago by the .trigger(once=True) option. Deduplication is done for it based on the combined key fields values but all records older than 2 hours based on the 'time' field will be ignored thanks to the .withWatermark() option/function. So target table will have deduplicated data withOUT the late records arriving more than 2 hours later based on the 2 hours watermark buffer set for the readStream.
upvoted 1 times
arekm
11 hours, 7 minutes ago
Deduplication happens each hour (when the job is run) and contains records from the last 2 hours based on the time column. However, the source system might generate [customer_id, order_id] combination with a newer time value set apart more than 2 hours from the original record. It will be a new record, however from the key perspective a duplicate. Since deduplication happens in 2h windows, the new record will be appended (default for structured streaming) to the target table.
upvoted 1 times
...
...
shaojunni
2 months, 3 weeks ago
Selected Answer: E
Data arrive outside of watermark will be dropped.
upvoted 2 times
...
quaternion
4 months, 3 weeks ago
Selected Answer: E
Watermark("time", "2 hours") --> does'nt let records arriving more than 2 hours late to be written dropDuplicates --> removes duplicate records from the records that are read
upvoted 2 times
...
Isio05
6 months, 4 weeks ago
Selected Answer: A
It's A, rows are deduplicated only in 2hrs window, therefore final table may eventually contain duplicates
upvoted 3 times
...
QuangTrinh
7 months ago
Selected Answer: E
Should be E. Watermarking (withWatermark("time", "2 hours")): This sets a 2-hour watermark on the time column. The watermark specifies the event time threshold for data completeness, meaning that data older than 2 hours will be considered late and may be dropped. Deduplication (dropDuplicates(["customer_id", "order_id"])): This operation removes duplicates based on the composite key (customer_id and order_id). However, it only works within the window defined by the watermark.
upvoted 1 times
arekm
11 hours, 6 minutes ago
Deduplication happens each hour (when the job is run) and contains records from the last 2 hours based on the time column. However, the source system might generate [customer_id, order_id] combination with a newer time value set apart more than 2 hours from the original record. It will be a new record, however from the key perspective a duplicate. Since deduplication happens in 2h windows, the new record will be appended (default for structured streaming) to the target table.
upvoted 1 times
...
...
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

SaveCancel
exam
Someone Bought Contributor Access for:
SY0-701
London, 1 minute ago