Welcome to ExamTopics
ExamTopics Logo
- Expert Verified, Online, Free.
Mail Us[email protected]
Menu
Databricks Discussions
exam questions

Exam Certified Data Engineer Professional All Questions

View all questions & answers for the Certified Data Engineer Professional exam
Go to Exam

Exam Certified Data Engineer Professional topic 1 question 15 discussion

Actual exam question from Databricks's Certified Data Engineer Professional
Question #: 15
Topic #: 1
[All Certified Data Engineer Professional Questions]

A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.
Which approach would simplify the identification of these changed records?

  • A. Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
  • B. Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.
  • C. Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
  • D. Modify the overwrite logic to include a field populated by calling spark.sql.functions.current_timestamp() as data are being written; use this field to identify records written on a particular date.
  • E. Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.
Show Suggested Answer Hide Answer
Suggested Answer: E 🗳️
by Eertyy at Aug. 27, 2023, 6:26 p.m.

Comments

Chosen Answer:
This is a voting comment (?) , you can switch to a simple comment.
Switch to a voting comment New
Submit Cancel
Eertyy
Highly Voted 1 year, 3 months ago
E is right answer
upvoted 6 times
...
Melik3
Most Recent 3 months, 2 weeks ago
I don't understand why E is correct. With E we are updating only data needed but we are then doing prediction on the whole table which means that we are doing again predictions on not changing records which is not efficient
upvoted 1 times
benni_ale
1 month ago
"write logic to make predictions on the CHANGED records identified by the change data feed". the only thing partially wrong about E is that it has never been stated that the table has a change data feed enables.
upvoted 1 times
...
...
leopedroso1
9 months, 1 week ago
E is the correct one. By removing overwrite with merge, this will lead to an UPSERT causing updating only the data needed ( When Matched Upate + When not mached insert clauses). Then, with the CDC the capability of identifying is also satisfied.
upvoted 2 times
...
AziLa
10 months, 1 week ago
correct ans is E
upvoted 1 times
...
kz_data
10 months, 2 weeks ago
Selected Answer: E
E is correct
upvoted 1 times
...
sturcu
1 year, 1 month ago
Selected Answer: E
E is Correct
upvoted 3 times
...

Get IT Certification

Unlock free, top-quality video courses on ExamTopics with a simple registration. Elevate your learning journey with our expertly curated content. Register now to access a diverse range of educational resources designed for your success. Start learning today with ExamTopics!

Start Learning for free
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

SaveCancel