Welcome to ExamTopics
ExamTopics Logo
- Expert Verified, Online, Free.
Mail Us[email protected]
Menu
Databricks Discussions
exam questions

Exam Certified Data Engineer Professional All Questions

View all questions & answers for the Certified Data Engineer Professional exam
Go to Exam

Exam Certified Data Engineer Professional topic 1 question 76 discussion

Actual exam question from Databricks's Certified Data Engineer Professional
Question #: 76
Topic #: 1
[All Certified Data Engineer Professional Questions]

A data pipeline uses Structured Streaming to ingest data from Apache Kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka-generated timestamp, key, and value. Three months after the pipeline is deployed, the data engineering team has noticed some latency issues during certain times of the day.

A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recorded by Apache Spark) as well as the Kafka topic and partition. The team plans to use these additional metadata fields to diagnose the transient processing delays.

Which limitation will the team face while diagnosing this problem?

  • A. New fields will not be computed for historic records.
  • B. Spark cannot capture the topic and partition fields from a Kafka source.
  • C. New fields cannot be added to a production Delta table.
  • D. Updating the table schema will invalidate the Delta transaction log metadata.
  • E. Updating the table schema requires a default value provided for each field added.
Show Suggested Answer Hide Answer
Suggested Answer: A 🗳️
by dmov at Dec. 29, 2023, 4:55 p.m.

Comments

Chosen Answer:
This is a voting comment (?). It is better to Upvote an existing comment if you don't have anything to add.
Switch to a voting comment New
Submit Cancel
dmov
Highly Voted 11 months ago
Selected Answer: A
Looks like A to me. Does anyone think otherwise?
upvoted 7 times
...
vctrhugo
Most Recent 9 months, 3 weeks ago
Selected Answer: A
When the schema of a Delta table is updated to include new fields, these fields will only be populated for new records ingested after the schema update. The new fields will not be retroactively computed for historic records already stored in the Delta table. Therefore, the additional metadata fields (current timestamp, Kafka topic, and partition) will not exist in the historic data, limiting the scope of the diagnosis to new data ingested after the schema update.
upvoted 4 times
...

Get IT Certification

Unlock free, top-quality video courses on ExamTopics with a simple registration. Elevate your learning journey with our expertly curated content. Register now to access a diverse range of educational resources designed for your success. Start learning today with ExamTopics!

Start Learning for free
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

SaveCancel