Welcome to ExamTopics
ExamTopics Logo
- Expert Verified, Online, Free.

Unlimited Access

Get Unlimited Contributor Access to the all ExamTopics Exams!
Take advantage of PDF Files for 1000+ Exams along with community discussions and pass IT Certification Exams Easily.

Exam Certified Data Engineer Professional topic 1 question 49 discussion

Actual exam question from Databricks's Certified Data Engineer Professional
Question #: 49
Topic #: 1
[All Certified Data Engineer Professional Questions]

A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, using display() calls to confirm code is producing the logically correct results as new transformations are added to an operation. To get a measure of average time to execute, the user is running each cell multiple times interactively.
Which of the following adjustments will get a more accurate measure of how code is likely to perform in production?

  • A. Scala is the only language that can be accurately tested using interactive notebooks; because the best performance is achieved by using Scala code compiled to JARs, all PySpark and Spark SQL logic should be refactored.
  • B. The only way to meaningfully troubleshoot code execution times in development notebooks Is to use production-sized data and production-sized clusters with Run All execution.
  • C. Production code development should only be done using an IDE; executing code against a local build of open source Spark and Delta Lake will provide the most accurate benchmarks for how code will perform in production.
  • D. Calling display() forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results.
  • E. The Jobs UI should be leveraged to occasionally run the notebook as a job and track execution time during incremental code development because Photon can only be enabled on clusters launched for scheduled jobs.
Show Suggested Answer Hide Answer
Suggested Answer: D 🗳️

Comments

Chosen Answer:
This is a voting comment (?) , you can switch to a simple comment.
Switch to a voting comment New
practicioner
3 weeks ago
Selected Answer: B
B and D are correct. The question says "which statements" which suggests us that this is a question with multiple choices
upvoted 1 times
...
HelixAbdu
1 month, 1 week ago
Both D and B are correct. But in real life some times clients dose not accept to gave you there production data to test easily. Also it says in B it is “the only way” ans this is not true for me So i will go with D
upvoted 2 times
...
ffsdfdsfdsfdsfdsf
5 months, 4 weeks ago
Selected Answer: B
These people voting D have no reading comprehension.
upvoted 3 times
...
alexvno
5 months, 4 weeks ago
Selected Answer: B
Close env size volumes as possible so results make sense
upvoted 2 times
...
halleysg
6 months ago
Selected Answer: D
D is correct
upvoted 3 times
...
Curious76
6 months, 1 week ago
Selected Answer: D
I will go with D
upvoted 1 times
...
agreddy
6 months, 2 weeks ago
D is the correct answer A. Scala is the only language accurately tested using notebooks: Not true. Spark SQL and PySpark can be accurately tested in notebooks, and production performance doesn't solely depend on language choice. B. Production-sized data and clusters are necessary: While ideal, it's not always feasible for development. Smaller datasets and clusters can provide indicative insights. C. IDE and local Spark/Delta Lake: Local environments won't replicate production's scale and configuration fully. E. Jobs UI and Photon: True that Photon benefits scheduled jobs, but Jobs UI can track execution times regardless of Photon usage. However, Jobs UI runs might involve additional overhead compared to notebook cells. Option D addresses the specific limitations of using display() for performance measurement
upvoted 4 times
...
guillesd
7 months ago
Selected Answer: B
Both B and D are correct statements. However, D is not an adjustment (see the question), it is just an afirmation which happens to be correct. B, however, is an adjustment, and it will definitely help with profiling.
upvoted 3 times
...
DAN_H
7 months, 1 week ago
Selected Answer: D
As B not talking about how to deal with display() function. We know that way to testing performance for the whole notebook need to avoid using display as it is way to test the code and display the data
upvoted 3 times
...
zzzzx
7 months, 1 week ago
B is correct
upvoted 1 times
...
spaceexplorer
7 months, 2 weeks ago
Selected Answer: D
D is correct
upvoted 1 times
...
divingbell17
8 months, 1 week ago
Selected Answer: B
Calling display() forces a job to trigger - doesnt make sense display is used to display a df/table in tabular format, has nothing to do with a job trigger
upvoted 2 times
guillesd
7 months ago
Actually they mean a spark job. This is true, whenever you call display, spark needs to execute the transformations up to this point to be able to collect the results.
upvoted 2 times
...
...
ervinshang
8 months, 3 weeks ago
D is correct
upvoted 1 times
...
rok21
9 months ago
Selected Answer: B
B is correct
upvoted 1 times
...
sturcu
10 months, 3 weeks ago
Selected Answer: B
Yes D is a True statement. But it does not answer the question. The ask is for "which adjustments will get a more accurate measure of how code is likely to perform in production". Answer D just describes why the chosen approach is not correct. It does not provide a solution.
upvoted 1 times
sturcu
10 months, 3 weeks ago
D would be the answer if it was preceded by: We should avoid calling display() too often or clear the cache before running each cell.
upvoted 2 times
...
...
tkg13
1 year ago
Is it not B?
upvoted 1 times
BrianNguyen95
1 year ago
Option B one of possibility happening. Option D fully meaning
upvoted 3 times
...
...
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

SaveCancel
Loading ...