Exam Certified Data Engineer Professional topic 1 question 49 discussion

Both B and D are correct statements. However, D is not an adjustment (see the question), it is just an afirmation which happens to be correct. B, however, is an adjustment, and it will definitely help with profiling.

Transformations like select(), filter(), and join() are lazy — they build a logical execution plan but do not trigger actual computation. Actions like display(), collect(), count(), or write() are what trigger execution of the plan. When a user repeatedly executes the same code, Spark may cache intermediate results or re-optimize the plan, which makes performance inconsistent and not representative of real production runs. Therefore, measuring execution time by re-running code interactively in a notebook (especially using display()) is not a reliable performance benchmark.

Lazy Evaluation in Spark: Many transformations in Spark only modify the logical query plan until an action (like display(), collect(), or write()) is triggered.

suretly B

Explanation: Using display() in Databricks forces a job to trigger and display the output, which can lead to an inaccurate measure of performance when benchmarking code. This is because display() triggers the job and materializes the result, which does not accurately reflect how the code will perform in production when the job is run without the display output. Additionally, repeated execution of the same logic (with caching) may not give you meaningful performance results since the results are cached in memory and not representative of fresh computations, as they would occur in a production environment. To get a more accurate measure of execution time, the user should focus on using appropriate job execution techniques, such as running the notebook with "Run All" and avoiding reliance on display() calls, which are not representative of how the pipeline would behave in production.

Answer B, see discussion under benni_ale.

Answer D. While Option D doesn't directly provide an alternative adjustment, it points out a critical issue in the way interactive notebooks might give misleading results. It would be advisable to avoid using display() as a benchmark for performance in production-like environments.

Without much thought, I would vote for option B, but since it says 'the ONLY,' it makes me hesitate. While option D only points out the issues with the data engineer's executions, it doesn’t really provide the adjustments that need to be made. On the other hand, option B at least gives you a way to simulate production behavior. I’ll vote for B, but as I said, the word 'only' makes me doubt, because it’s not the only way.

Answer: D. Explanation: Lazy Evaluation: Spark employs lazy evaluation, meaning transformations are not executed until an action (e.g., display(), count(), collect()) is called. Using display() triggers the execution of the transformations up to that point. Caching Effects: Repeatedly executing the same cell can lead to caching, where Spark stores intermediate results. This caching can cause subsequent executions to be faster, not reflecting the true performance of the code. Why not B: Production-Sized Data and Clusters: While using production-sized data and clusters (as mentioned in option B) can provide insights into performance, it's not the only way to troubleshoot execution times. Proper testing can often be conducted on smaller datasets and clusters, especially during the development phase.

B and D are correct. The question says "which statements" which suggests us that this is a question with multiple choices

Both D and B are correct. But in real life some times clients dose not accept to gave you there production data to test easily. Also it says in B it is “the only way” ans this is not true for me So i will go with D

These people voting D have no reading comprehension.

Close env size volumes as possible so results make sense

D is correct

I will go with D

D is the correct answer A. Scala is the only language accurately tested using notebooks: Not true. Spark SQL and PySpark can be accurately tested in notebooks, and production performance doesn't solely depend on language choice. B. Production-sized data and clusters are necessary: While ideal, it's not always feasible for development. Smaller datasets and clusters can provide indicative insights. C. IDE and local Spark/Delta Lake: Local environments won't replicate production's scale and configuration fully. E. Jobs UI and Photon: True that Photon benefits scheduled jobs, but Jobs UI can track execution times regardless of Photon usage. However, Jobs UI runs might involve additional overhead compared to notebook cells. Option D addresses the specific limitations of using display() for performance measurement

As B not talking about how to deal with display() function. We know that way to testing performance for the whole notebook need to avoid using display as it is way to test the code and display the data