Exam Certified Data Engineer Professional topic 1 question 21 discussion

I believe this is a case of the least bad option, not exactly the best option possible. - A is wrong because in Streaming you very rarely have any executors idle, as all cores are engaged in processing the window of data; - B is wrong because triggering every 30s will not meet the 10s target processing interval; - C is wrong in two manners: increasing shuffle partitions to any number above the number of available cores in the cluster will worsen performance in streaming; also, the checkpoint folder has no connection with trigger time. - D is wrong because, keeping all other things the same as described by the problem, keeping the trigger time as 10s will not change the underlying conditions of the delay (i.e.: too much data to be processed in a timely manner). E is the only option that might improve processing time.

Answer E, see explanation by RafaelCFC.

Option A emphasizes utilizing idle executors to begin processing the next batch while longer-running tasks from previous batches finish. This approach can help maintain a steady flow of data processing and reduce the likelihood of bottlenecks.

If microbatch execution occasionally exceeds 30 seconds, a trigger interval of 5 seconds would cause multiple batches to queue up while the previous batch is still running. This would exacerbate the delays and potentially lead to backpressure and failure. B is the best option in this case. If we assume for sure that execution time should be less than 10s, then in that case a 5s interval will make more sense and E should be the best answer.

In Databricks Runtime 11.3 LTS and above, the Trigger.Once setting is deprecated. Databricks recommends you use Trigger.AvailableNow for all incremental batch processing workloads. https://docs.databricks.com/en/structured-streaming/triggers.html Doesn't seem like E is a valid and recommended option given that it is deprecated.

Considering the best option for performance gain is: E. Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.

E is the answer. Enable the settings uses the 128 MB as the target file size https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size

E is correct as A is wrong because in Streaming you very rarely have any executors idle

I think is E is correct

correct answer is E

Only C. Even if you trigger more frequently you decrease both load and time for this load. E doesn't change anything.

Changing trigger interval to "one" will cause this to be a "batch" and will not execute in microbranches. This will not help at all

correct answer is E

sorry, the caveat is holding all other variables constant.. that means we are not allowed to change trigger intervals. is C the answer then

what if in between those 5 seconds trigger interval if there are more records, that would still increase the time it takes to process.. i doubt E is correct. I will go with answer D. it is not to execute all queries within 10 secs. it is to execute trigger now batch every 10 seconds.

A option also is about setting trigger interval to 5 seconds, just to understand.. why its not the answer

for sure E