Exam Certified Data Engineer Professional topic 1 question 21 discussion

In Databricks Runtime 11.3 LTS and above, the Trigger.Once setting is deprecated. Databricks recommends you use Trigger.AvailableNow for all incremental batch processing workloads. https://docs.databricks.com/en/structured-streaming/triggers.html Doesn't seem like E is a valid and recommended option given that it is deprecated.

Considering the best option for performance gain is: E. Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.

E is the answer. Enable the settings uses the 128 MB as the target file size https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size

E is correct as A is wrong because in Streaming you very rarely have any executors idle

I think is E is correct

I believe this is a case of the least bad option, not exactly the best option possible. - A is wrong because in Streaming you very rarely have any executors idle, as all cores are engaged in processing the window of data; - B is wrong because triggering every 30s will not meet the 10s target processing interval; - C is wrong in two manners: increasing shuffle partitions to any number above the number of available cores in the cluster will worsen performance in streaming; also, the checkpoint folder has no connection with trigger time. - D is wrong because, keeping all other things the same as described by the problem, keeping the trigger time as 10s will not change the underlying conditions of the delay (i.e.: too much data to be processed in a timely manner). E is the only option that might improve processing time.

correct answer is E

Only C. Even if you trigger more frequently you decrease both load and time for this load. E doesn't change anything.

Changing trigger interval to "one" will cause this to be a "batch" and will not execute in microbranches. This will not help at all

correct answer is E

sorry, the caveat is holding all other variables constant.. that means we are not allowed to change trigger intervals. is C the answer then

what if in between those 5 seconds trigger interval if there are more records, that would still increase the time it takes to process.. i doubt E is correct. I will go with answer D. it is not to execute all queries within 10 secs. it is to execute trigger now batch every 10 seconds.

A option also is about setting trigger interval to 5 seconds, just to understand.. why its not the answer

for sure E

correct anwer is E

correct answer is E. D means a job will need to acquire resources in 10s which is impossible without serverless