Exam Certified Data Engineer Associate topic 1 question 72 discussion

Ans E : Development and production modes You can optimize pipeline execution by switching between development and production modes. Use the Delta Live Tables Environment Toggle Icon buttons in the Pipelines UI to switch between these two modes. By default, pipelines run in development mode. When you run your pipeline in development mode, the Delta Live Tables system does the following: Reuses a cluster to avoid the overhead of restarts. By default, clusters run for two hours when development mode is enabled. You can change this with the pipelines.clusterShutdown.delay setting in the Configure your compute settings. Disables pipeline retries so you can immediately detect and fix errors. In production mode, the Delta Live Tables system does the following: Restarts the cluster for specific recoverable errors, including memory leaks and stale credentials. Retries execution in the event of specific errors, for example, a failure to start a cluster. https://docs.databricks.com/en/delta-live-tables/updates.html#optimize-execution

The pipeline runs in Continuous Pipeline Mode, so datasets will be updated at set intervals. Since the pipeline is in Development Mode, the compute resources will persist until manually shut down.

E is correct ! Option B: "All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist until the pipeline is shut down." This option correctly reflects that the pipeline continues running, updating datasets at intervals, and only stops when manually shut down. Compute resources persist throughout this process. Option E: "All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist to allow for additional testing." While this is very similar, it adds the phrase "to allow for additional testing," which might imply that the resources are persisting just for testing purposes. This can be misleading because the primary reason for resource persistence in Continuous mode is to keep the pipeline active and processing data, not solely for testing.

The answer is E. The compute resources will persist even after the pipeline is shut down.

e as teh cluster actually persits differently from b

E. All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist to allow for additional testing. Explanation: In Development mode, Delta Live Tables persistently updates datasets at set intervals. The pipeline continuously processes incoming data until manually stopped or shut down. Compute resources, including the cluster used for processing, persist without automatic restarts or retries (as it is the behavior in Development mode). This persistence allows for ongoing processing of data, enabling additional testing or continued data processing until the pipeline is manually shut down. Therefore, option E accurately captures the behavior expected in Development mode, emphasizing the continuous update of datasets and the persistence of compute resources until the pipeline is manually terminated.

E seems the correct answer

Why E? It persists with same functionality as was before, not for "additional testing"?

https://docs.databricks.com/en/delta-live-tables/updates.html#continuous-vs-triggered-pipeline-execution https://docs.databricks.com/en/delta-live-tables/testing.html#use-development-mode-to-run-pipeline-updates

E is correct

E. All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist to allow for additional testing.