Exam Professional Data Engineer topic 1 question 268 discussion

- Graceful Data Transition: Draining the old pipeline ensures it processes all existing data in its buffers and watermarks before shutting down, preventing data loss or inconsistencies. - Minimal Latency Increase: The latency increase will be limited to the amount of time it takes to drain the old pipeline, typically within the acceptable 10-minute threshold.

I don't think C is correct, as it will immediately fire the window: "Draining can result in partially filled windows. In that case, if you restart the drained pipeline, the same window might fire a second time, which can cause issues with your data. " https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#effects Maybe "A" means launching a replacement job? https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching

Why not B? https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace

There is requirement to avoid data loss. https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace "To avoid data loss, in most cases, draining is the preferred action."

- Draining the old pipeline ensures that it finishes processing all in-flight data before stopping, which prevents data loss and inconsistencies. - After draining, you can start the new pipeline, which will begin processing new data from where the old pipeline left off. - This approach maintains a smooth transition between the old and new versions, minimizing latency increases and avoiding data gaps or overlaps. ==> Other options, such as updating, snapshotting, or canceling, might not provide the same level of consistency and could lead to data loss or increased latency beyond the acceptable 10-minute window. Draining is the safest method to ensure a seamless transition.

I would choose B as mentioned by Alizcert, a simple drain may cause problem Dataflow snapshots save the state of a streaming pipeline, which lets you start a new version of your Dataflow job without losing state. Snapshots are useful for backup and recovery, testing and rolling back updates to streaming pipelines, and other similar scenarios.

C option

Option C, draining the old pipeline solves all requests

C. Drain the old pipeline, then start the new pipeline.