Exam Professional Data Engineer topic 1 question 31 discussion

To all the New Guys Here. Please don't get confused with all the people's fight over here. Just google the question and you will get the correct ans in many website. Still I recommend to refer this website for question. for this Particular problem ans is A. Reason is here --> https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#python have time to read the full page when to use Update using Json mapping and when to use Drain. (you will have question following for Drain option though). Thumb rule is this, # If any major change to windowing transformation (like completely changing window fn from fixed to sliding) in Beam/Dataflow/you want to stop pipeline but want inflight data --> use Drain option. # For all other use cases and Minor changing to windowing fn (like just changing window time of sliding window) --> Use Update with Json mapping. In this case it is Code change to new version. so, Update with Json mapping. Simple as that. All the Best Guys.

C and D are incorrect because canceling the old pipeline can cause data loss https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline A is incorrect because updating pipeline does not include any drain flag https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline

Option C & D are incorrect as Cancel Option will lead to loose the data Option B is very Close, since the new Code make pipeline incompatible by providing transform mapping JSON file you can handle this

As you said, Drain stops the pipeline but it does not solve the compatibility issue. The pipeline will not be able to be updated which is the core problem of the question.

Changing the pipeline graph without providing a mapping. When you update a job, the Dataflow service attempts to match the transforms in your prior job to the transforms in the replacement job in order to transfer intermediate state data for each step. If you've renamed or removed any steps, you'll need to provide a transform mapping so that Dataflow can match state data accordingly.

While other options might work in some cases, they have drawbacks: B. Transform mapping JSON: This option is mainly for schema changes and doesn't guarantee data completion before shutdown. C. New pipeline, same subscription: This risks duplicate processing of data if both pipelines run concurrently. D. New pipeline, new subscription: This loses the current pipeline's state and potentially data, making it impractical for incompatible changes. Therefore, the most reliable and data-safe approach is to update the current pipeline with the drain flag for seamless transition and data integrity. Remember, always test updates in a staging environment before deploying to production.