Exam Professional Data Engineer topic 1 question 278 discussion

- Exception Handling in DoFn: Implementing an exception handling block within DoFn in Dataflow to catch failures during processing is a direct way to manage errors. - Side Output to New Topic: Using a side output to redirect failed messages to a new Pub/Sub topic is an effective way to isolate and manage these messages. - Monitoring: Monitoring the num_unacked_messages_by_region on the new topic can alert you to the presence of failed messages.

Option C - dead letter topic is built in and requires no changes https://cloud.google.com/pubsub/docs/handling-failures Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.

See here: https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#unsupported-features It's not recommended to use Pub/Sub dead-letter topics with Dataflow (...) Instead, implement the dead-letter pattern explicitly in the pipeline

Option B. Here's why: Side Output for Failed Messages: Dataflow allows you to use side outputs to handle messages that fail processing. In your DoFn , you can catch exceptions and write the failed messages to a separate PCollection . This PCollection can then be written to a new Pub/Sub topic. New Pub/Sub Topic for Monitoring: Creating a dedicated Pub/Sub topic for failed messages allows you to monitor it specifically for alerting purposes. This provides a clear view of any issues with your business logic. topic/num_unacked_messages_by_region Metric: This Cloud Monitoring metric tracks the number of unacknowledged messages in a Pub/Sub topic. By monitoring this metric on your new topic, you can identify when messages are failing to be processed correctly.

I would like to know why isn't anyone considering the option C.

option B

Option B

Option B - Raaad explanation is complete

B. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.