Exam Professional Machine Learning Engineer topic 1 question 113 discussion

Previously Google pattern was Pub/Sub -> Dataflow -> BQ but now it looks as there is new Pub/Sub -> BQ https://cloud.google.com/blog/products/data-analytics/pub-sub-launches-direct-path-to-bigquery-for-streaming-analytics

The question specifies that transformation occurs in Bigquery. This means the new direct pub/sub to bigquery streaming path is correct.

Need PubSub and Dataflow both for this

Werner123 i agree

User data would most likely include PII, for that case it is still recommended to use Dataflow since you need to remove/anonymise sensitive data.

I would have added "with / without data transformation" to the question to choose the right answer between A or D

I had my doubts between A and D. But since the transformation will occur in bigquery I think Pubsub suffices.

Agree with TNT87. From the same link: “For Pub/Sub messages where advanced preload transformations or data processing before landing data in BigQuery (such as masking PII) is necessary, we still recommend going through Dataflow.” It’s “analyze user activity data”, not merely streaming IoT into BigQuery so that concerns like privacy are per se n/a. One can deal with PII after landing in BigQuery as well, but apparently that’s not what they recommend.

Pub/Sub -> DataFlow -> BigQuery

D. Configure Pub/Sub and a Dataflow streaming job to ingest the data into BigQuery. This solution involves using Google Cloud Pub/Sub as the messaging service to receive the data from the mobile application, and then using Google Cloud Dataflow to transform and load the data into BigQuery in real time. Pub/Sub is a scalable and reliable messaging service that can handle high-volume real-time data streaming, while Dataflow provides a unified programming model to develop and run data processing pipelines. This solution is suitable for handling large volumes of user activity data from mobile applications and ingesting it into BigQuery in real-time for analysis and ML experimentation.

A agree with pshemol

need dataflow