Exam Professional Data Engineer topic 1 question 112 discussion

Should be B

C. This is a tricky one. The issue here is the unreliable connection between data collection and data processing infrastructure, and to resolve it in a cost-effective manner. However, it also mentions that the company is using leased lines. I think replacing the leased lines with Cloud InterConnect would solve the problem, and hopefully not be an added expense. https://cloud.google.com/interconnect/docs/concepts/overview

Option B: Have the data acquisition devices publish data to Cloud Pub/Sub. Rationale: Managed Service: Cloud Pub/Sub is a fully managed service, reducing the operational overhead compared to managing Kafka clusters. Reliability and Scalability: Cloud Pub/Sub can handle high volumes of data with low latency and provides built-in mechanisms for reliable message delivery, even in the face of intermittent connectivity. Cost-Effective: Cloud Pub/Sub offers a pay-as-you-go pricing model, which can be more cost-effective than setting up and maintaining dedicated network infrastructure like Cloud Interconnect. Global Availability: Cloud Pub/Sub is available globally and can handle data from multiple regions efficiently.

Even with Cloud Pub/Sub, unpredictable latency or delays could still occur due to the unreliable leased lines connecting your event collection infrastructure and event processing infrastructure. While Cloud Pub/Sub offers reliable message delivery within its own network, the handoff to your processing infrastructure is still dependent on the leased lines. Replacing leased lines with Cloud Interconnect could potentially resolve the overall issue of unpredictable latency in event processing pipeline but it could be unnecessary expense provided data centers distributed world wide. Cloud Pub/Sub along with other optimization techniques like Cloud VPN or edge computing might be sufficient.

I don't know why B is the most voted. The issue here is unreliable connectivity and C is the perfect use-case for that

its says with unpredictable latency and here no need to worry about connection So B is the right one

The question is misleading. But should be C since it addresses the unpredictablility and latency directly.

Best answer is A, By using Kafka, you can buffer the events in the data centers until a reliable connection is established with the event processing infrastructure. But go with B, its google asking :P

B. Have the data acquisition devices publish data to Cloud Pub/Sub. This would provide a reliable messaging service for your event data, allowing you to ingest and process your data in a timely manner, regardless of the reliability of the leased lines. Cloud Pub/Sub also offers automatic retries and fault-tolerance, which would further improve the reliability of your event delivery. Additionally, using Cloud Pub/Sub would allow you to easily scale up or down your event processing infrastructure as needed, which would help to minimize costs.

Are they talking about GCP in this question? Where is the event processing infrastructure? Answer A, might be correct!

pub/sub is region is a global service It's important to note that the term "global" in this context refers to the geographical scope of the service

As usual the answer is hidden somewhere in the Google Cloud Blog: "In the case of our automotive company, the data is already stored and processed in local data centers in different regions. This happens by streaming all sensor data from the cars via MQTT to local Kafka Clusters that leverage Confluent’s MQTT Proxy." "This integration from devices to a local Kafka cluster typically is its own standalone project, because you need to handle IoT-specific challenges like constrained devices and unreliable networks." 🔗 https://cloud.google.com/blog/products/ai-machine-learning/enabling-connected-transformation-with-apache-kafka-and-tensorflow-on-google-cloud-platform

B is the answer.

yeah, changing whole architecture arround the world for the use of pub/sub is so much more cost efficient than Cloud Interconnect (which is like 3k$).. It's C.

Cloud Pub/Sub, it supports batch & streaming , push and pull capabilities Answer B

It has to be B.

Feels like everyone is wrong. A. Deploy small Kafka clusters in your data centers to buffer events. - Silly in a GCP cloudnative context, plus they have messaging infra anyway B. Have the data acquisition devices publish data to Cloud Pub/Sub. - They have messaging infra, so why? Unless they want to replace, it, but that doesn't change the issue C. Establish a Cloud Interconnect between all remote data centers and Google. - Wrong, because Interconnect is basically a leased line. There must be some telecoms issue with it, which we can assume is unresolvable e.g. long distance remote locations and sometimes water ingress, and the telco can't justify sorting it yet, or is slow to, or something. Leased lines usually don't come with awful internet connectivity, so sound physical connectivity issue. Sure, an Interconnect is better, more direct, but a leased line should be bullet proof. D. Write a Cloud Dataflow pipeline that aggregates all data in session windows. - The only way to address dodgy/delayed data delivery