Exam Professional Data Engineer topic 1 question 282 discussion

It should B, Storage Write API has "3 GB per second throughput in multi-regions; 300 MB per second in regions"

- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics. - Regional Table: Choosing a regional location for the BigQuery table could potentially provide better performance and lower latency, as it would be closer to the Dataflow job if they are in the same region.

3 GB per second throughput in multi-regions; 300 MB per second in regions

Go With (B) Not (A), Max throughput for regional currently only 300 MB/s as for the dcos

Honestly, I think exam topic should do better job. the answers only make more mislead.

You need a write latency of 1.5 GBs per second. Given the high throughput requirement, a regional BigQuery table (Option A) is generally preferred over a multi-regional table due to potentially lower write latency in multi-region. Simple.

As per GCP document multi-region meets the troughput requirement.

It’s A

A. Implement the BigQuery Storage Write API and guarantee that the target BigQuery table is regional. Here's the breakdown: Why Option A is Superior Exactly-Once Delivery: The BigQuery Storage Write API intrinsically supports exactly-once delivery using stream offsets. This guarantees that each message is written to BigQuery exactly one time, even in the case of retries due to the lack of native exactly-once support in your message bus. High Throughput: The Storage Write API is optimized for high-throughput scenarios. It can handle the expected ingestion throughput of 1.5 GB per second. Regional Tables: Using a regional BigQuery table aligns with best practices when utilizing the Storage Write API, as it helps to minimize latency and reduce potential cross-region communication costs.

- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics. - The multiregional table ensures that your data is highly available and can be streamed into BigQuery across multiple regions. It is better suited for high-throughput and low-latency workloads, as it provides distributed write capabilities that can handle large data volumes, such as the 1.5 GB per second you expect to stream.

https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery For new projects, we recommend using the BigQuery Storage Write API instead of the tabledata.insertAll method. The Storage Write API has lower pricing and more robust features, including exactly-once delivery semantics https://cloud.google.com/bigquery/docs/write-api#advantages

B is correct. When aiming for exactly-once delivery in a Dataflow streaming job, the key is to use the BigQuery Storage Write API, as it provides the capability to handle large-scale data ingestion with the correct semantics, including exactly-once delivery.

3 GB per second throughput in multi-regions; 300 MB per second in regions https://cloud.google.com/bigquery/quotas#write-api-limits

streamed into BigQuery with exactly-once delivery semantics >>> Storage Write API ingestion throughput into BigQuery to be about 1.5 GB per second >>> multiregional (check throughput rate here >>> https://cloud.google.com/bigquery/quotas#write-api-limits)

- Choosing a regional target BigQuery table ensures that data is stored redundantly in a single region, providing high availability and durability.

According to this documentation, its B https://cloud.google.com/bigquery/quotas#write-api-limits

Write API support 2.5 GB / sec speed and support exactly-once delivery semantics https://cloud.google.com/bigquery/docs/write-api#connections whereas in streaming duplicates can come and needed to remove them manually https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery#dataavailability