Exam Professional Data Engineer topic 1 question 137 discussion

It should be B and C

BC is correct Why the comments is deleted?

If there's a write speed bottleneck on bigtable, more dataflow workers won't make a difference. If I add more bigtable nodes, or group my writes together, I can increase update throughput. https://cloud.google.com/dataflow/docs/guides/write-to-bigtable#best-practices

the goal is to reduce the write latency not to improve data flow code

The "Correct Answers" are just put in with a random generator :-) B and C

B - opportunity to parallelise the process C - increase throughput

Exactly opposite answers in the discussions

B. Increase the maximum number of Dataflow workers by setting maxNumWorkers in PipelineOptions: Increasing the number of Dataflow workers can help parallelize the processing of your data, which can result in faster data updates to Bigtable and improved concurrency. You can set maxNumWorkers to a higher value to achieve this. C. Increase the number of nodes in the Bigtable cluster: Increasing the number of nodes in your Bigtable cluster can improve the overall throughput and reduce latency when writing data. It allows Bigtable to handle a higher rate of data ingestion and queries, which is essential for supporting additional concurrent users.

C definetely is correct, as it improves the read and write performance of Bigtable. However, I do think that the second option is actually D instead of B, because the question specifically states that the pipeline aggregates data. Flatten merges multiple PCollection objects into a single logical PCollection, allowing for faster aggregation of time series data.

B - I believe it is consensus. D - The question mentions "a Dataflow job that "aggregates" and writes time series metrics to Bigtable". So CoGroupByKey performs a shuffle (grouping) operation to distribute data across workers. https://cloud.google.com/dataflow/docs/guides/develop-and-test-pipelines

I read this question as: BigTable Write operations are all over the place (key-wise), and BigTable doesn't like that. When creating groups (batch writes), of similar keys (close to each other), BigTable is happy again, which I loosely translate into DE.

B is correct. But I don't see how you increase the write throughput of Bigtable increasing its cluster size. It should be dataflow instance resources that have to be increased

BC make sense

BC only makes sense here , no mention of data, no mention of keeping cost low

B. Increase the maximum number of Dataflow workers by setting maxNumWorkers in PipelineOptions Most Voted C. Increase the number of nodes in the Bigtable cluster

Increase max num of workers increases pipeline performance in Dataflow Increase number of nodes in Bigtable increases write throughput