Exam Professional Data Engineer topic 1 question 68 discussion

Answer should be C

Answer: C - best suitable for the purpose with autoscaling and google recommended transform engine between pubsub n bq

C for me

using Cloud Dataflow for transformations with monitoring via Stackdriver and leveraging its default autoscaling settings, is the best choice. Cloud Dataflow is purpose-built for this type of workload, providing seamless scalability and efficient processing capabilities for streaming data. Its autoscaling feature minimizes manual intervention and helps manage costs by dynamically adjusting resources based on the actual processing needs, which is crucial for handling fluctuating data volumes efficiently and cost-effectively.

Option C suggests using Cloud Dataflow to run the transformations and monitoring the job system lag with Stackdriver while using the default autoscaling setting for worker instances. While using Cloud Dataflow is a suitable choice for processing data from Cloud Pub/Sub to BigQuery, and monitoring with Stackdriver provides valuable insights, the specific emphasis on configuring non-default Compute Engine machine types (as mentioned in option D) gives you more control over cost optimization and performance tuning. By configuring non-default machine types, you can precisely tailor the computational resources to match the specific requirements of your workload. This fine-grained control allows you to optimize costs further by avoiding over-provisioning of resources, especially if your workload is memory-intensive, CPU-bound, or requires specific configurations that are not met by the default settings.

At first I answered C. However, Dataproc is indeed cheaper than Dataflow. And both of them can scale automatically horizontically. Dataflow horizontal scaling applies to both primary and secondary nodes. Scaling secondary nodes scales up CPU/compute and scaling primary nodes scales up both memory and CPU/compute. I don't quite understand the second part of answer B where it says I should allocate resources accordingly. I guess I could do that, but auto-scaling should be enough.

Answer should be C

C. Use Cloud Dataflow to run your transformations. Monitor the job system lag with Stackdriver. Use the default autoscaling setting for worker instances. Cloud Dataflow is a managed service that allows you to write and execute data transformations in a highly scalable and fault-tolerant way. By default, it will automatically scale the number of worker instances based on the input data volume and job performance, which can help minimize costs. Monitoring the job system lag with Stackdriver can help you identify any issues that may be impacting performance and take action as needed. Additionally, using the default autoscaling setting for worker instances can help you minimize manual intervention and ensure that resources are used efficiently.

C is the answer.

@admin why all the answers are wrong. I paid 30 euros for this web and its garbage. Dataproc has no sense in this scenario, because you want to have minimal intervention/operation. D is not a good practice, the answer is C.

C only as referred by MaxNRG

C. Dataproc does not seem to be a good solution in this case as it always require a manual intervention to adjust resources. Autoscaling with dataflow will automatically handle changing data volumes with no manual intervention, and monitoring through Stackdriver can be used to spot bottleneck. Total execution time is not good there as it does not provide a precise view on potential bottleneck.

Dataflow, Stackdriver and autoscaling

Admin, please take a look on the comments. Almost all answers are wrong

Answer should be C as dataflow is unpredictable size ( input that will vary in size), dataproc is with known size

Option C is the answer

Vote for C