Exam Professional Machine Learning Engineer topic 1 question 32 discussion

D is correct since this question is focusing on server performance which development env is higher than production env. It's already throttling so increase the pressure on them won't help. Both A and C is essentially doing this. B is a bit mysterious, but we definitely know that D would work.

https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/README.md#batch-scheduling-parameters-and-tuning A may help to some extent, but it primarily affects how many requests are processed in a single batch. It might not directly address latency issues. D is a valid approach for optimizing TensorFlow Serving for CPU-specific optimizations, but it's a more involved process and might not be the quickest way to address latency issues.

A is wrong - Increasing max_batch_size reduces latency by batching more requests together, but this introduces delays since the system must wait to accumulate a full batch. - This approach can improve throughput but may increase per-query latency, which contradicts the goal of reducing latency.

I do not think D is correct as D is focused on optimizing CPU utilization, not on the batching process or managing latency. Since our goal is to improve serving latency, optimizing batching via the max_batch_size parameter is a more straightforward and effective solution.

A would work

max_batch_size: Increasing the max_batch_size parameter allows TensorFlow Serving to process more requests in a single batch. This can improve throughput and reduce latency, especially in high-query environments, as it allows more efficient utilization of CPU resources by processing larger batches of requests at once. Answer A

I think the correct is D, because the question is about reducing latency. As for A, increasing the batch size might event hurt latency if the system is overwhelmed to serve more multiple requests

it is D

C) Batch enqueued

increasing the max_batch_size TensorFlow Serving parameter, is not the best choice because increasing the batch size may not necessarily improve latency. In fact, it may even lead to higher latency for individual requests, as they will have to wait for the batch to be filled before processing. This may be useful when optimizing for throughput, but not for serving latency, which is the primary goal in this scenario.

I think A is correct, as D implies changes to the infrastructure (question says you must not do that).

increasing the max_batch_size TensorFlow Serving parameter, is not the best choice because increasing the batch size may not necessarily improve latency. In fact, it may even lead to higher latency for individual requests, as they will have to wait for the batch to be filled before processing. This may be useful when optimizing for throughput, but not for serving latency, which is the primary goal in this scenario.

max_batch_size parameter controls the maximum number of requests that can be batched together by TensorFlow Serving. Increasing this parameter can help reduce the number of round trips between the client and server, which can improve serving latency. However, increasing the batch size too much can lead to higher memory usage and longer processing times for each batch.

Definetely D to improve the serving latency of an ML model on AI Platform, you can recompile TensorFlow Serving using the source to support CPU-specific optimizations and instruct GKE to choose an appropriate baseline minimum CPU platform for serving nodes, this way GKE will schedule the pods on nodes with at least that CPU platform.

Went with D

A is correct. max_batch_size TensorFlow Serving parameter

CPU-only: One Approach If your system is CPU-only (no GPU), then consider starting with the following values: num_batch_threads equal to the number of CPU cores; max_batch_size to a really high value; batch_timeout_micros to 0. Then experiment with batch_timeout_micros values in the 1-10 millisecond (1000-10000 microsecond) range, while keeping in mind that 0 may be the optimal value. https://github.com/tensorflow/serving/tree/master/tensorflow_serving/batching