Exam Professional Machine Learning Engineer topic 1 question 151 discussion

"Evaluator leverages the TensorFlow Model Analysis library to perform the analysis, which in turn use Apache Beam for scalable processing." Since Dataflow is Google Cloud's serverless Apache Beam offering, this option can easily be implemented to address the issue while leaving the evaluation logic as such identical https://www.tensorflow.org/tfx/guide/evaluator#evaluator_and_tensorflow_model_analysis

Links already provided below: “That works fine for one hundred records, but what if the goal was to process all 187,002,0025 rows in the dataset? For this, the pipeline is switched from the DirectRunner to the production Dataflow runner.” [Option A] https://blog.tensorflow.org/2020/03/tensorflow-extended-tfx-using-apache-beam-large-scale-data-processing.html. "Metrics to configure (only required if additional metrics are being added outside of those saved with the model).” https://www.tensorflow.org/tfx/guide/evaluator#using_the_evaluator_component will thus add, not “limit the number of metrics in the evaluation step”. [Option D]

with D we're downgrading evaluation. Dataflow is serverless so no infrastructure overhead is included

Limiting Metrics: TensorFlow Model Analysis (TFMA) allows you to define a subset of metrics that you are interested in during the evaluation step. By using tfma.MetricsSpec(), you can specify a subset of metrics to be computed during the evaluation, which can help reduce the memory requirements. Out-of-Memory Error: Out-of-memory errors during model evaluation often occur when the system is trying to compute and store a large number of metrics, especially if the model or dataset is large. By limiting the number of metrics using tfma.MetricsSpec(), you can potentially reduce the memory footprint and resolve the out-of-memory error.

Based on the question's context, the correct option to stabilize the pipeline without downgrading the evaluation quality while minimizing infrastructure overhead is: D. Add tfma.MetricsSpec() to limit the number of metrics in the evaluation step. The question specifies that the evaluation step is failing due to an out-of-memory error. In such a scenario, limiting the number of metrics to be computed during evaluation using tfma.MetricsSpec() can help reduce memory requirements and potentially resolve the out-of-memory issue.

By adding tfma.MetricsSpec(), you can limit the number of metrics that are computed during the evaluation step, thus reducing the memory requirement. This will help stabilize the pipeline without downgrading the evaluation quality, while minimizing infrastructure overhead. This option is a quick and easy solution that can be implemented without significant changes to the pipeline or infrastructure. Option A: Including the flag -runner=DataflowRunner in beam_pipeline_args to run the evaluation step on Dataflow may help to increase memory availability, but it may also increase infrastructure overhead.

D does not harm the evaluation quality.

Surely removing evaluation metrics downgrades the quality of the evaluation

I'm not very sure, but wouldn't be A?.D is degrading evaluation quality (if you're getting less metrics, then the evaluation is worse, at least less complete)

TFX 0.30 and above adds an interface, with_beam_pipeline_args, for extending the pipeline level beam args per component tfma.MetricSpec() OOB has recommended metrics; reducing any further might not serve the purpose.

Add tfma.MetricsSpec () to limit the number of metrics in the evaluation step. Limiting the number of metrics in the evaluation step using tfma.MetricsSpec() can reduce the memory usage during evaluation and address the out-of-memory error. This can help stabilize the pipeline without downgrading the evaluation quality or incurring additional infrastructure overhead. Running the evaluation step on Dataflow or custom Compute Engine VMs can be resource-intensive and expensive, while migrating the pipeline to Kubeflow would require additional setup and configuration. ANSWER D

A is wrong , it does not even make sense , the default runner for evaluator component of TFX is data flow so setting runner to dataflow does not change anything , the answer is D because it does not include the any infrastructure minpulation and reduce the memory useable of the TfX component

Answer A

https://blog.tensorflow.org/2020/03/tensorflow-extended-tfx-using-apache-beam-large-scale-data-processing.html