Exam Professional Machine Learning Engineer topic 1 question 6 discussion

I think B is the right answer. A: Doesn't make sense. If you don't use the new product, it becomes useless. C: Conventional products are also necessary as data. D: I don't understand the need to wait until the threshold is exceeded.

answer is B

Task: "You also want to use AI Platform's continuous evaluation service to ensure that the models have high accuracy on your test dataset." Docs: https://cloud.google.com/vertex-ai/docs/evaluation/introduction "After your model is deployed to production, periodically evaluate your model with new incoming data. If the evaluation metrics show that your model performance is degrading, consider re-training your model. This process is called continuous evaluation."

D: Its definitely not a clear choice. B is the most obvious answer - you know you've got new data coming in, so why not incorporate it immediately into training. EXCEPT the question clearly states that Vertex continual evaluation should feature.

=New Question6= You work for a global footwear retailer and need to predict when an item will be out of stock based on historical inventory dat a. Customer behavior is highly dynamic since footwear demand is influenced by many different factors. You want to serve models that are trained on all available data, but track your performance on specific subsets of data before pushing to production. What is the most streamlined and reliable way to perform this validation? A. Use the TFX Mode!Validator tools to specify performance metrics for production readiness B. Use k-fold cross-validation as a validation strategy to ensure that your model is ready for production. C. Use the last relevant week of data as a validation set to ensure that your model is performing accurately on current data. D. Use the entire dataset and treat the area under the receiver operating characteristics curve (AUC ROC) as the main metric.

A. Keep the original test dataset unchanged even if newer products are incorporated into retraining. : This would not test on new products. B. Extend your test dataset with images of the newer products when they are introduced to retraining. Most Voted : old+new products testing. Great C. Replace your test dataset with images of the newer products when they are introduced to retraining. : No need of old product to be tested? old product recognition might change when new products are added in training. Option Not good. D. Update your test dataset with images of the newer products when your evaluation metrics drop below a pre-decided threshold.: why wait? no need

You need to correctly classify newer products, so you need the new training data ==> A is wrong; You need to keep doing a good job on older dataset, you can't just ignore it ==> C is wrong; You know when you are introducing new products, there is no need to wait for a drop in preformaces ==> D is wrong; B is correct

B correct

The best approach is option B: Extend your test dataset with images of the newer products. This ensures accurate evaluation as your product catalog evolves.

My initial confusion with option B arose from the phrase "with images of the newer products when they are introduced to retraining." Initially, I mistakenly interpreted it as recommending the use of the same images in both training and testing, which is incorrect. However, upon further reflection, I realized that using the same product does not necessarily mean using identical images. Therefore, I now believe that option B is the most suitable choice.

A and C make no sense - you don't want to lose any of the performance on existing products. D - Why would you wait for your performance to drop in the first place? That's a reactive rather than proactive approach. The answer is B

B for sure

B is the only thing we do in practice

Went with B

you can't just replace the old product data with just new product, until you don't sell old product anymore

Ans: B A would not use the newer data hence not a ideal option C Replacing will not be a good option as it will replace older data with newer data which in turn hampers accuracy D waiting for threshold is not a better option

B is the most plausible answer. The key principle is that test set should represent ground truth distribution to infer credible model evaluation. So once new products become available, test set should be updated to reflect the new product distribution