Exam Professional Machine Learning Engineer topic 1 question 85 discussion

D - https://cloud.google.com/natural-language/automl/docs/beginners-guide - https://cloud.google.com/vertex-ai/docs/text-data/classification/evaluate-model

A. Number of messages flagged by the model per minute => NO, no measure of model performance B. Number of messages flagged by the model per minute confirmed as being inappropriate by humans.=> DONT THINK SO, because we need the total number of messages (flagged?) C. Precision and recall estimates based on a random sample of 0.1% of raw messages each minute sent to a human for review. => I think YES, because as I understand it that would be based on a sample of ALL messages not just the ones that have been flagged. D. Precision and recall estimates based on a sample of messages flagged by the model as potentially inappropriate each minute => I think NO, because the sample includes only flagged messages, meaning positives, so you cannot really measure recall.

C is correct: A random sample of raw messages provides an unbiased evaluation of the model's performance across all types of content Option D is problematic because: Creates a biased sample by only reviewing flagged messages Cannot detect false negatives (missed inappropriate content)

I went with B. Remember how to calculate Recall: TP/(TP+FN). Since "sample of messaged flagged by the model" are only P cases, you won't have your F cases reviewed by a human, therefore you won't have FN, therefore it's not D. I also believe that 0.1% of raw messages is going to have too little P cases, therefore not C. And then we remain with option B, which is not optimal, but it is the best we can do in this situation.

It is absolutely not possible to calculate recall with D because we only have positives in the sample we need false negatives. Because of the high quantity of total data, 0.1% is fine, the answer is C

Precision and recall are critical metrics for evaluating the performance of classification models, especially in contexts where both the accuracy of positive predictions (precision) and the ability to identify all positive instances (recall) are important. In this case: Precision (the proportion of messages flagged by the model as inappropriate that were actually inappropriate) helps ensure that the model minimizes the burden on human moderators by not flagging too many false positives, which could overwhelm them. Recall (the proportion of actual inappropriate messages that were correctly flagged by the model) ensures that the model is effective at catching as many inappropriate messages as possible, reducing the risk of harmful content being missed.

I go with C

Let's consider below hypothetical scenario: Total number of comments per minute: 10,000 Comments actually inappropriate: 500 If we use a random sample of only 0.1% of raw messages (10 comments) for evaluation, there's a high chance that this small sample may not include any or only a few inappropriate comments. As a result, the precision and recall estimates based on this sample may be skewed, leading to unreliable assessments of the model's performance. Thus, C is ruled out.

C does not make sense to me since it is a very small random sample. It is also only messages that have been sent to humans for review meaning that there is bias in that result set.

D only caring for observations flagged by the model means we don't control for false negatives (approved actually inappropriate messages). B seems like a better option to me: the wording confuses me a bit, but I understand it as the true and false positives (human flagged comments and their modelled label)

In favor of D

Given the context of content moderation, a balanced approach is often preferred. Therefore, option C, precision and recall estimates based on a random sample of raw messages, is a good choice. It provides a holistic view of the model's performance, taking into account both false positives (precision) and false negatives (recall), and it reflects how well the model is handling the entire dataset.

A --> Conveys model'a activity levels but nit accuracy B --> Accuracy to some extend but wont give full picture as it does not account False negatives C --> Using a random sample of the raw messages allows you to estimate precision and recall for the overall activity, not just the flagged content. D --> Specifically measures on the subset of data that it flagged Both C & D work well in this case, but the specificity is higher in option D and hence will go with D

Google Cloud used to have a service called "continuous evaluation", where human labelers classify data to establish a ground truth. Thinking along those lines, the answer is C as it's the logical equivalent of that service. https://cloud.google.com/ai-platform/prediction/docs/continuous-evaluation

Question is to measure model performance so has to be precision & recall , hence D.

D. Precision and recall estimates based on a sample of messages flagged by the model as potentially inappropriate each minute You will need precision and recall to identify fals positives and false negatives. A very small random sample doesn't help specially becasue probably you will have skewed data. So D.

Went with D