Exam Professional Machine Learning Engineer topic 1 question 150 discussion

B. Recall In a highly imbalanced dataset like the one described (96% of examples are in the negative class), the metric that would give the most confidence in the model's performance is recall. Recall (also known as sensitivity or true positive rate) is the proportion of actual positive cases that were correctly identified by the model. In this context, it means the percentage of images containing the company's logo that the model correctly classified as positive out of all the actual positive cases. Since the dataset is heavily skewed, a high recall value would indicate that the model is effectively capturing the positive cases (images with the logo) despite the class imbalance. F1 score (D) is a balance between precision and recall and is a useful metric for imbalanced datasets. However, in this specific case, recall is more important because we want to be confident in detecting the logo images, even if it comes at the cost of having some false positives (lower precision).

Precision vs. Recall: Precision focuses on the percentage of predicted positive cases (logo present) that are actually correct. Recall emphasizes the model's ability to identify all actual positive cases (correctly identifying all logos). In a highly imbalanced dataset, a naive model could simply predict "no logo" for every image and achieve very high accuracy (almost 96%!). However, this wouldn't be a useful model since it would miss all the actual logos (low recall). F1 Score: The F1 score strikes a balance between precision and recall. It takes the harmonic mean of these two metrics, giving a more comprehensive picture of the model's performance in both identifying logos (recall) and avoiding false positives (precision).

The focus here is on detecting images with logo (that is the minority class). Recall is the right metric. If the question would have highlighted that also the detection of images without logo is important I would have voted for D. But here is not the case. And of course F1 avoids that the model always mark as "with logo", but this is more on how the metric is used and interpreted. Otherwise when should be Recall useful?

Went with D

B See #90, should be F score with Recall weights more than Precision.

I went with B.

F1 score provides a comprehensive evaluation by penalizing models that excel in just one aspect at the expense of the other. By considering both precision and recall, it helps identify models that effectively balance true positive identification with minimal false positives, making it a more suitable metric for imbalanced data like your logo detection problem.

See #90!

F1 score works well for imbalanced data sets

https://stephenallwright.com/imbalanced-data-metric/