Exam Professional Data Engineer topic 1 question 128 discussion

This is a case of underfitting - not overfitting (for over fitting the model will have extremely low training error but a high testing error) - so we need to make the model more complex - answer is D

should be D

Could be B (data requirement for task is vague), but let's assume 100 million data points is enough and rule that out. Indication of overfitting is significantly better performance on training data compared to unseen data. Here we are told that the unseen data is performing significantly better which is the opposite of what we should see if it were overfitting. Rule out C. Symptoms of model underfitting is poor performance in BOTH training AND unseen data. While underfitting might be the issue, the more pressing concern is that the test set is clearly not representative of the overall data and could be skewed. This is further supported by the 90/10 split (academic/industry standard is 80/20 or 75/25 based on the Pareto principle: https://en.wikipedia.org/wiki/Pareto_principle). A 90/10 split would be useful if we were doing k-fold cross validation (https://machinelearningmastery.com/k-fold-cross-validation/), however there is no indication of such in the prompt. Note: The question does not explicitly say that the model is performing poorly/errors are significantly bad, just that the error is twice as high in the training set (they could both have low error values). So whilst it could be a case of underfitting (D), the first step taken should be addressing the obviously problematic data representation by adjusting the train-test split (option A).

It is underfitting problem, which means that the used models is too easy.

This is A. The key is that 90/10 is a weirdly small test set, that stood out to me straight away (I work professionally as a machine learning engineer and have the cert). Next tip, that everyone seems to be ignoring - this is not underfit OR overfit. The model outperforms on the TEST set, this is not a miswording. Test scores higher than train. The time you might expect to see this is if your test set is too small to be a representative sample, leading to unrepresentative results. Seeing as the question already set up this conclusion with the 90/10 thing, it's definitely A. None of the others (or indeed anything else) can address Test outperforming Train, and the conclusion of others below that this is due to a poorly worded question is a bizarre conclusion.

Underfitting scenario

It is an underfitting situation - D

Should be C C. Try out regularization techniques (e.g., dropout or batch normalization) to avoid overfitting: This is a reasonable approach. Regularization techniques can help prevent overfitting, especially when the model shows a significantly higher error on the training set compared to the test set. D. Increase the complexity of your model (e.g., introducing an additional layer or increasing the size of vocabularies or n-grams): This could potentially exacerbate the overfitting issue. Increasing model complexity without addressing overfitting concerns may lead to poor generalization on new data.

Are the questions in this relevant for the new exam or are these all now outdated?

https://stats.stackexchange.com/questions/497050/how-big-a-difference-for-test-train-rmse-is-considered-as-overfit#:~:text=RMSE%20of%20test%20%3C%20RMSE%20of,is%20always%20overfit%20or%20underfit. RMSE of test > RMSE of train => OVER FITTING of the data. RMSE of test < RMSE of train => UNDER FITTING of the data. so for answer is D

Underfitting models: In general High Train RMSE, High Test RMSE. Overfitting models: In general Low Train RMSE, High Test RMSE. https://daviddalpiaz.github.io/r4sl/regression-for-statistical-learning.html

I passed the exam today. I am pretty sure it is overfitting. Answer must be c

RMSE is more on training. That means, model is not performing well on training dataset but performing well on testing dataset. This happens in the case of underfitting. So D.

RMSE training = 2 x testing When training > testing, it is a case of underfitting Hence D

chatGPT says option C

"root-mean-squared error (RMSE) of your model is twice as high on the train set as on the test set." means the RMSE of training set is two time of RMSE of test set, which indicates the training is not as good as test, then underfiting, so D.

Based on the given information, this scenario indicates a case of overfitting. Overfitting occurs when a machine learning model performs well on the training data but poorly on unseen data (test data). In this case, the root-mean-squared error (RMSE) of the model is twice as high on the train set (the data used for training) compared to the test set (the data used for evaluation). This suggests that the model is fitting the training data too closely and is not generalizing well to new, unseen data. So with dropout method we can overcome the overfitting so C is correct