Section 5.5 Evaluating Success

One primary way to evaluate a model is through a validation set—a distinct subset of data reserved specifically for testing a trained model’s quality. Because the validation set is completely separate from the training data, this practice guards against overfitting, a common failure mode where a model learns the training data too closely and fails to generalize to new information.

Overfitting is a common problem in machine learning. It occurs when a model learns the training data too well, including its noise and random fluctuations, rather than the underlying patterns that generalize to new data.

An overfitted model may achieve perfect accuracy on the training data but perform poorly on new, unseen data. This is like a student who memorizes the exact answers to practice problems but cannot solve slightly different versions of the same problem. The student has learned the specific examples but not the underlying concepts.

Overfitting tends to happen when models are too complex relative to the amount of training data. A complex model can fit almost any data perfectly, but at the cost of poor generalization. This is sometimes called the bias-variance tradeoff: simpler models have higher bias but lower variance, while complex models have lower bias but higher variance.

Traditionally, a complete dataset is divided into three mutually exclusive subsets:

The validation set serves as a "middle ground" between training and final evaluation. During the development process, data scientists often try many different model configurations, which are known as hyperparameters. The validation set is used to evaluate these different configurations and choose the best one without ever touching the test set.

Using a validation set helps prevent overfitting to the test set. If a model is evaluated on the test set too many times during development, it can inadvertently learn patterns in the test set, and the final evaluation will be overly optimistic. The validation set allows for honest evaluation during development while keeping the test set pristine for final assessment.

Ideally, each example in the dataset should belong to strictly one of these three subsets. For example, a single data point should never appear in both the training set and the validation set.

The quality of predictions from a model is formally measured using a loss function. A loss function quantifies how much a model is penalized when its predicted output differs from the actual true value.

Different loss functions are selected depending on the problem class, as certain errors carry far greater consequences than others. For example, in medical diagnostics, incorrectly predicting that a patient does not have a serious condition when they actually do (a missed diagnosis) is generally far more dangerous than incorrectly flagging a healthy patient for follow-up testing.

In classification problems, prediction errors are categorized based on the nature of the mistake:

The distinction between false positives and false negatives is important because the cost of each type of error can be very different in practice. In some situations, false positives are more serious. For example, in a spam filter, a false positive (flagging a legitimate email as spam) might cause you to miss an important message. In other situations, false negatives are more serious. In a medical screening, a false negative (missing a disease that is actually present) could be life-threatening.

The choice of how to balance these errors depends on the application. This is often represented using a confusion matrix, which shows the counts of true positives, false positives, true negatives, and false negatives. From this matrix, we can compute various performance metrics, such as accuracy, precision, and recall.

Ultimately, evaluating a machine learning model requires assessing its performance across hundreds or thousands of individual predictions. A truly successful model does not simply score well on its training data—it demonstrates strong generalization when presented with new, unseen data in real-world scenarios.

Generalization is the ability of a model to perform well on new, unseen data. This is the ultimate goal of machine learning. A model that cannot generalize is not learning—it is merely memorizing.

Achieving good generalization requires a balance of several factors: having enough high-quality training data, choosing a model of appropriate complexity, using regularization techniques to prevent overfitting, and thoroughly evaluating the model on a held-out test set. The entire machine learning process is designed with generalization in mind: training, validation, and testing are all steps toward building a model that can succeed in the real world.