Section 4.4 The Data Split

Before any analysis or learning can take place, data must first be collected. Compiling a dataset involves identifying a target population—the entire group of individuals, objects, or events about which conclusions are to be drawn—as well as defining and measuring relevant features and labels.

Because it is rarely feasible to collect data from an entire target population, practitioners rely on sampling to collect a representative subset. In practice, machine learning teams frequently utilize existing, pre-compiled datasets rather than conducting new data collection from scratch. However, if the sampled data does not accurately reflect the broader population, the model risks inheriting significant representation bias.

Example.

For a loan-approval system, the target population could be people who live in the state in which the system will be used, have previously applied for loans, own credit cards, and so on. The particular sample that ends up in the data set will be a subset of this target population, heavily depending on the sampling method used (e.g., sourcing information from public records or surveying people). There is also the question of which features to measure, such as debt history, number of credit cards, income, and occupation. Some of these things will be chosen to serve as labels: for example, information about whether the person received or paid back a loan in the past.

Subsection 4.4.1 Data Preprocessing and Data Splitting

Depending on the data modality (e.g., text, images, tabular data) and the specific learning task, raw datasets undergo data preprocessing and data cleaning before being fed into a model. Common preprocessing steps include handling missing values, normalizing feature scales, encoding categorical variables, and removing noisy or corrupted records.

Once cleaned, datasets are partitioned to support the model lifecycle. The data is divided into training data used during model optimization, validation data set aside for hyperparameter tuning and model selection, and test data reserved exclusively for final evaluation.

Example.

For the loan-approval system, preprocessing might involve addressing missing data (e.g., imputing missing credit history values via interpolation), simplifying the feature space (e.g., grouping occupations in broader categories like “physician” rather than encoding detailed specialties), or normalizing continuous measurements (e.g., scaling income so it lies on a 0-to-1 scale). If a resulting data set included 1,000 examples (e.g., data collected from 1,000 people), 600 might be allocated for training, 100 as a validation set during training, and 300 for postdevelopment testing.

Subsection 4.4.2 Data Splitting Strategies

Deciding how to split data into training and validation sets is a crucial step. It ensures the model gets enough varied examples to learn from, while leaving aside enough fair, unbiased data to accurately measure its performance. Different strategies balance randomness, fairness, and coverage in different ways:

Cross-Validation (CV): Imagine dividing your dataset into equal groups (called folds). The model trains on all groups except one, tests its accuracy on the remaining group, and repeats this process until every single group has had a turn acting as the test set. A special variation where every individual data point gets its own turn as a test set is called leave-one-out cross-validation (LOO-CV).
Bootstrap and Monte-Carlo Cross-Validation (MCCV): These methods work like drawing names out of a hat. In Bootstrap, after a data point’s name is drawn for the training set, it is placed back into the hat, meaning the same example can be chosen multiple times (typically resulting in roughly 63% of the data used for training and 37% for validation). Monte-Carlo Cross-Validation works similarly, but names are drawn without putting them back, ensuring each data point is only picked once per round.
Bootstrapped Latin Partition (BLP): This approach combines randomness with careful organization. It shuffles data within specific categories so that every category is evenly represented in both training and testing, while guaranteeing every sample gets tested exactly once.
Kennard-Stone (K-S) and SPXY Algorithms: Instead of picking samples randomly, these smart selection methods measure the "distance" between data points to deliberately pick the most diverse, wide-ranging examples for training. This ensures the model learns from a well-rounded dataset rather than accidentally missing important edge cases.

Subsection 4.4.3 Model Building and Training

When performing sample classification across complex datasets, most classification models possess one or more model parameters that control the overall complexity of the model. While higher model complexity provides greater discriminating power, it also increases the risk of overfitting.

Overfitting occurs when a trained model performs extremely well on the specific samples used during training but performs poorly on new, unknown samples—meaning the model fails to generalize effectively. To find an optimal set of parameters that achieves a balance between model complexity and generalizability, it is essential to partition the available data into distinct training and validation sets.

Example.

The development team would first instantiate a specific model architecture and define its objective function. During the optimization process, the model attempts to learn a function that maps inputs (e.g., income, occupation, credit history) to the target output (e.g., whether the applicant paid back a previous loan).

Subsection 4.4.4 Model Validation and Data Splitting

The training set is used to build the model across multiple parameter settings, after which each trained model configuration is evaluated against the validation set. The validation set contains samples with known ground truth, but these labels are withheld from the model during testing. Predictions on the validation set allow developers to assess model accuracy and select optimal parameters based on the lowest validation error—a procedure known as model selection.

Historically, it was assumed that performance measured on a validation set served as an unbiased estimator of general performance. However, recent studies demonstrate that validation performance—including estimates derived from cross-validation or single train-test splits—can often yield over-optimistic or erroneous estimates of model accuracy.

Furthermore, validation procedures are susceptible to evaluation bias. This occurs when the validation dataset or chosen benchmark metrics fail to accurately represent the target population or do not adequately capture performance disparities across sensitive demographic subgroups.

Example.

The team might train a number of candidate models with varying hyperparameter settings or architectures (e.g., decision trees vs. neural networks). They evaluate these models based on validation error to choose the best configuration, reserving a completely separate test set for final performance reporting.

Subsection 4.4.5 Testing on Unseen Data

After the optimal model is selected and tuned, its true generalization capability must be evaluated using an independent, blind test set (often called a holdout set). Crucially, the test data is strictly isolated and never touched during the feature selection, model training, or hyperparameter optimization phases. Using test data prior to this final evaluation risks data leakage, which leads to over-optimistic performance claims that fail when deployed in real-world environments.

To ensure a model works reliably beyond the specific environment where it was built, researchers often evaluate models against external benchmark datasets—standardized datasets used across the field to test and compare different algorithms under identical conditions.

This process, known as external validation, allows direct comparison against existing methods in the literature and verifies whether the model’s predictive power holds across different collection environments, instruments, or demographic groups.

However, even when evaluating on a blind test set, it remains impossible to know with complete mathematical certainty how well the measured test performance matches the true underlying distribution of the entire real-world population. In real-world applications, the true population distribution is inherently unknown. Operators must rely on the statistical assumption that the blind test set serves as an unbiased, accurate estimator of model performance for all future, unseen samples drawn from that same distribution.

While sampling an entire population is impossible in practice, proper resampling strategies and sufficient sample sizes help approximate the central limit theorem, giving confidence that the observed test performance reflects underlying reality rather than random chance. Ultimately, the estimated test performance depends heavily on the chosen metrics (e.g., accuracy, precision, recall, or root-mean-square error), the degree of overlap between datasets, and how effectively the original data was partitioned.

Example.

In the loan approval model, the development team reserves a final 15% holdout set of loan applications that was completely hidden during training and hyperparameter tuning. Evaluating the finalized model on this blind test set provides an unbiased estimate of how accurately and fairly the system will evaluate real-world applicants once live.

Subsection 4.4.6 Model Postprocessing

Once a model has been trained, various model postprocessing steps that may be needed before its predictions can be used in real-world applications. For example, if a model performing binary classification outputs a raw probability score, but the system requires a discrete, categorical answer, developers must select appropriate classification thresholds to convert continuous outputs into hard decisions.

Example.

The resulting model for predicting loan approval likely outputs a continuous score between 0 and 1. The team might choose to transform this score into discrete buckets (e.g., low risk of defaulting, unsure, high risk of defaulting) or a binary recommendation (e.g., should/should not receive a loan).

Checkpoint 4.4.1. Model Deployment Performance Drop.

A data science team is developing a model to predict whether a patient will be readmitted to the hospital within 30 days of discharge. The team has collected a dataset of 10,000 patient records and preprocesses the data. They split the data into training, validation, and test sets. During development, they use the validation set to compare multiple model architectures and select the best one. After selecting the final model, they evaluate it on the test set and achieve 92% accuracy. The team then deploys the model, but it performs significantly worse on new patients, achieving only 78% accuracy.

Which of the following best explains the most likely cause of this performance drop?

The team used the test set during model selection, causing data leakage and over-optimistic performance estimates.
The team used the validation set for model selection, not the test set. If the test set had been used during model selection, the 92% accuracy would indeed be over-optimistic, but that is not what happened in this scenario.
The team used the validation set to select the model, but the test set was not representative of the target population.
Correct! The validation set was used appropriately for model selection, and the test set gave 92% accuracy. However, the drop to 78% on new patients suggests the test set did not accurately represent the broader target population. Even with proper data splitting, if the test set differs from the real-world distribution, performance will not generalize.
The team used cross-validation instead of a simple train-test split, which overfitted the model to the training data.
Cross-validation typically reduces overfitting by training on multiple folds and averaging performance, providing more reliable estimates than a single split. The team used a validation set approach, not cross-validation.
The team used the validation set to select the model, but the test set should have been used for model selection instead.
Using the test set for model selection would defeat its purpose as a blind evaluation set. The test set must remain untouched during model development to provide an unbiased estimate of real-world performance.

You have attempted of activities on this page.

Prev Top Next