Cross-validation estimates how well a model may perform on data it has not seen by repeatedly training it on some observations and evaluating it on others. Choose a splitting method that reflects how new data will arrive, keep every learned preprocessing step inside each training fold, and reserve a final test set when you need an independent check.
What cross-validation measures
A model can fit its training data well yet perform poorly on new cases. Measuring performance on the same observations used to fit the model gives an overly optimistic picture, because the model has already had an opportunity to adapt to them. Cross-validation addresses this by dividing available data into training and validation portions several times.
In a common procedure called k-fold cross-validation, the data is divided into k roughly equal folds. The model is trained on all but one fold, then evaluated on the held-out fold. This repeats until each fold has served as validation data. The reported score is usually an average of the fold scores, sometimes accompanied by their spread.
Here, “validation” means data used to estimate performance during development. It is distinct from a final test set, which should remain untouched until choices such as model type and settings are settled. Cross-validation does not guarantee future performance; it provides an estimate whose usefulness depends on whether the split resembles the real prediction task.
How to choose a splitting strategy
Start by defining the prediction you will make. Will a model predict outcomes for entirely new people, later dates, new locations, or new measurements from people it has already encountered? The answer determines what must be kept separate. A random split is appropriate only when observations can reasonably be treated as independent and drawn from the same underlying process.
- Ordinary k-fold: A useful default for independent observations in a regression task. Shuffle before splitting when row order has no meaning.
- Stratified folds: For classification, these aim to preserve class proportions in each fold. This can make comparisons more stable when a class is uncommon, provided there are enough examples to distribute.
- Grouped folds: Keep all observations from the same person, household, device, site, or other relevant group in one fold. This estimates performance on groups not represented in training.
- Time-based splits: Train on earlier observations and validate on later ones when the intended task is forecasting or predicting future outcomes.
- Repeated splits: Repeating a suitable procedure with different partitions can show how sensitive results are to the particular split. It does not create new independent data.
These choices are not interchangeable. Suppose a dataset contains multiple records per patient. If the goal is to predict for new patients, a random row split can place one patient’s records in both training and validation. A grouped split is more appropriate. If the goal is instead to predict a later visit for a patient whose earlier visits are already available, a chronological split may better match the use case.
Stratification, groups, and time
Stratification is a way to balance class representation, not a cure for every data problem. It does not prevent related records from crossing folds, fix biased sampling, or make a tiny dataset adequate. If a class has too few examples, some folds may not contain it at all; the evaluation plan may need fewer folds, a different metric, or more data.
Grouping matters whenever observations share information that could make them unusually similar. Examples include repeated measurements, several documents by one author, or samples collected at the same location. Decide what counts as a group from the data-generating process, not simply from whichever column is easiest to use. If groups are the intended unit of future prediction, group boundaries should be respected in validation.
For time-dependent data, random shuffling can let the model learn from the future while being evaluated on the past. Use chronological ordering and ensure each validation period follows its training period. Some forecasting tasks also require a gap between training and validation to account for delayed labels, overlapping target windows, or information that would not yet be available at prediction time. The necessary gap depends on the task and should be justified rather than chosen mechanically.
Preventing leakage
Data leakage occurs when information unavailable at prediction time influences model fitting or evaluation. It can produce convincing scores that fail in deployment. Leakage may enter through the features themselves, the way data is split, or a preprocessing operation performed before cross-validation.
For example, suppose missing values are filled using the average of the entire dataset before the folds are made. Each validation fold then influences a transformation applied to its own values. The same problem can arise when scaling features, selecting features, reducing dimensions, or choosing a decision threshold using information from validation observations.
The safe pattern is to fit every data-dependent operation on the training portion of each fold and then apply that fitted operation to the held-out portion. A preprocessing pipeline makes this sequence explicit:
- Split the observations into training and validation folds.
- Fit imputation, scaling, encoding, feature selection, and other learned transformations using only the training fold.
- Transform the training fold and the held-out fold with those fitted operations.
- Fit the predictive model on transformed training data and score it on transformed validation data.
Some transformations are fixed rules and do not learn from the dataset, but many operations that look routine do learn values or choices. Treat them as part of model fitting unless you can establish that they use no information from the validation observations.
Feature timing and target leakage
A feature can be technically present in a dataset and still be invalid for prediction. Ask whether it would be known at the moment the model is meant to make its prediction. A field recorded after an outcome, a status updated in response to that outcome, or an aggregate calculated using future records may reveal the answer indirectly. This is often called target leakage.
Leakage can also result from duplicate or near-duplicate records crossing folds, labels derived from overlapping time windows, or a feature that encodes a group-specific outcome. Inspect how each feature was created and when it becomes available. When in doubt, recreate the prediction setting: freeze information at the intended prediction time, then split the resulting examples in a way that matches future use.
Preprocessing is not the only source of leakage. Repeatedly trying models and settings against the same validation results can gradually tune choices to that validation data. Cross-validation reduces dependence on one arbitrary holdout, but it does not make unlimited experimentation harmless. Keep a final test set for a last evaluation, or use nested cross-validation when the data is limited and a less biased estimate after model selection is important.
Metrics and interpreting results
Choose a metric that reflects the actual goal. Accuracy can hide poor performance on a rare class; precision, recall, a confusion matrix, or a ranking metric may be more informative depending on the consequences of different errors. For regression, an average error measure may be useful, but inspect whether it penalizes large errors in a way that matches the task.
Compute the metric separately on each held-out fold, then summarize across folds. An average is easy to interpret, while fold-to-fold variation signals sensitivity to the split or to the composition of the data. Do not treat that variation as a formal confidence interval without suitable assumptions. Folds share training observations, so their scores are not independent measurements.
When fold sizes differ or the metric is nonlinear, averaging fold scores and pooling predictions can produce different summaries. State which method you use and why. For an imbalanced classification task, for instance, pooling all out-of-fold predictions can provide a useful overall view, while per-fold scores reveal whether results depend heavily on a particular partition.
Cross-validation and the final test set
Cross-validation is often used to compare candidate models or tune settings. Once the chosen procedure is settled, refit it on all data available for development. If a separate test set was held back, evaluate on it once for the final report. Do not move the test set into routine model selection after seeing its results; doing so turns it into another validation resource.
For small datasets, nested cross-validation can separate tuning from evaluation. In each outer split, an inner cross-validation process chooses settings using only the outer training data. The selected model is then evaluated on the outer held-out fold. This is more computationally demanding, but it helps account for the optimism introduced when many settings are compared on the same validation results.
There is no single best split strategy for every dataset. The practical objective is to simulate the boundary between information available for training and information that will be new at prediction time. A simple, well-matched split is more informative than a sophisticated procedure that answers the wrong question.
Common mistakes and a practical checklist
Before trusting a score, check the evaluation design as carefully as the model. A high score is not persuasive if the split permits information to cross the boundary or if the metric does not represent the real task.
- Write down what a future prediction concerns: a new row, person, group, location, or time period.
- Check for repeated entities, duplicates, chronological structure, and features created after the target event.
- Choose random, stratified, grouped, or time-aware folds to match that structure.
- Place learned preprocessing and feature selection inside the fold-level training pipeline.
- Select metrics before comparing models, and inspect both average performance and variation.
- Keep a final test set untouched when an independent final check is needed.
A common misconception is that more folds always produce a better estimate. More folds change the size of each training and validation portion and increase computation; they do not fix a mismatched split, leakage, or unrepresentative data. Choose a fold count that is feasible and compatible with the number of examples, groups, and classes, then explain the choice.
Frequently Asked Questions
Should I use cross-validation or a train-test split?
A train-test split is simple and can be suitable when there is enough data and the split reflects the intended use. Cross-validation evaluates multiple partitions and is useful for model comparison or limited data. Either method can mislead if the split ignores groups, time, or leakage.
Should I shuffle before cross-validation?
Shuffle independent observations when their order is arbitrary and a random split matches the task. Do not shuffle time-ordered data when predicting the future, and do not rely on shuffling alone when related observations must stay together.
Can I scale or impute data before splitting?
Usually, fit scaling and imputation separately within each training fold, then apply the fitted transformations to that fold’s validation data. Fitting them on the full dataset lets validation observations influence the evaluation.
What is the difference between cross-validation and a test set?
Cross-validation is generally part of development: it helps estimate performance and compare choices. A final test set is held back for an independent evaluation after those choices are made. Repeatedly using the test results for tuning weakens its independence.
Is a high cross-validation score proof that a model will work?
No. It is an estimate for a particular dataset, split design, metric, and prediction setting. Distribution changes, biased data, leakage, or differences between the evaluation and real use can all reduce future performance.