HomeLearnCoursesHackathonsAccount
Time Series Forecasting
Why Random Splits Fail and How to Backtest Properly · 1/2

The problem with a random train/test split

Other courses on this site cover train/test splitting and cross-validation as standard practice, and for i.i.d. data, randomly holding out a subset of rows is exactly right. Applied to time series, it's exactly wrong. A random split can place a training example from next month next to a test example from last week, meaning the model is effectively being evaluated on the past after having seen the future during training. Even k-fold cross-validation, a workhorse technique elsewhere, suffers from the same flaw when applied naively to sequential data, because every fold still mixes past and future across the train/validation boundary.

The fix is to split chronologically: everything before some cutoff date is training data, everything after is test data, full stop. This mirrors the actual deployment condition, where a model only ever has access to the past when it's asked to predict the future. It's a simpler rule than cross-validation, but it's the one rule in this domain that cannot be relaxed without invalidating the entire evaluation.