Can an AI catch data leakage before you trust the score?

A new experiment plants four common leaks in real pipelines and asks Claude models to spot them.

A model that reports 99% accuracy is often a model with a leak. In a new experiment, Nate Rosidi built four versions of that problem using real datasets: a feature that defines the label, a feature built with future information, a random split that ignores relational structure, and preprocessing done in the wrong order. In the first case, using same-day closing prices pushed validation accuracy to 97%; using only prior-day data brought it back to 51.6%.

He then described each pipeline to four Claude models without saying a leak existed. All four caught the planted leak every time, including the fastest and cheapest model. The differences appeared in what else they noticed. In the delivery-duration case, only three of four flagged that shuffling a timestamped table before splitting let training rows fall after test rows.

The lesson is not that AI review replaces judgment. It is that a cheap, fast review can catch patterns that are easy to miss when a score looks good. The useful habit is to ask a model to inspect the pipeline before trusting the number, and to ask what it would expect an honest result to look like.