Analysis · Technology
When an AI Test Set Secretly Contains Its Training Data
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
A hospital can contribute images to an AI training project and later supply those same images to a shared testing collection. The resulting overlap may be hidden. An apparently independent evaluation could then overestimate how well the system handles unfamiliar cases.
Sarah Hickman, Gabrielle Baxter and Fiona Gilbert highlighted this risk in their 2021 review of AI adoption in breast imaging. They examined published research and implementation challenges, rather than conducting a new clinical experiment. Their methodological warning was specific: knowing which institution supplied a dataset is insufficient without knowing how its images have already been used.
The authors distinguished two kinds of independence. Testing on previously unseen data checks whether an algorithm transfers beyond its development material. Testing by an independent institution can reduce evaluator bias and support comparisons between products. A credible assessment needs to consider both; a different evaluator does not automatically guarantee unfamiliar data.
Their proposed response was to track data back to its source and record its subsequent uses. They described this as a needed capability, with substantial implementation difficulties, especially for data already processed. The review did not measure how frequently concealed overlap occurred or demonstrate that a particular tracking system solved it.
Even a clean test set leaves another gap. Retrospective image analysis cannot establish how AI changes clinicians' decisions during routine work. The review therefore called for prospective evaluation and continued monitoring, including cancer detection, recall rates and reading time. Strong classification performance alone does not establish improved patient outcomes or cost savings.
For teams evaluating breast imaging AI, the practical lesson is to examine the history behind the benchmark: where the images came from, whether they were previously used, and who conducted the evaluation. That scrutiny strengthens the evidence for a deployment decision, while leaving real workflow benefits to be tested.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Adoption of artificial intelligence in breast imaging: evaluation, ethical constraints and limitations Sarah Hickman, Gabrielle Baxter, Fiona J. Gilbert
