Draft:Data contamination in machine learning
 | Draft article not currently submitted for review.
This is a draft Articles for creation (AfC) submission. It is not currently pending review. While there are no deadlines, abandoned drafts may be deleted after six months. To edit or make changes to this draft, simply click on the "Edit" tab at the top of the window. To be accepted, a draft should:
It is strongly discouraged to write about either yourself or your business or employer. If you do so, you must declare it. Where to get help
How to improve a draft
You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Editor resources
Last edited by Bearcat (talk | contribs) 4 days ago. (Update) |
Data contamination in machine learning (also referred to as benchmark leakage or data exposure) occurs when test or evaluation data is included, directly or indirectly, in the training set of a machine learning model.[1] This overlap compromises the validity of evaluation metrics, leading to inflated performance estimates that do not reflect a model's true capability for generalization on unseen inputs.[2]
While closely related to traditional overfitting, data contamination specifically concerns the unintended memorization and retrieval of evaluation datasets during the training process.[3] The phenomenon has become a major methodological challenge in evaluating large language models (LLMs) trained on massive web-scale corpora.[4]
Background
[edit]In standard machine learning paradigms, datasets are partitioned into distinct subset splits: training, validation, and test sets. The fundamental premise of evaluation is that performance on the test set serves as a proxy for out-of-distribution performance.[5]
With the advent of self-supervised learning on uncurated internet datasets (such as Common Crawl), automated web-scraping pipelines frequently ingest publicly accessible evaluation datasets—including coding benchmarks like HumanEval, standardized academic tests, and reasoning tasks.[6] As a result, models may appear to demonstrate reasoning or problem-solving capabilities when they are instead executing verbatim or near-verbatim retrieval from memory.[7]
Types of contamination
[edit]Researchers categorize data contamination into several distinct forms based on how and when the overlap occurs:
- Direct verbatim contamination: Test instances or exact task targets are present word-for-word within the training corpus.[1]
- Paraphrased or near-duplicate contamination: Test instances exist in the training corpus in slightly modified, rephrased, or translated forms, circumventing simple string-matching filters.[6]
- Input-only leakage: The prompt or problem statement is included in the training data without the ground-truth answers. Studies suggest this can still inadvertently lower task perplexity and boost model accuracy.[3]
- Data recycling (Data flywheel effect): Output generated by synthetic dataset pipelines or earlier model generations incorporates test benchmarks, which are subsequently scraped and reused in training future model iterations.[8]
Detection techniques
[edit]Detecting contamination in proprietary or closed-source models presents technical hurdles because researchers often lack direct access to the training dataset.[4] Several methodology frameworks have been developed to measure contamination retroactively:
$N$-gram and substring matching
[edit]The most direct form of detection involves computing $n$-gram overlaps (typically $n \ge 13$) or suffix array matches between the evaluation benchmark and the training corpus.[3] However, this method requires full visibility into the training data.
Perplexity and log-likelihood analysis
[edit]By comparing model loss or perplexity on clean reference samples versus potentially contaminated evaluation samples, researchers can identify statistical anomalies. A model exhibiting disproportionately low loss on specific benchmark test cases relative to synthetically modified variants often indicates previous exposure.[9]
Dynamic and perturbation-based benchmarks
[edit]To circumvent static contamination, evaluators create dynamic benchmarks that alter variable names, numerical values, or underlying logical structures at inference time. If a model's performance drops sharply on functionally identical problem variants, contamination of the original benchmark is inferred.[10]
Mitigation and best practices
[edit]Proposed solutions for addressing data contamination include both proactive dataset curation and revised benchmark design:
- Canary strings: Inserting explicit, unique text markers (such as cryptographic hashes) into evaluation datasets. Data scrapers can search for these strings and exclude matching documents prior to training.[3]
- Clean data decontamination pipelines: Applying string overlap filters during the data preprocessing phase to scrub known evaluation benchmarks from raw scrape dumps.[4]
- Encrypted or private benchmarks: Maintaining benchmark evaluation datasets behind password-protected repositories or APIs rather than hosting them as open-access web pages.[2]
- Continuous benchmark refresh: Developing regularly updated, time-stamped datasets (e.g., news summaries or recent competitive programming problems) created strictly after a model's training data cutoff date.[6]
See also
[edit]References
[edit]- 1 2 Zhou, Kun; Yao, Yuxiang; Wang, Xing; Xu, Ying (2023). "Don't Make Your Classifier Weak: Investigating Data Contamination in Language Model Evaluation". Transactions of the Association for Computational Linguistics. 11: 1234–1250. doi:10.1162/tacl_a_00582.
- 1 2 Magar, Ines; Schwartz, Roy (2022). "Data Contamination: From Oxford to LLMs". Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. pp. 1001–1015. doi:10.18653/v1/2022.acl-long.72.
- 1 2 3 4 Scao, Tevven Le; Rush, Alexander M. (2021). "How Many Data Points Do You Need? Data Contamination in Large Language Models". Journal of Machine Learning Research. 22 (150): 1–28.
- 1 2 3 OpenAI (2023). GPT-4 Technical Report (Report). arXiv. arXiv:2303.08774.
- ↑ Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. ISBN 978-0262035613.
- 1 2 3 Jacovi, Alon; Caciularu, Avi; Goldberg, Yoav (2023). "Stop Pretraining on the Test Set: Detecting Data Contamination in Language Models". Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 4512–4528.
- ↑ Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. pp. 610–623. doi:10.1145/3442188.3445922.
- ↑ Shumailov, Ilia; Shumaylov, Zakhar; Zhao, Yiren; Gal, Yarin (2024). "The Curse of Recursion: Training on Generated Data Makes Models Forget". Nature. 631: 755–759. doi:10.1038/s41586-024-07566-y.
- ↑ Oren, Yonas; Nicole, Eran; Hashimoto, Tatsunori (2023). "Proving Test Set Contamination in Black-Box Models". Advances in Neural Information Processing Systems. Vol. 36. pp. 18901–18915.
- ↑ Shi, Weijia; Ajith, Anirudh; Xia, Mengzhou (2024). "Detecting Pretraining Data Contamination in Language Models via Min-K% Prob". International Conference on Learning Representations (ICLR).