Knowledge cutoff

In machine learning, a knowledge cutoff (or data cutoff) is the point in time beyond which a large language model has not been trained on new data. Since large language models are pretrained, any model's knowledge is fixed at what it was trained on before deployment; information about events after this date is absent from the model's training data. The model cannot access information about later events without a system for real-time data access such as retrieval-augmented generation, which fetches new information from an external database.

Knowledge cutoffs can introduce limitations like hallucinations, where the model generates confident (but false) statements, information gaps, and reduced accuracy on evolving knowledge. Research has shown that knowledge cutoffs have safety-critical implications in domains such as healthcare, where outdated knowledge can lead to harmful recommendations. In the clinical domain, models with a later knowledge cutoff had greater accuracy on questions reflecting newer guidelines.

Description

A large language model is trained before deployment on static snapshots of data collected from the internet, books, and other sources up to a specific knowledge-cutoff date. During training, an LLM can learn linguistic patterns, semantics, and contextual meanings. The model is then trained to estimate the probability of each candidate word that could follow a given sentence, and it uses these probabilities to generate text. Because its knowledge is frozen at the point of training, a model with a fixed knowledge cutoff cannot provide information about facts or developments that emerged later from its pretrained knowledge alone. Some models can retrieve newer information through search tools, but their judgment remains shaped by the data they were trained on. Prompts about developments after the cutoff may therefore receive incomplete, outdated, or incorrect answers. Training the most advanced AI systems has become increasingly expensive; a 2024 study by researchers at Stanford University and Epoch AI projected that the cost of training the largest AI systems would reach the billions by 2027.

The GPT-4 model has a knowledge cutoff of September 2021, the GPT-4 Turbo model has a knowledge cutoff of December 2023, and the GPT-5 model has a knowledge cutoff of September 2024. GPT-OSS-120B has a knowledge cutoff of May 2024, and Llama 4 Maverick has a knowledge cutoff of August 2024.

Effects

Information gaps

Knowledge cutoffs create information gaps. LLMs lack knowledge of events or discoveries not included in their training data. Because these models are trained to generate the most probable sequence of words based on their training patterns, they may produce confident but incorrect outputs when queried beyond that information. A study by Cacioli et al. created a 363-question benchmark based on two versions of the Infectious Diseases Society of America's COVID-19 treatment guidelines. Models whose knowledge cutoffs predated the newer guideline, such as GPT-3.5-Turbo and Llama-2, performed worse on these questions (76.03 percent and 35.26 percent, respectively), while models with later cutoffs, such as GPT-4o and Llama 3.3, had over 90-percent accuracy. The study concluded that the recency of information must be treated as a safety-critical attribute on a par with alignment or interpretability, highlighting the safety concerns of knowledge cutoffs in clinical decision-making.

Effective vs. reported dates

A study by Pęzik et al. at the University of Łódź indicated that a model's knowledge does not necessarily match its cutoff date. The effective cutoff, the date up to which it can reliably know information, often differs by subject and is influenced by the distribution of information within the training data; some topics may reflect later knowledge than others, and knowledge that predates the cutoff may be absent. Training data has uneven information across topics. Due to the high cost of retraining large language models, the models are rarely retrained to extend their knowledge cutoff. Some models can use integrated search tools to access more recent information, which makes it unclear if an answer comes from the model's original training or from a live search; GPT-4 can access its search tool and provide real-time information.

Mitigation strategies

Retrieval-augmented generation

Main article: Retrieval-augmented generation

Retrieval-augmented generation augments a large language model with updated data from external sources, allowing it to generate better-informed responses. In RAG, the language model is connected to an external knowledge base or search engine to retrieve live data. This allows the model to find current information relevant to a query and incorporate it into its response, with citations. Grounding a model in external data which ties a model's answers to its retrieved sources helps reduce the frequency of hallucinations and improves output accuracy. The external knowledge base might be outdated or contain biases, however, which may lead to incorrect information (or hallucinations). Google's AI Overviews has made false claims and provided unreliable results, since the model may misinterpret the sources it fetched or fail to retrieve high-quality sources. Retrieval alone cannot fully compensate for an outdated knowledge cutoff; the model's fundamental understanding is rooted in its training data. This can be mitigated with techniques such as reinforcement learning from human feedback, which aligns an AI model with human preferences; this technique can enhance the quality and reliability of a large language model's responses.

Incremental learning

Main article: Incremental learning

Another approach is incremental learning (or continual learning), a method of machine learning in which new data is continuously used to extend the existing model's knowledge. It aims to prevent catastrophic forgetting, where AI abruptly forgets what it has learned. In practice, however, it often fails to prevent catastrophic forgetting completely. This technique allows efficient, incremental updates to a model without the cost of a full retraining cycle. One continual-learning technique is fine-tuning, which allows AI labs to precisely adjust a model's behavior. A more efficient method of fine-tuning involves methods such as Low-Rank Adaptation. This does not provide real-time awareness, however, since adding modules to the system may result in catastrophic forgetting as the weights in the model become biased in favor of the new data set.