Training Language Models That Can Continue to Learn
July 13, 2026The usefulness of a pretrained model depends not only on what it has learned, but also on its plasticity, i.e., its ability to keep learning. However, despite its importance for downstream adaptation, model plasticity remains understudied in pretraining. We find that using stronger weight decay during pretraining improves model plasticity. Contrary to common practice, minimizing pretraining validation loss does not necessarily produce the most plastic model, i.e., the model that performs best after further training. We also find potential explanations for why weight decay improves plasticity, including more structured internal representations, simpler attention patterns, and reduced overfitting.
Intro
Modern language models are built in two broad stages. First, a base model is pretrained on a large corpus of text. Then, it is adapted during post-training, undergoing further training using techniques such as supervised fine-tuning, alignment, reinforcement learning. In this sense, the pretrained model is not an end in itself; it is a starting point. However, many research studies and training practices treat the pretrained model as if its downstream quality can be summarized by its validation loss under the assumption that the lower this loss, the better the model will perform after subsequent training (i.e., perform better downstream). Following this assumption, it’s common to choose the pretraining model checkpoint or hyperparameter setting that minimizes the loss and carry out subsequent training using this model. In new research (Weight Decay Improves Language Model Plasticity, ICML 2026), we show how this reasoning can be flawed.
We study the plasticity of pretrained language models, that is, the ability of the model to adapt during subsequent training. A good pretrained model should not merely be good at predicting tokens from its pretraining distribution; it should also be plastic, i.e., able to adapt to new tasks, new objectives, and new constraints. Plasticity is especially important for pretrained models because, in modern language model development, these models almost always undergo additional training (e.g., post-training).
Weight decay during pretraining improves language model plasticity
Weight decay is a key regularization parameter during pretraining. In classical multi-epoch training, it was understood primarily as a regularizer that improves generalization. In contemporary large-scale pretraining, which often involves a single epoch over massive datasets, weight decay has also been shown to play a decisive role in optimization stability and convergence. In this work, we ask: What role does weight decay play in preserving or improving model plasticity? We find that models pretrained with stronger weight decay have higher plasticity, adapting better during subsequent training and achieving better final performance on the downstream task.
In one experiment, we pretrain a diverse set of language models (various model families, sizes, and tokens-per-parameter ratios) with different weight decay values. Then, we fine-tune these models on various chain-of-thought tasks (MetaMathQA, MedMCQA, PubMedQA, MMLUProCoT, RACE, and SimpleScaling). We find that models pretrained with higher weight decay perform better after fine-tuning, suggesting that they are better able to learn from and adapt to new data—that is, they are more plastic. Although a weight decay value of 0.1 is the common default in language model pretraining, we find that larger values can be better when downstream adaptability is the goal. The results are shown in Figure 1 below.
We test the robustness of this finding across a wide range of settings, including the six chain-of-thought reasoning tasks discussed above (Figure 1), five language understanding and commonsense reasoning tasks, one safety-alignment task, and numerous pretraining and fine-tuning hyperparameter configurations. In all cases, we find consistent evidence that weight decay improves model plasticity.
Better pretraining performance does not necessarily imply better downstream performance
We also examine the validity of the common assumption about model performance across stages of training and ask: Do models that perform best after pretraining also perform best after subsequent training? We find that this relationship is not guaranteed: models with similar pretraining losses can perform differently downstream, and models with lower pretraining losses can perform better or worse downstream than models with higher pretraining losses.
Thus, counterintuitively and contrary to popular assumption, models that perform worse after pretraining can sometimes achieve better performance after further training. This finding has important implications for model development: it suggests that optimizing for pretraining loss alone, a common practice, may not always lead to the best final model.
Why might weight decay improve model plasticity?
A natural next question is: why might weight decay improve model plasticity? To investigate this, we study how weight decay mechanistically shapes the pretrained models and highlight three key findings.
First, models pretrained with larger weight decay tend to learn representations that are more linearly separable. Linear probes trained on hidden representations achieve higher accuracy on sentiment and topic classification tasks, suggesting that information is encoded in a more accessible form.
Second, larger weight decay lowers the rank of attention matrices. This suggests that weight decay imposes a form of structural compression on attention, rather than merely shrinking weights in a generic way.
Third, larger weight decay reduces overfitting to the pretraining data. As weight decay increases, the train-validation gap decreases, suggesting that the model becomes less tightly fit to the pretraining distribution and remains more capable of adapting later.
Together, these findings point toward a richer interpretation of weight decay. It is not simply a penalty term; it shapes the geometry of representations, the structure of attention, and the degree of overfitting of the model. Each of these effects may help pretrained models learn new data more effectively, thereby improving plasticity.
Why this matters
This work offers several important insights for research and practice. First, pretraining should be evaluated as part of the full model training pipeline, not as an isolated stage. In doing so, plasticity emerges as an important (but previously understudied) characteristic of pretrained models because these models are meant to undergo subsequent training. Our results show that weight decay plays a key role in shaping this plasticity.
Second, the optimal hyperparameter setting depends on the objective. Different objectives can have different optimal settings, as observed when optimizing weight decay for pretraining versus downstream performance. This suggests that a single setting may not simultaneously satisfy multiple objectives, requiring weighing trade-offs and prioritizing objectives.
Third, performance at one stage of training may not reliably predict performance at a later stage. In particular, we find that models that perform worse after pretraining can sometimes perform better after further training. This suggests that optimizing only for an early training stage, a common practice, may not yield the best final model.
This work also raises new research questions. How does the plasticity benefit of larger weight decay scale to much larger models? Do similar effects appear in multimodal, vision-language, or reinforcement-learning-heavy foundation models? And can we develop proxy metrics for plasticity that do not require costly downstream fine-tuning sweeps?
Altogether, this work reveals a new role for weight decay: beyond regularization and optimization, it shapes a model’s future learnability. It also highlights plasticity as an important property of pretrained models. For researchers studying and practitioners building language models, these findings re-define the goal of pretraining: the best pretrained model is not necessarily the one that best learned the pretraining objective – it may be the one that can continue to learn.