Kempner Institute Co-Director’s ICML Plenary Showcases Broad Science-of-Pretraining Effort
Sham Kakade, in his invited talk, highlights a line of work showing that a remarkably simple model can accurately forecast the outcomes of pretraining
Kempner Institute Co-Director Sham Kakade will present a synthesis of the Kempner's sustained science-of-pretraining efforts at an invited plenary talk at the 2026 International Conference on Machine Learning (ICML).
Allston, MA— How much can a simplified quadratic model actually reveal about large-scale deep-learning optimization behavior in the pretraining of large language models (LLMs)?
Quite a lot, says Kempner Institute Co-Director Sham Kakade who will present a synthesis of the Kempner Institute’s sustained science-of-pretraining efforts at an invited plenary talk at the 2026 International Conference on Machine Learning (ICML) in Seoul, South Korea, on July 8.
The talk, entitled “How Far Can Quadratics Take Us? Lessons for LLM Pretraining,” highlights several student- and postdoc-led projects from the Kempner that together support the case that quadratics deserve a more central place in how we think about and study LLM pretraining using optimization algorithms, or optimizers.
To make the case for using quadratic models to understand pretraining behavior, Kakade points to recent work from the Kempner on quadratics in discrete time. In this research, Kempner scientists find that a simple, analytically-solvable quadratic model sharply predicts real large-scale optimization behavior, including critical batch size, learning-rate effects, batch-size effects, and momentum’s role.
Notably, the quadratic model is able to make quantitative forecasts, rather than simply qualitative ones.
“Quadratics allow for a very fine-grained analysis of training dynamics,” said Alex Meterez, a Kempner graduate fellow who collaborated on the research. “This makes them a useful sandbox for pretraining, where we have consistently found a remarkable agreement between theoretical predictions and experiments. Moreover, the special structure present in these particular quadratics can be further used to develop better algorithms.”
Beyond their quantitative forecasting power, the researchers found that the quadratic predictions continued to hold at scale. Tested at pretraining scale on the Kempner’s AI cluster, the researchers were able to validate their findings as both model and dataset size increased.
These findings have several important practical implications, says Kakade. They show how to scale pretraining in a principled way, based on the dynamics of the optimization algorithm itself. In particular, the research explains how to train a model so that the number of tokens is not tightly tied to model size, and how to perform “horizon-free” pretraining that doesn’t require choosing a fixed end time in advance. It also provides insight into what momentum actually helps with in the pretraining process, and what it does not.
“The quadratic model is useful because it tells you exactly what to measure and what behaviors to look for, so its predictions can be tested directly against real training runs,” explains Alex Damian, former Kempner Research Fellow and current assistant professor at MIT, who collaborated on this work. “For example, using the Kempner cluster, we were able to estimate a key quantity, the Hessian spectrum, throughout training, which ties into fundamental questions like training stability and the origin of scaling laws.”
According to Kakade, these results, which show how quadratic models can guide principled pretraining based on optimizer dynamics, complement ongoing work from the Kempner related to principled pretraining based on model dimensions. That work on width and depth scaling, led by Kempner associate faculty member Cengiz Pehlevan, shows how to scale pretraining in a principled way as model dimensions grow.
“This new picture of optimization dynamics emerging from the Kempner Institute sits naturally alongside the pretraining-scaling work I’ve been doing in my group” explains Pehlevan. “One tells you how to transfer best hyperparameter choices across model dimensions; the other tells you how those best choices emerge from the dynamics of training. You want both.”
According to Kakade, quadratics might be simple in structure, but their potential predictive power could shed light on some of machine learning’s biggest pretraining challenges.
“For a long time optimization theory has given us bounds — reassuring, but rarely predictive,” explains Kakade. “What’s exciting here is that a very simple model turns out to forecast pretraining: it tells you which run will win, and by how much.”
“That’s the kind of theory that can actually change how people train models.”
About the Kempner
The Kempner Institute seeks to understand the basis of intelligence in natural and artificial systems by recruiting and training future generations of researchers to study intelligence from biological, cognitive, engineering, and computational perspectives. Its bold premise is that the fields of natural and artificial intelligence are intimately interconnected; the next generation of artificial intelligence (AI) will require the same principles that our brains use for fast, flexible natural reasoning, and understanding how our brains compute and reason can be elucidated by theories developed for AI. Join the Kempner mailing list to learn more, and to receive updates and news.
