Deeper Learning

The authors perform a rigorous study of a variety of LLM training optimizers and find that they are all fairly similar except for SGD, which is notably worse.

• Paper

The authors theoretically and empirically demonstrate that generative models in chess can outperform the experts that train them by low-temperature sampling.

• Preprint
• Code
• Dataset
• Play with our models

Second part of a two-part blog post covering recent findings from the authors

By analyzing the dynamics of a random feature model trained with gradient descent, the authors reproduce and explain many observations about neural scaling laws, including compute optimal scaling of model size and training time.

• Paper
• Blog Post (Part 1)

Part 1 of a two-part blog post covering recent findings from the authors

A discussion of expository material and the authors' recent papers related to large width and depth limits.

• Paper
• Paper
• Paper

A new way to label different types of uncertainty in unconstrained text and simple methods to predict those labels, including a completely unsupervised approach.

• Code
• Paper

Improved efficiency of State Space Models sacrifices some core capabilities for modern LLMs.

• Pre-print

The Matryoshka Way

Introducing an algorithmic method to elastically deploy large models: the #MatFormer.

• Code
• Paper

A story of sinusoids and inductive biases

Mechanic interpretability results explained using known inductive biases.

• Pre-print

Robust watermarking in AI is impossible under natural assumptions.

• Paper