Deeper Learning

The authors demonstrate, through theory and experiments, that in MoEs, experts improve memorization more than reasoning.

• Preprint

The authors introduce an updated framework for understanding visual object recognition and category selectivity: contrastive coding.

• Paper

Introducing PINNACLE, a novel contextual AI model for single-cell protein biology that supports a broad array of biomedicalAI tasks by tailoring outputs to the cell type context in which the model makes predictions.

• Paper
• Research Briefing
• Code
• Project website
• Demo

The authors perform a rigorous study of a variety of LLM training optimizers and find that they are all fairly similar except for SGD, which is notably worse.

• Paper

The authors theoretically and empirically demonstrate that generative models in chess can outperform the experts that train them by low-temperature sampling.

• Preprint
• Code
• Dataset
• Play with our models

Second part of a two-part blog post covering recent findings from the authors

By analyzing the dynamics of a random feature model trained with gradient descent, the authors reproduce and explain many observations about neural scaling laws, including compute optimal scaling of model size and training time.

• Paper
• Blog Post (Part 1)

Part 1 of a two-part blog post covering recent findings from the authors

A discussion of expository material and the authors' recent papers related to large width and depth limits.

• Paper
• Paper
• Paper

A new way to label different types of uncertainty in unconstrained text and simple methods to predict those labels, including a completely unsupervised approach.

• Code
• Paper

Improved efficiency of State Space Models sacrifices some core capabilities for modern LLMs.

• Pre-print

The Matryoshka Way

Introducing an algorithmic method to elastically deploy large models: the #MatFormer.

• Code
• Paper