Deeper Learning

The authors introduce ProCyon, a multimodal foundation model to model, generate, and predict protein phenotypes.

• Preprint
• GitHub
• Project website
• Dataset

Scaling Laws for all Datasets

The authors develop a method to predict how large language models scale with compute across different datasets, enabling more efficient training and better understanding of data-compute tradeoffs.

• Preprint
• Training code
• Notebooks
• Huggingface models

The authors empirically show that the critical batch size for pre-training scales with data size rather than model size, followed by theoretical justifications for it.

• Preprint

The authors demonstrate, through theory and experiments, that in MoEs, experts improve memorization more than reasoning.

• Preprint

The authors introduce an updated framework for understanding visual object recognition and category selectivity: contrastive coding.

• Paper

Introducing PINNACLE, a novel contextual AI model for single-cell protein biology that supports a broad array of biomedicalAI tasks by tailoring outputs to the cell type context in which the model makes predictions.

• Paper
• Research Briefing
• Code
• Project website
• Demo

The authors perform a rigorous study of a variety of LLM training optimizers and find that they are all fairly similar except for SGD, which is notably worse.

• Paper

The authors theoretically and empirically demonstrate that generative models in chess can outperform the experts that train them by low-temperature sampling.

• Preprint
• Code
• Dataset
• Play with our models

Second part of a two-part blog post covering recent findings from the authors

By analyzing the dynamics of a random feature model trained with gradient descent, the authors reproduce and explain many observations about neural scaling laws, including compute optimal scaling of model size and training time.

• Paper
• Blog Post (Part 1)

Part 1 of a two-part blog post covering recent findings from the authors

A discussion of expository material and the authors' recent papers related to large width and depth limits.

• Paper
• Paper
• Paper