Deeper Learning

Alignment Reduces Conceptual Diversity of Language Models

By: Sonia Murthy, Tomer Ullman, and Jennifer Hu

The authors use a new way of measuring the conceptual diversity of synthetically-generated LLM “populations” to investigate whether LLMs capture the conceptual diversity of human populations.

ProCyon: A Multimodal Foundation Model for Protein Phenotypes

By: Owen Queen, Robert Calef, and Marinka Zitnik

The authors introduce ProCyon, a multimodal foundation model to model, generate, and predict protein phenotypes.

• Preprint
• GitHub
• Project website
• Dataset

Loss-to-Loss Prediction

Scaling Laws for all Datasets

By: David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach and Sham Kakade

The authors develop a method to predict how large language models scale with compute across different datasets, enabling more efficient training and better understanding of data-compute tradeoffs.

• Preprint
• Training code
• Notebooks
• Huggingface models
• TMLR referee report

How Does Critical Batch Size Scale in Pre-training? (Decoupling Data and Model Size)

By: Hanlin Zhang, Depen Morwani, Nikhil Vyas, Udaya Ghai, Jingfeng Wu and Difan Zou

The authors empirically show that the critical batch size for pre-training scales with data size rather than model size, followed by theoretical justifications for it.

• Preprint

Mixture of Parrots 🦜🦜🦜: Experts Improve Memorization More Than Reasoning

By: Samy Jelassi and Eran Malach

The authors demonstrate, through theory and experiments, that in MoEs, experts improve memorization more than reasoning.

• Preprint

Contrastive Learning Explains the Emergence and Function of Visual Category Selectivity

By: Jacob Prince, George Alvarez, and Talia Konkle

The authors introduce an updated framework for understanding visual object recognition and category selectivity: contrastive coding.

• Paper

Context Matters for Foundation Models in Biology

By: Michelle M. Li and Marinka Zitnik

Introducing PINNACLE, a novel contextual AI model for single-cell protein biology that supports a broad array of biomedicalAI tasks by tailoring outputs to the cell type context in which the model makes predictions.

• Paper
• Research Briefing
• Code
• Project website
• Demo

Anything but SGD: Evaluating Optimizers for LLM Training

By: Rosie Zhao, Depen Morwani, David Brandfonbrener and Nikhil Vyas

The authors perform a rigorous study of a variety of LLM training optimizers and find that they are all fairly similar except for SGD, which is notably worse.

• Paper

Transcendence: Generative Models Can Outperform the Experts That Train Them

By: Edwin Zhang and Eran Malach

The authors theoretically and empirically demonstrate that generative models in chess can outperform the experts that train them by low-temperature sampling.

• Preprint
• Code
• Dataset
• Play with our models

A Dynamical Model of Neural Scaling Laws

Second part of a two-part blog post covering recent findings from the authors

By: Blake Bordelon, Alex Atanasov, and Cengiz Pehlevan

By analyzing the dynamics of a random feature model trained with gradient descent, the authors reproduce and explain many observations about neural scaling laws, including compute optimal scaling of model size and training time.

• Paper
• Blog Post (Part 1)