Deeper Learning

Traveling Waves Integrate Spatial Information Through Time

By: Mozes Jacobs, Roberto Budzinski, Lyle Muller, Demba Ba, and T. Anderson Keller

Through the use of recurrent neural networks trained to solve tasks requiring the integration of global information, but with constrained local connectivity, the authors find neurons learn to encode and transmit information to other spatially distant neurons through traveling waves.

• Preprint
• Repository

Alignment Reduces Conceptual Diversity of Language Models

By: Sonia Murthy, Tomer Ullman, and Jennifer Hu

The authors use a new way of measuring the conceptual diversity of synthetically-generated LLM “populations” to investigate whether LLMs capture the conceptual diversity of human populations.

ProCyon: A Multimodal Foundation Model for Protein Phenotypes

By: Owen Queen, Robert Calef, and Marinka Zitnik

The authors introduce ProCyon, a multimodal foundation model to model, generate, and predict protein phenotypes.

• Preprint
• GitHub
• Project website
• Dataset

Loss-to-Loss Prediction

Scaling Laws for all Datasets

By: David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach and Sham Kakade

The authors develop a method to predict how large language models scale with compute across different datasets, enabling more efficient training and better understanding of data-compute tradeoffs.

• Preprint
• Training code
• Notebooks
• Huggingface models
• TMLR referee report

How Does Critical Batch Size Scale in Pre-training? (Decoupling Data and Model Size)

By: Hanlin Zhang, Depen Morwani, Nikhil Vyas, Udaya Ghai, Jingfeng Wu and Difan Zou

The authors empirically show that the critical batch size for pre-training scales with data size rather than model size, followed by theoretical justifications for it.

• Preprint

Mixture of Parrots 🦜🦜🦜: Experts Improve Memorization More Than Reasoning

By: Samy Jelassi and Eran Malach

The authors demonstrate, through theory and experiments, that in MoEs, experts improve memorization more than reasoning.

• Preprint

Contrastive Learning Explains the Emergence and Function of Visual Category Selectivity

By: Jacob Prince, George Alvarez, and Talia Konkle

The authors introduce an updated framework for understanding visual object recognition and category selectivity: contrastive coding.

• Paper

Context Matters for Foundation Models in Biology

By: Michelle M. Li and Marinka Zitnik

Introducing PINNACLE, a novel contextual AI model for single-cell protein biology that supports a broad array of biomedicalAI tasks by tailoring outputs to the cell type context in which the model makes predictions.

• Paper
• Research Briefing
• Code
• Project website
• Demo

Anything but SGD: Evaluating Optimizers for LLM Training

By: Rosie Zhao, Depen Morwani, David Brandfonbrener and Nikhil Vyas

The authors perform a rigorous study of a variety of LLM training optimizers and find that they are all fairly similar except for SGD, which is notably worse.

• Paper

Transcendence: Generative Models Can Outperform the Experts That Train Them

By: Edwin Zhang and Eran Malach

The authors theoretically and empirically demonstrate that generative models in chess can outperform the experts that train them by low-temperature sampling.

• Preprint
• Code
• Dataset
• Play with our models