Research Roundup

This issue of the Kempner Research Roundup features publications, preprints, and awards from December 2023 to February 2024.

* indicates equal contribution
^† indicates Kempner community member

Awards

Affiliate Faculty member Melanie Weber was selected as a 2024 Alfred P. Sloan Fellow in Mathematics!

Publications

Exploring the Promise and Limits of Real-Time Recurrent Learning
Kazuki Irie^†, Anand Gopalakrishnan, Jürgen Schmidhuber
ICLR (2024)

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, Naomi Saphra^†
ICLR (2024)

Dynamic Masking Rate Schedules for MLM Pretraining
Zachary Ankner, Naomi Saphra^†, Davis Blalock, Jonathan Frankle^†, Matthew L. Leavitt
ICLR (2024)

Effective Structural Encodings via Local Curvature Profiles
Lukas Fesser, Melanie Weber^†
ICLR (2024)

Forward learning with top-down feedback: empirical and analytical characterization
Srinivasan Ravi, Mignacco Francesca, Sorboro Martino, Refinetti Maria, Cooper Avi, Kreiman Gabriel^†, Dellaferrera Giorgia
ICLR (2024)

On the hardness of learning under symmetries
Bobak Kiani*, Thien Le*, Hannah Lawrence*, Stefanie Jegelka, Melanie Weber^†
ICLR (2024)

TRAM: Bridging Trust Regions and Sharpness Aware Minimization
Tom Sherborne, Naomi Saphra^†, Pradeep Dasigi, Hao Peng
EACL (2024)

Transformable linkage-based gripper for multi-mode grasping and manipulation
Junghan Kwon*, David Bombara*, Clark Teeple, Joonhaeng Lee, Chuck Hoberman, Robert Wood, Justin Werfel^†
IEEE Robotics and Automation Letters (December 2023)

Tuned compositional feature replays for efficient stream learning
Talbot Morgan, Zawar Rushikesh, Badkundri Rohil, Zhang Mengmi, Kreiman Gabriel^†
IEEE Transactions in Neural Networks and Learning Systems (December 2023)

Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions
Kazuki Irie^†, Róbert Csordás, Jürgen Schmidhuber
EMNLP (2023) [short paper]

Approximating Two-Layer Feedforward Networks for Efficient Transformers
Róbert Csordás, Kazuki Irie^†, Jürgen Schmidhuber
Findings of EMNLP (2023)

Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
Benjamin L. Edelman^†, Surbhi Goel, Sham Kakade^†, Eran Malach†, Cyril Zhang
NeurIPS (2023) [spotlight]

Combining Behaviors with the Successor Features Keyboard
Wilka Carvalho^†, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran
NeurIPS (2023)

Contrastive Training of Complex-Valued Autoencoders for Object Discovery
Aleksandar Stanić*, Anand Gopalakrishnan*, Kazuki Irie^†, Jürgen Schmidhuber
NeurIPS (2023)

The Hidden Linear Structure in Score-Based Models and its Application
Binxu Wang^†, John Vastola
Workshop on Diffusion Models @ NeurIPS (2023)

Understanding Learning Dynamics of Neural Representations via Feature Visualization at Scale
Chandana Kuntala, Carlos R Ponce, Deepak Kumar Sharma, Binxu Wang^†
UniReps (Unifying Representations in Neural Models) Workshop @ NeurIPS (2023)

Preprints

Adaptive algorithms for shaping behavior
William L. Tong^†, Anisha Ayer, Venkatesh N. Murthy^†, Gautam Reddy

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
Kenneth Li^†, Samy Jelassi, Hugh Zhang^†, Sham Kakade^†, Martin Wattenberg, David Brandfonbrener^†

Measuring and Controlling Persona Drift in Language Model Dialogs
Kenneth Li^†, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas^†, Hanspeter Pfister, Martin Wattenberg^†

This inaugural issue of the Kempner Research Roundup highlights publications, preprints, and awards from throughout 2023.

* indicates equal contribution
^† indicates Kempner community member

NeurIPS 2023 Acceptances

Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency[spotlight]
Owen Queen, Thomas Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, Marinka Zitnik^†

Full-Atom Protein Pocket Design via Iterative Refinement [spotlight]
Zaixi Zhang, Zepu Lu, Zhongkai Hao, Marinka Zitnik^†, Qi Liu

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model [spotlight]
Kenneth Li*^†, Oam Patel*, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg^†

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks [spotlight]
Blake Bordelon, Cengiz Phlevan^†

Learning Curves for Heterogeneous Feature-Subsampled Ridge Ensembles
Benjamin S. Ruben, Cengiz Pehlevan^†

Neural circuits for fast Poisson compressed sensing in the olfactory bulb
Jacob A. Zavatone-Veth*, Paul Masset*, William L. Tong*^†, Joseph D. Zak, Venkatesh N. Murthy*^†, Cengiz Pehlevan*^†

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
Bariscan Bozkurt, Cengiz Pehlevan^†, Alper T Erdogan

Long Sequence Hopfield Memory
Hamza Tahir Chaudhry*, Jacob A. Zavatone-Veth*, Dmitry Krotov, Cengiz Pehlevan^†

Learning curves for deep structured Gaussian feature models
Jacob A. Zavatone-Veth, Cengiz Pehlevan^†

Cognitive Steering in Deep Neural Networks via Long-Range Modulatory Feedback Connections
Talia Konkle^†, George Alvarez^†, Scaling Data-Constrained Language Models
Niklas Muennighoff, Alexander M. Rush, Boaz Barak^†, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, Colin Raffel

Successor-predecessor intrinsic exploration
Changmin Yu, Neil Burgess, Maneesh Sahani, Samuel J. Gershman^†

Combining Behaviors with the Successor Features Keyboard
Wilka Carvalho†, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran

Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation
David Brandfonbrener^†, Ofir Nachum, Joan Bruna

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
Nikhil Vyas*, Alexander Atanasov*, Blake Bordelon*, Depen Morwani, Sabarish Sainathan^†, Cengiz Pehlevan^†

Dynamics of Temporal Difference Reinforcement Learning
Blake Bordelon, Paul Masset, Henry Kuo, Cengiz Pehlevan^†

Other Publications

Prompting is not a substitute for probability measurements in large language models
Jennifer Hu^†, Roger Levy
EMNLP (2023)

Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models
Sonia K. Murthy^†, Kiera Parece, Sophie Bridgers, Peng Qian, Tomer Ullman^†
Findings of EMNLP (2023)

Emergence of belief-like representations through reinforcement learning
Jay Hennig, Sandra Romero Pinto, Takahiro Yamaguchi, Scott Linderman, Naoshige Uchida, Samuel Gershman^†
PLoS Computational Biology (2023)

Learning Compositional Tasks from Language Instructions
Lajanugen Logeswaran, Wilka Carvalho^†, Honglak Lee
Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) (2023)

Composing Task Knowledge With Modular Successor Feature Approximators
Wilka Carvalho^†, Angelos Filos, Richard L. Lewis, Honglak Lee, Satinder Singh
ICLR (2023)

Transformable linkage-based gripper for multi-mode grasping and manipulation
Junghan Kwon*, David Bombara*, Clark Teeple, Joonhaeng Lee, Chuck Hoberman, Robert Wood, Justin Werfel†
IEEE Robotics and Automation Letters (2023)

Comparing the Evaluation and Production of Loophole Behavior in Children and Large Language Models
Sonia K. Murthy^†, Sophie Bridgers, Kiera Parece, Elena L. Glassman, Tomer Ullman
First Workshop on Theory of Mind in Communicating Agents, ICML (2023)

Cortical topographic motifs emerge in a self-organized map of object space
Fenil R. Doshi^†, Talia Konkle^†
Science Advances (2023)

The neural architecture of theory-based reinforcement learning
Momchil S. Tomov, Pedro A. Tsividis, Thomas Pouncy, Joshua B. Tenenbaum, Samuel J. Gershman^†
Neuron (2023)

Face-deprived networks show distributed but not clustered face-selective maps
Fenil R. Doshi^†, Talia Konkle^†
Vision Sciences Society (2023)

Feedforward Neural Networks can capture Human-like Perceptual and Behavioral Signatures of Contour Integration
Fenil R. Doshi^†, Talia Konkle^†, George Alvarez^†
Cognitive Computational Neuroscience (CCN) (2023)

High-performance evolutionary algorithms for online neuron control
Binxu Wang^†, Carlos R. Ponce
Proceedings of the Genetic and Evolutionary Computation Conference (2023)

Structure-inducing pre-training
Matthew B. A. McDermott, Brendan Yap, Peter Szolovits & Marinka Zitnik^†
Nature Machine Intelligence (2023)

Coordinated drift of receptive fields in Hebbian/anti-Hebbian network models during noisy representation learning
Shanshan Qin, Shiva Farashahi, David Lipshutz, Anirvan M. Sengupta, Dmitri B. Chklovskii & Cengiz Pehlevan^†
Nature Neuroscience (2023)

What can 5.17 billion regression fits tell us about artificial models of the human visual system?
Colin Conwell, Jacob S. Prince, George A. Alvarez^†, Talia Konkle^†
3rd Workshop on Shared Visual Representations in Human and Machine Intelligence (SVRHM), NeurIPS

Human-like signatures of contour integration in deep neural networks
Fenil R. Doshi^†
Talk presented at Vision Science Society (2022)

Preprints

Chain-Of-Thought Reasoning is a Policy Improvement Operator
Hugh Zhang^†, David C. Parkes

Pruning for Feature-Preserving Circuits in CNNs
Chris Hamblin, Talia Konkle^†, George Alvarez^†

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics
Caleb Weinreb, Mohammed Abdal, Monium Osman^†, Libby Zhang, Sherry Lin, Jonah Pearl, Sidharth Annapragada, Eli Conlin, Winthrop F. Gillis, Maya Jay, Ye Shaokai, Alexander Mathis, Mackenzie Weygandt Mathis, Talmo Pereira, Scott W. Linderman, Sandeep Robert Datta