Grammar, Reasoning, Learning: Three Short Stories on Comparative & Rational Analysis of Language Model Capabilities

Join us for a talk by Andrew Lampinen, Staff Research Scientist at DeepMind. This talk is part of the Kempner Seminar Series, a research-level seminar series on recent advances in the field.

There has been substantial debate about the capabilities of language models—which aspects of language they can acquire, whether they can be said to ‘reason’, and whether they can truly ‘learn’ in context. In this talk, I will suggest that approaches from cognitive science can provide useful tools for approaching these questions. Specifically, I will focus on comparative methods (comparing capabilities across different systems) and rational analysis (analyzing behavior as a rational adaptation to an environment).

I will illustrate different aspects of these ideas through three examples from our recent work: 1) comparing processing of recursive syntactic structures in language models and humans, 2) evaluating the way that both language models and humans entangle content in their responses to logical reasoning problems, and 3) understanding how in-context learning emerges from properties of training distributions. I will outline how these disparate phenomena can be understood using these cognitive methods, and the implications for evaluating and understanding language model behaviors.

Coming from Longwood? Sign up to take the shuttle.