Mental Programs in Humans and Machines

Join us for a talk by Kevin Ellis, Assistant Professor of Computer Science at Cornell University. This talk is part of the Kempner Seminar Series, a research-level seminar series that covers topics related to the basis of intelligence in natural and artificial systems.

Consider experimenting to learn how to use a new appliance, webpage, or toy: Within tens of minutes, we can learn how something new works, and use that knowledge to achieve novel goals, make sense of similar devices, and communicate our new knowledge in natural language. How could an AI system similarly acquire, transfer, and communicate its knowledge of how things work? To make progress on this question, we study cognitive models and AI systems which represent knowledge as symbolic “programs” written in Python, natural language, and/or logic.

I present results suggesting that encoding knowledge in natural language better explains human learning than representing knowledge in a bespoke formal language, but that LLMs on their own are insufficient: Instead, extra probabilistic machinery is needed to explain human inductive reasoning. Then, I present AI systems which expand on this insight by generating programs to explain their observations, focusing on agents which program world models to describe how new things work, with applications in program synthesis and robotics. Last I compare symbolic programs against purely neural representations, including in-context learning and its extensions, finding that neither strictly dominates the other, and instead that they play complementary roles in inductive reasoning.

Kevin Ellis is an Assistant Professor at Cornell University in Computer Science, having previously completed his PhD at MIT in Cognitive Science. His research studies the intersection of program synthesis, AI, and human cognition, and was previously recognized with an NSF CAREER Award, coverage by the New York Times, a selection as one of the best human behavior articles in Nature Communications its year, and a best paper award at the most recent ARCPrize contest.