Kempner Researchers Harness Generative AI to Reveal What Neurons “Want”
A new study uses generative models to reveal how neurons in the brain’s visual system respond to textures and objects
Binxu Wang (right) and her former Ph.D. advisor and current collaborator Carlos Ponce (left) use generative AI to let neurons dictate the properties of test images, revealing unforeseen patterns in the neurons’ preferences.
Photo credit: Kris Brewer
At a Glance
- Kempner Institute researchers used generative AI to uncover how neurons in the visual system respond to different kinds of images.
- A real-time feedback loop enabled neurons to “teach” image generators what types of visual information they are tuned to.
- The results help explain how the brain processes visual information, and may inspire smarter, more flexible AI systems.
How can scientists figure out what visual information a neuron really responds to? For decades, neuroscientists have tried to answer this question by showing animals pictures — of faces, trees, houses, and other animals, for example. The ways that neurons in vision-related brain areas respond to these pictures gives researchers clues about the kinds of information that the neurons are processing from the pictures.
While this approach has yielded important discoveries, it also has notable limitations. “When you see a neuron responding to a cat, it’s tempting to call it a ‘cat neuron’ or ‘face neuron,'” says Binxu Wang, a research fellow at the Kempner Institute for the Study of Natural and Artificial Intelligence. “While this can be useful, it can also be a trap.”
The problem, says Wang, is that the test images are chosen by human researchers. Because the researchers rely on their own visual intuitions to choose and interpret images, they may miss the very patterns that neurons in the primate visual system respond to most strongly. In other words, images chosen by humans might not fully reflect the kinds of information that the neurons preferentially respond to.
In a new paper titled “Neuronal tuning aligns dynamically with object- and texture manifolds across the visual hierarchy,” just published in Nature Neuroscience, Wang and her former Ph.D. advisor and current collaborator Carlos Ponce, assistant professor of Neurobiology at Harvard Medical School and affiliate faculty at the Kempner Institute, present a new approach to this problem. They use generative AI to let the neurons themselves shape the properties of test images, revealing hidden structure in the neurons’ visual preferences.
Wang and Ponce have designed a sophisticated real-time feedback method that links neurons with deep image generators, which are AI algorithms that learn statistical patterns from large datasets to produce images. In this approach, if a neuron (in this case that of a monkey) fires more in response to a given picture, the generator adapts its input to produce similar images in the next round; if it fires less, the generator shifts away from images with similar visual features.
Here, the neurons themselves act as critics for the image generator, offering feedback through closed-loop algorithms. “We wrote algorithms to propose new images that the neurons would like,” Wang explains. “It’s kind of like Instagram or Facebook, where recommendation algorithms converge on content that is ‘highly activating’ to the user.”
Over many cycles the images become increasingly tuned to a given neuron’s responses, revealing the visual features that the neuron is specialized to detect. In other words, the generator gradually learns the neuron’s “preferences,” which the neuron expresses via its firing rate. This feedback method is repeated for neurons in different brain areas, shedding light on their distinctive coding properties that contribute to visual information processing.
Aligning Brains and Computer Image Generators
Wang and Ponce applied their innovative method to a set of brain areas including the primary visual cortex (V1), area V4 in the visual cortex, and the posterior inferotemporal cortex (PIT). These brain areas are part of a processing pathway in the visual system known as the “ventral stream,” which is essential for recognizing objects.
Within the ventral stream, neural firing patterns move from simple to more complex: cells in V1 fire for edges and textures, while higher areas like the posterior inferotemporal cortex (PIT) enable recognition of complete objects. But how exactly neurons along this pathway represent the staggering diversity of natural images has been a long-standing mystery.
Wang and Ponce tackled this question using two image generators: DeePSim, which produced swirling, textured patterns, and BigGAN, which conjured more photorealistic objects. They recorded brain activity in macaque monkeys in response to rapid image bursts of just 100 milliseconds. A clear pattern emerged: V1 neurons lit up for DeePSim textures, V4 neurons responded more to optimized textures than to objects, but less decisively, and PIT neurons ultimately aligned with both systems, showing strong responses to objects as well as textures.
Wang likens the relationship between neurons and image generators to a patron commissioning a painting. “Imagine a patron and an artist who speak different languages,” she says. “The patron asks for photorealistic art, but the artist is too abstract or modern. Their tastes just don’t match.” Early visual neurons communicate easily with a generator that produces abstract textures, but they struggle to “get what they want” from a generator that creates realistic objects.
A neuron in the visual system might respond best to certain patterns—such as edges, curves, textures, or more complex shapes. A generator produces images featuring different kinds of patterns by moving through a “latent space,” a mathematical space that represents different visual possibilities. If a neuron’s preferences align neatly with this latent space—almost like a clear linear direction to move along—it can easily “steer” the generator toward images it favors. But when the relationship is more complex, resembling a treacherous landscape, it becomes much harder for the neuron’s activity to guide the generator toward images that strongly activate it.
Wang and Ponce discovered that DeePSim’s texture-based space aligned naturally with early visual neurons, while BigGAN’s object-based space matched better with higher-level regions like PIT. The findings build upon Nobel Prize–winning work by David Hubel and Torsten Wiesel, who showed that V1 neurons act as edge detectors, increasing their firing in response to straight line patterns in images. Wang sees her study as extending that legacy. “Our predecessors in the field weren’t wrong,” she says. “But our research is revealing far more variation in neuronal selectivity than they originally uncovered.”
What Neurons Like, and Why It Matters
Wang and her team also discovered that neuronal alignment with the texture-based space and the object-based space is dynamic, switching rapidly over time. PIT neurons, for example, initially fired in response to textures but then shifted toward object-like responses. This suggests object perception unfolds over time instead of being hardwired from the start.
It also underscores a gap between biological and artificial vision. Most current computational models of the visual system lack the dynamic aspect of the neural code, since they map images to static responses. Closing that gap will be a crucial challenge for future NeuroAI research.
This work suggests that generative models can be used as tools to mitigate human bias in neuroscience research. “One point of this paper is to show that we should leverage generative models more, to broaden our imagination and better understand what neurons really like,” Wang says. She also thinks that the flexibility of biological neurons in representing textures, objects, and perhaps other categories could inspire more adaptive, resilient AI systems.
Creativity in Brains and Machines
At the heart of Wang’s research is a fascination with creativity itself. “A theme in my work is creativity, or being generative, which means going beyond what is already known,” she says. “This manifests both in how we study the neurons, and in another line of my work that examines how AI systems generate anything more than what they have seen.” Comparing neurons with generative AI systems suggests a potential shared approach: both learn simple patterns and combine them to create something new.
“People sometimes worry that generative models aren’t truly creative,” Wang notes. “But human creativity isn’t entirely different. We also learn from what we’ve seen — absorbing motifs from masters and experience — and then combine them in ways that feel fresh and original.”
*****
To learn more about this research, read the paper in Nature Neuroscience.
About the Kempner
The Kempner Institute seeks to understand the basis of intelligence in natural and artificial systems by recruiting and training future generations of researchers to study intelligence from biological, cognitive, engineering, and computational perspectives. Its bold premise is that the fields of natural and artificial intelligence are intimately interconnected; the next generation of artificial intelligence (AI) will require the same principles that our brains use for fast, flexible natural reasoning, and understanding how our brains compute and reason can be elucidated by theories developed for AI. Join the Kempner mailing list to learn more, and to receive updates and news.
PRESS CONTACT:
Deborah Apsel Lang | kempnercommunications@harvard.edu
