Parametric Neural Control Differentiates Top Neural Network Models of Primate Visual Cortex

Deep neural network (DNN) encoding models have emerged as powerful tools for predicting visually evoked brain responses, from single-unit activity in macaque visual cortex to fMRI responses across the human visual hierarchy. Yet neural encoding scores have increasingly saturated, with recent surveys ^[1] showing that hundreds of distinct models achieve nearly identical scores despite substantial differences in architecture and training objectives.

Our new preprint^[2] tests whether these leading models provide equally good explanations of visual feature tuning: the patterns of selectivity and invariances that biological neurons show over the space of possible visual inputs. To test this, we propose that a model that accurately predicts brain responses to neural images should itself be able to leverage the brain-model mapping to synthesize new images that precisely control brain responses in subsequent experiments.

We find that models that are indistinguishable in standard prediction benchmarks are strongly distinguished by this neural control test. Asked to synthesize images that systematically manipulate neural firing, the models diverged strongly in how well the brain responses aligned with their predictions, and all of them overestimated their control capacity. High brain predictivity over a large and diverse set of natural images therefore does not guarantee that a model has captured a neuron’s visual feature tuning.

Using neural control to compare predictive models of visual cortex

We applied a closed-loop paradigm for model comparison, run over two different phases of experimental data collection, to see whether popular models (DINOv2, CLIP, ResNets, SigLIP2, and others) can identify the features encoded in visual cortex. It has four steps. First, in a calibration phase, we recorded neural responses from five macaques to a diverse set of 969 natural images, targeting 25 neural sites across V3, V4, inferotemporal cortex, and the superior temporal sulcus. Second, we fit a separate encoding model to each site using each of ten deep neural networks: nine trained models spanning convolutional and transformer architectures, plus one untrained model that serves as a baseline reference. Third, we used each fitted model to synthesize controller images predicted to drive each site’s firing to specific target levels (see below). Finally, in a control phase, we presented all of these images back to the animals, over 27,500 synthesized stimuli in total, and tested whether the observed degree of neural modulation matched each model’s predictions.

Generating synthetic controller stimuli via feature accentuation

The key methodological innovation within this loop is a gradient-based stimulus synthesis technique called feature accentuation^[3][4], adapted from mechanistic interpretability techniques in deep learning. Because a neural encoding axis is a weighted combination of features in a DNN layer, it provides an explicit hypothesis about the stimulus attributes to which a neural site is tuned. Starting from any seed image, we can accentuate it along the encoding axis in fine increments, producing a parametric stimulus sweep. We call this “axis-aligned feature accentuation”.

Generating these sweeps is inherently ill-posed: for a given level of predicted activity, there are potentially infinite images that achieve it, and unconstrained optimization in pixel space tends to produce either hallucinated high-frequency artifacts or imperceptible adversarial perturbations^[3][4]. We therefore regularized the synthesis to suppress both. Each accentuated image sweep targeted 11 predicted response levels, ranging from 25% below to 50% above the site’s native response range. If an encoding model has genuinely captured the site’s tuning, neural responses should vary systematically across the sweep, tracking the model’s predictions.

Divergent model hypotheses about neural tuning

Models with similar prediction scores made qualitatively different hypotheses about the features that would drive neural firing. Even for the same target site and seed image, some models emphasized holistic structure and others enhanced textures or high-frequency content.

Finding 1: High natural-image predictivity does not imply effective neural control

First we consider a single channel in anterior IT and the ResNet50 deep neural network, one of the canonical visual encoders in our model set: the same site and model shown in Figures 1 and 3. As mentioned above, and in line with other benchmarks, ResNet50 supported a highly predictive encoding axis when fit on the calibration set of natural images (r = 0.87), and this axis remained stable in the second recording phase (r = 0.95 over re-presented held-out images). However, the same encoding axis was an exceedingly poor predictor of the site’s activity when tested over its axis-aligned accentuation images (r = 0.28): image sweeps predicted to span from 25% below to 50% above the natural image range led to effectively flat modulation.

Finding 2: Neural control breaks the saturation and reveals hidden differences among models’ brain-alignment

Prediction accuracy over each model’s own accentuated stimuli was lower on average, far more variable, and systematically different across models: despite matched natural-image predictivity, models diverged strongly in their ability to control neural firing. We defined the control score as the Pearson correlation between the model’s predicted responses and the measured neural responses, and the control gain as the slope of the regression of measured responses on predicted responses, where a slope of 1 would imply correctly scaled neural modulation. All models overestimated their control capacity, with slopes below 1 across the population. Two models stood out: CLIPAG^[5], an adversarially fine-tuned CLIP vision transformer, and ResNet50-Robust^[6], a ResNet50 trained to perform object recognition under adversarial attack. Although they differ in architecture, objective, and training data, both incorporate explicit adversarial training, whereas none of the other models do. They led on both measures, and several site-model combinations yielded impressive degrees of parametric control (for example, r = 0.91 and slope = 0.81 for CLIPAG at a V4 site). These differences in control were relatively consistent across brain regions and animals, from V3 and V4 through IT and STS, in all five monkeys.

Finding 3: Input-gradient structure predicts neural control better than adversarial robustness

As shown, both models that performed best on parametric neural control involved some form of adversarial training. Does adversarial sensitivity predict control outcomes more generally? We implemented adversarial attacks targeting all 250 encoding models to measure how easily tiny pixel perturbations could swing each model’s predicted response.

Across all ten models, adversarial sensitivity did correlate with control, but the correlation was carried entirely by the two adversarially trained models (Figure 7, left). Somewhat surprisingly, adversarial sensitivity generalized poorly as a predictor of control beyond the adversarially trained models (dashed line in Figure 7, left).

A stronger predictor came from the input gradients: the map showing how changing each pixel affects a model’s predicted neural response. For some models, the consequential pixels were coherently focused on object and surface boundaries, while others showed diffuse, high-frequency patterns spread across the image (insets in Figure 7, right). We quantified this spatial structure with a spectral measure, based on the radial Fourier power spectrum of the input gradients over held-out natural images, using the participation ratio as a measure of low-frequency spectral concentration. In our study, we observe that lower participation ratio closely tracks steeper rates of decay, indicating greater low-frequency content. This participation ratio strongly predicted parametric neural control across the 250 individual axes (r = −0.64) and across the ten model means (r = −0.90; Figure 7, right, which shows the nine trained models). Critically, and unlike adversarial sensitivity, the association held even among the seven conventionally trained models, excluding both the untrained baseline and the adversarially trained models: weaker at the level of individual axes (r = −0.26), but just as strong across model means (r = −0.93), and it generalized to held-out models, sites, and animals (cross-validated R² = 0.42–0.45). This is a measurable property of model-to-brain mappings that prospectively predicts neural controllability, providing a way to pre-screen models before future closed-loop experiments. A key next step is to test whether this signature generalizes to other experiments.

Discussion and implications for NeuroAI

The study of visual feature tuning using parametric stimulus sets was instrumental in launching the modern field of visual neuroscience (e.g., Hubel & Wiesel, 1962)^[7]. Part of why the field of NeuroAI now relies so heavily on large encoding models is because it became too hard to pre-hypothesize what features high-level visual neurons are tuned to. Our paradigm returns the toolkit of parametric stimulus design to experimenters, with the key difference that the test stimuli come from model hypotheses rather than from our own intuition.

Deploying these accentuated stimuli for model comparison achieves stronger model differentiation by shifting the comparison up the ladder of causal inference^[8]. Our procedure converts each encoding model into a causal hypothesis: a specific and testable trajectory of stimuli that should modulate the neural response, with a predicted firing rate for every image. These encoding-based control paradigms carry stronger inferential power than standard encoding benchmarks, because they intervene on the input according to a model’s own predictions, and for most models those predictions failed. When neural responses track the predicted rates across a parametric sweep, the model has correctly predicted the neural consequences of its prescribed stimulus manipulations.

Our results also bear on current debates of representational convergence between neural systems, including the “platonic”^[9] and “universal”^[10] representation hypotheses. Many models appear to converge on linearly mappable features while differing in their input sensitivities and internal computations. This divergence qualifies recent broad claims of representational universality, and feature accentuation makes these algorithmic differences visible to experimenters.

At the same time, our findings offer a refinement of what adversarial training is doing for brain alignment. A growing body of work has converged on the observation that adversarially trained models have distinctive forms of brain and behavioral alignment^{[11][12][13][14][15]}. Here, adversarial sensitivity separated the two robust models from the rest, but it did not explain graded variation in control among the conventionally trained models. The stronger predictor was the spatial organization of the pixels that each encoding model relies on (i.e., the input gradient). On this view, adversarial training is one route, currently the most effective among the models tested here, to producing this more brain-aligned gradient structure, and other mechanisms that produce the same signature may work just as well. Several inductive biases, including blurred^[16] or noisy^[17] training regimes, feedback architectures^[18], and more developmentally realistic visual diets^[19], may show representations that are naturally more adversarially robust, and are natural candidates to test next.

Finally, our results suggest that expanding naturalistic benchmarks may not distinguish deep networks in their underlying alignment to biological systems, whereas causal experiments that put model-derived hypotheses to the test can be powerful avenues to iterate toward better models.

1. Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2024). A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15, 9383. https://doi.org/10.1038/s41467-024-53147-y[↩]
2. Prince, J. S., Wang, B., Fel, T., Jagadeesh, A. V., Vaziri, P. A., Alvarez, G. A., Livingstone, M. S., & Konkle, T. (2026). Parametric neural control differentiates top neural network models of primate visual cortex. bioRxiv. https://doi.org/10.64898/2026.08.16.745063[↩]
3. Hamblin, C., Fel, T., Saha, S., Konkle, T., & Alvarez, G. (2024). Feature accentuation: Revealing ‘what’ features respond to in natural images. arXiv, 2402.10039. https://doi.org/10.48550/arXiv.2402.10039[↩][↩]
4. Fel, T., Boissin, T., Boutin, V., Picard, A., Novello, P., Colin, J., Linsley, D., Rousseau, T., Cadène, R., Goetschalckx, L., Gardes, L., & Serre, T. (2023). Unlocking feature visualization for deep network with MAgnitude Constrained Optimization. Advances in Neural Information Processing Systems, 36, 37813–37826. https://doi.org/10.48550/arXiv.2306.06805[↩][↩]
5. Ganz, R., & Elad, M. (2024). CLIPAG: Towards generator-free text-to-image generation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 3843–3853. https://doi.org/10.1109/WACV57701.2024.00380[↩]
6. Salman, H., Ilyas, A., Engstrom, L., Kapoor, A., & Mądry, A. (2020). Do adversarially robust ImageNet models transfer better? Advances in Neural Information Processing Systems, 33. https://doi.org/10.48550/arXiv.2007.08489[↩]
7. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1), 106–154. https://doi.org/10.1113/jphysiol.1962.sp006837[↩]
8. Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511803161[↩]
9. Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). Position: The platonic representation hypothesis. Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 20617–20642. https://doi.org/10.48550/arXiv.2405.07987[↩]
10. Chen, Z., & Bonner, M. F. (2025). Universal dimensions of visual representation. Science Advances, 11(27), eadw7697. https://doi.org/10.1126/sciadv.adw7697[↩]
11. Feather, J., Leclerc, G., Mądry, A., & McDermott, J. H. (2023). Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, 26(11), 2017–2034. https://doi.org/10.1038/s41593-023-01442-0[↩]
12. Gaziv, G., Lee, M. J., & DiCarlo, J. J. (2023). Strong and precise modulation of human percepts via robustified ANNs. Advances in Neural Information Processing Systems, 36. proceedings.neurips.cc[↩]
13. McNeal, N., & Ratan Murty, N. A. (2025). Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models. arXiv, 2509.23333. https://doi.org/10.48550/arXiv.2509.23333[↩]
14. Gaziv, G., Goulding, S., Ayvazian-Hancock, A., Bai, Y., & DiCarlo, J. J. (2025). Noninvasive precision modulation of high-level neural population activity via natural vision perturbations. arXiv, 2506.05633. https://doi.org/10.48550/arXiv.2506.05633[↩]
15. Skrill, D., Feather, J., & Norman-Haignere, S. V. (2026) Neural prediction decorrelation reveals that adversarial robustness substantially improves DNN prediction accuracy across the entire human auditory cortex. bioRxiv. https://doi.org/10.64898/2026.08.05.743059[↩]
16. Jang, H., & Tong, F. (2024). Improved modeling of human vision by incorporating robustness to blur in convolutional neural networks. Nature Communications, 15, 1989. https://doi.org/10.1038/s41467-024-45679-0[↩]
17. Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D. D., & DiCarlo, J. J. (2020). Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. Advances in Neural Information Processing Systems, 33. https://doi.org/10.1101/2020.06.16.154542[↩]
18. Konkle, T., & Alvarez, G. (2023). Cognitive steering in deep neural networks via long-range modulatory feedback connections. Advances in Neural Information Processing Systems, 36. proceedings.neurips.cc[↩]
19. Lu, Z., Thorat, S., Cichy, R. M., & Kietzmann, T. C. (2026). Adopting a human developmental visual diet yields robust and shape-based AI vision. Nature Machine Intelligence, 8, 735–748. https://doi.org/10.1038/s42256-026-01228-6[↩]