Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry

Adversarial Concept Search (ACS) identifies meaningful conceptual scenarios that are likely to induce model failure without evaluating the model on specific inputs that instantiate those scenarios. Here, the authors identify challenging scenarios for an LLM without explicitly evaluating them or even accessing specific input examples and use the LLM’s representational geometry to predict model failures caused by interference between salient features.

If you use a Large Language Model (LLM) routinely, you might be impressed by its capabilities on many tasks and then surprised by an embarrassing, simple error. Perhaps the model occasionally fails at arithmetic multiplication. Perhaps it flubs a simple question, if you ask in your native language. Maybe it gets each step of reasoning right, but messes up when it has to put them all together. Modern evaluation work often focuses on finding and testing the most challenging scenarios. But when models get most prompts right, it gets harder and harder to find the surprising scenarios they’ll get wrong. 

Humans cannot always intuit which scenarios are most challenging for LLMs. To capture difficult edge cases, developers often either design problems that are difficult for humans or curate extensive benchmarks which are comprehensive enough to include a few challenges. To create challenging datasets—whether manually crafted or synthetically generated—we need labor, resources, and often-unreliable intuition. What if we could instead anticipate which scenarios a model will fail on and focus on those?

We call this objective Adversarial Concept Search (ACS): the task of identifying meaningful conceptual scenarios that are likely to induce model failure, without evaluating the model on specific inputs that instantiate those scenarios. In our new paper, we identify challenging scenarios for an LLM without explicitly evaluating them or even accessing specific input examples. Specifically, we predict model failures caused by interference between features using the LLM’s representational geometry.

To explore challenging scenarios, we check specific concept combinations. Many LLM errors are failures of compositional generalization, the model’s ability to handle new combinations of atomic concepts. For example, if the model knows who wrote 1984 when asked in English, but not when asked in Hungarian, then it fails to generalize to the combination of that fact with Hungarian. Compositional generalization may involve an intractably large space of scenarios with possible combinations —we can’t easily translate every known fact into every language, and even if it is easy to produce inputs, it may be expensive to evaluate them. If we can efficiently predict which combinations will be most challenging, we can produce small datasets which capture many of the model’s remaining errors.

Our predictions rely on the hypothesis that models make mistakes when features stored in superposition cause interference. LLMs encode many features in limited dimensions by sharing linear directions. In principle, this compression can be lossless— that is, we can recover the disentangled features — when feature activations are sparse. In practice, however, we posit that models learn a lossy coding of their input features: when multiple non-orthogonal feature encodings are simultaneously active, they can interfere with one another and damage model performance.

This interference is especially salient in compositional settings, where multiple task-relevant features must be jointly represented. We will show that LLM failures can arise from geometric interference between atomic concept representations, and that these failures can be predicted from the angles between their encoded feature directions.

Across settings that require systematic composition—toy programmatic tasks, multihop reasoning, and multilingual factual recall—our method reliably anticipates failure modes without evaluating corresponding compositional inputs. Instead, it predicts errors from a description of the concepts involved and their atomic representations. This allows developers to prioritize generating or collecting coherent, challenging inputs rather than evaluating the full combinatorial space. Using representational geometry, we could identify high-risk examples, construct targeted stress tests, and provide a scalable foundation for adversarial training on realistic deployment scenarios.

---

Compositionality and Lossy Superposition

A system like an LLM generalizes compositionally when it successfully recombines known atomic concepts in novel configurations. For example, if the training distribution includes “black dog” and “white cat,” a compositional model should reliably process “black cat.” This capacity is crucial for robust performance in new settings.

In the language of compositionality, properties like “black” and “dog” are atomic concepts which can be combined in new ways. For interpretability, we might associate each of these concepts with one or more linear features. Our new work shows that, by measuring the geometry of these features, we can predict whether a model will generalize to a specific scenario. In particular, models fail when there are smaller angles between the most relevant features.

Compositionality has long been associated with orthogonal atomic feature representations; if each feature is represented by its own dedicated direction, then one feature can’t just overwrite another . But modern theories of feature superposition show that non-orthogonal encodings can still support lossless reconstruction, as long as feature activations are sparse. In practice, however, we contend that superposition is often lossy: when multiple non-orthogonal features are active together, they can interfere with one another and make feature recovery less robust.

To make this concrete, suppose an input activates a sparse feature vector $\mathbf{z}$ with salient support $S$. In an idealized setting, the model could recover $\mathbf{z}$ from its encoded representation $A\mathbf{z}$ with zero recovery error: 

\[ \left\lVert \mathbf{z} – \hat{\mathbf{z}} \right\rVert_2 = 0. \] 

However, under noise or imperfect decoding, recovery becomes less robust when the active feature encodings are highly similar. This motivates measuring the local cumulative coherence of the salient support: 

\[ \alpha(S) = \max_{i \in S} \sum_{\substack{j \in S \\ j \neq i}} \left| \cos\!\left(\mathbf{a}_i,\mathbf{a}_j\right) \right|. \]

where a_i and a_j are feature encodings. Intuitively, this captures the most damaging feature: the salient feature that has the smallest angle with other features in the same support.

In real LLMs, the ground-truth latent features are not directly accessible. We therefore estimate salient feature encodings from residual-stream representations. Depending on the setting, the features associated with a particular concept might be a single activation vector, an average across contexts, or a low-rank subspace. 

Because geometry is often dominated by large-scale clusters, such as prompt format, task family, topic, or language, we center examples within each cluster before estimating concept-level geometry. 

We define Compositional Interference (CI) as a normalized version of this local coherence:

Here, C is the active concept set and S(C) is the corresponding salient feature support. Higher CI means that the relevant concept representations are closer together and more likely to interfere with each other; lower CI means they are more separated and more reliably composable. This lets us predict compositional failure from the geometry of atomic concepts, without evaluating the composed input itself.

Using CI for Adversarial Concept Search

ACS aims to answer: if resource constraints only allow us to test a small subset of all possible concept combinations, which subset should we collect to maximize error coverage?

For example, say you are looking for challenging prompts in Hungarian, but you can only afford to translate 5% of your English-language factual queries into Hungarian. Which 5% should you translate? If you select your test set uniformly at random, then your LLM’s accuracy on the test set will be close to its average across all possible combinations. Under our proposal, you would instead pick the 5% of queries with the highest CI between the original query and the Hungarian-specific features. Our hypothesis is that this 5% will have lower accuracy than the average across all possible combinations. Furthermore, as you lower the threshold for the minimum CI, we predict that model accuracy on each test set will gradually increase toward the mean as lower-interference examples are included. In addition to visualizing these cumulative test sets, we will also evaluate the predictive power of CI as a direct error classifier through PR-AUC.

---

Predicting Compositional Failure in Real LLMs

Let’s see how our predictions work in a real LLM, Llama-3.2-3B. We evaluate ACS on two tasks: multihop question answering and multilingual factual recall. In both settings, we focus on examples where the model succeeds on the atomic components. This lets us isolate compositional errors: the model has the pieces, but fails to combine them.

Multihop reasoning

In multihop question answering, each composed query is built from two concepts, the first-hop and second-hop queries. For example, take the composed query “In what year was the author of 1984 born?” This query combines atomic first-hop quesry, “Who was the author of 1984?” with the atomic second-hop query “In what year was George Orwell born?” The model needs to answer both of these atomic queries and compose them to get the final answer.

To focus specifically on compositional failures, we only include two-hop queries where the model answers the corresponding single-hop queries correctly. We compute CI between the residual activations from the atomic queries and compute CI between the two representations to predict failure on the composed query using only its atomic components.

Multilingual factual recall

Multilingual factual recall presents a different compositional setting. The models might know how to answer, “Who was the author of 1984?” but only when asked in English and not in Hungarian. To transfer knowledge, the model must answer a query in a target language by combining a factual representation with a language-specific representation. The model may know the fact in English and may understand the target language, but still fail when both components are required.

For each target-language query, we extract the fact representation from the corresponding English factual query and represent the language concept as a low-rank subspace estimated from multilingual text. We then compute CI between the English fact representation and the target-language subspace.

We can’t easily translate every single fact into every language. It would be much easier to translate just a few facts to find where the model gets it wrong. We therefore use CI to predict cross-lingual transfer failure without access to the translated input.

Results

Just as we hypothesised, the LLM has lower accuracy on ACS challenge sets with stricter CI cutoffs. Crucially, we predicted LLM errors without evaluating the composed query itself, solely from interference between constituent atomic queries.

The same pattern appears in multilingual factual recall. Across languages, we can predict multilingual failures without running LLMs on the non-English inputs, just by looking at the angle between a language subspace and a prompt activation. Used as a direct classifier, our error predictions outperform majority baselines for every language tested.

Figure 4 illustrates some example fact-language combinations, displayed with their CI scores. Qualitatively, correct model predictions tend to have lower CI, while model errors are associated with higher CI. 

In Figure 5, we show another view of these predictions by asking how much data we need to collect to find 25% of all compositional errors. In each case, we can beat random sampling. For example, we can find 25% of all Dutch errors by translating examining only the top 11% highest-CI examples, whereas random inspection would require 25% of the data on average. This brings our efficiency closer to the perfect predictor, which selects only true errors, than to the random predictor! In real LLM settings, CI helps us find compositional failures while inspecting substantially fewer examples.

Figure 6 also shows that ACS extends beyond individual examples to coarse-grained concepts. Compositional interference predicts dataset-level difficulty in both multilingual fact recall and multihop reasoning from descriptions like “birth year facts in Japanese.” Overall, the results suggest that CI can identify both specific high-risk examples and broader high-risk task categories.

Conclusion

Our new paper sets the stage for the biggest practical use case of model interpretability: stress testing and dataset development. All you need is interpretable linear features and simple geometry.

Acknowledgements

This work was enabled in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. IL and NS are supported by a grant from Coefficient Giving and the Berkeley Existential Research Institute (BERI).