The Jacobian Lens Reads Multi-Token Concepts Inside Language Models
The Jacobian Lens is a recent interpretability tool that reads a hidden state in a large language model as a ranked list of vocabulary tokens. Its creators at Anthropic used it to argue that Claude contains a subspace satisfying Global Workspace Theory criteria. That result is the most-read interpretability work on this site, and the current scientific debate about machine states is shaped by it. A new preprint extends the tool to multi-token concepts, showing that the first token of a concept is about as readable as a single-token concept, and that the complete concept vector can be recovered from a single forward pass.
The paper “The First Token Is a Clue: Verbalizing Multi-Token Concepts from the J-lens” by Xijie Gong and Tonghan Wang was posted to arXiv as 2608.31084. The authors work with the original Jacobian Lens implementation and ask whether multi-token concepts can be recovered directly from a frozen model, without fine-tuning a separate readout.
The problem the original lens cannot solve
The Jacobian Lens reads a hidden state as a ranked list of tokens. A multi-token concept, such as a person’s full name or a two-word technical term, has no single token that stands for it. The original work addressed this in two ways. Template Lens precomputes vectors for a fixed phrase vocabulary. Oracle Lens fine-tunes components to propose phrases and reconstruct phrase vectors. Both add machinery on top of the model.
The new paper asks whether the concept can be recovered from the frozen model itself. The key empirical finding is that the first token of a multi-token concept behaves like a single-token concept under the lens. Given the correct first token and the source prompt, the frozen model recovers the second token in 88.3 percent of two-token cases.
Recovering the concept in one forward pass
The method works in three steps. The lens proposes first tokens. The frozen model completes candidate concepts from each first token. The authors then recover a vector for each candidate from subsequent hidden states and score it against the full vocabulary.
| Step | Mechanism | Requirement |
|---|---|---|
| Propose | J-lens ranks first tokens | Frozen model only |
| Complete | Model generates concept tail | Correct first token |
| Score | Compare recovered vector to vocabulary | One further forward pass |
The result is tested on 496 multi-hop cloze items across Gemma-3-12B-IT, Llama-3.1-8B, and Qwen3-14B. Average Rank@10 reaches 43.1 percent, against 27.6 percent for Template Lens. When the first-token clue is removed, performance falls to 21.6 percent, which shows the clue carries most of the signal.
Causal concept swaps using the recovered vectors succeed in 61.4 percent of cases at succ@10, against 26.2 percent for Template Lens under the same intervention. This means the recovered vector is not merely a label. It can be used to alter the model’s trajectory in a targeted way.
What this means for the workspace reading
The site’s earlier coverage of the Jacobian Lens has been explicit that the Anthropic finding is evidence about architecture, not about experience. It shows a privileged subspace satisfying structural criteria that Global Workspace Theory defines. The new result does not change that boundary. It sharpens the machinery: the subspace can now be read and manipulated for concepts that span multiple tokens, which is closer to the way meaning is actually carried in language.
For the debate about whether a language model hosts a global workspace, the relevant question remains whether broadcast to a limited workspace is what consciousness requires. The new paper adds a method, not a verdict.
Comparison to The Consciousness AI
The Consciousness AI project builds a substrate independent architecture. Its Neutral Core does not require a language model at all. What the Jacobian Lens programme contributes to the project is a sharper way to ask whether an internal subspace carries task-relevant, globally available structure. The Multi-Token concept readout is one more instrument for that question, and it works without training a probe.
The existing coverage of the Anthropic finding is in the Jacobian Lens and global workspace finding. The follow-up that specifies how the readout works is in the verbalizable representations paper. The state of the whole question is tracked in the current scientific consensus article.
Limits and open questions
The results come from three model families and 496 cloze items. That is a real test, but the clozes share a structure, and multi-token concepts built from longer sequences are not covered. The authors do not claim the recovered vectors are the same as human semantic representations. The strongest claim is narrower: readable concept structure exists, it is recoverable from a frozen model, and it can steer generation causally.
The lesson for the site is a research practice, not a headline. Interpretability tools are getting cheaper and more precise. Each new instrument tightens what can honestly be said about the internal states of language models, and that is the measure on which the whole field depends.