Transient representation dynamics after prediction violations in language models - upcoming

When the human brain encounters something unexpected, its response does not end at recognizing that the event was unlikely. Unexpected events can produce a transient change in neural activity, including the well-known P300 event-related potential. An upcoming project im currently working on will ask whether an analogous phenomenon can be defined inside large language models: does surprise briefly change how strongly internal representations are updated, and how does that change propagate through the model?

In neuroscience, the P300 is a positive event-related potential associated with unexpected or task-relevant events. Its interpretation is not simply “the brain detected a low-probability stimulus”: theories of the P300 also connect it to attention and to updating an internal representation of the current context.^[1][2]

This raises an interesting question for language models. An autoregressive LLM already assigns a probability to the next token. If the observed token is \(x_t\), its surprise can be written in the usual information-theoretic form:

A very improbable continuation therefore has high surprisal. But surprisal is only a scalar at the output of the predictive distribution. The question I want to investigate is what happens inside the network after such an event.

Where is time in a transformer?

The analogy immediately runs into a problem. A biological P300 unfolds in physical time after a stimulus. A standard transformer does not have an equivalent continuous time coordinate. Instead, there are two separate directions along which computation can be followed.

The first is layer depth. For a fixed token \(t\), its representation is progressively transformed from the embedding layer through layers \(1,\ldots,L\). Work on transformer representations has already shown that information and anomaly signals can emerge differently at different depths.^[3][4] Layer depth is not physical time, but it does give an ordered sequence of computational transformations.

The second direction is token progression. After an unexpected token appears at position \(t^\star\), the model continues with \(t^\star+1,t^\star+2,\ldots\). Those later token representations are computed using a context that now contains the surprising event. This gives a second axis along which its effect could persist and eventually disappear.

The internal state can therefore be thought of as a two-dimensional object:

where \(\ell\) describes computational depth and \(t\) describes progression through the sequence. Rather than asking for a response as a function of milliseconds, the project will ask whether surprise produces a localized structure somewhere on this layer-by-token plane.

What would “excitation” mean in an LLM?

There is obviously no voltage to measure. A useful operational definition is instead the amount by which a representation changes as computation proceeds. For example, for token \(t\), a simple layerwise update magnitude is

If a surprising token causes unusually large internal revision, one possibility is that \(U_{\ell,t}\) temporarily becomes larger than it does for a closely matched predictable continuation. Other measurements could use changes in feature activations, distances in representation space or changes in the probability distribution decoded from intermediate layers. The important point is not to assume beforehand that any particular activation norm is a P300 equivalent.

A cleaner experiment would compare an unexpected continuation against a matched expected control. If \(t^\star\) is the manipulated token, we could construct a response surface such as

where \(k=0\) measures the unexpected token itself and \(k>0\) follows its effect into subsequent tokens.

A P300-like pattern, in this deliberately loose computational sense, would not merely mean that the unexpected token has high surprisal. The more interesting possibility is a transient response: an unexpected event produces increased representational revision, that revision develops across some portion of the network, affects subsequent processing and then returns toward its ordinary regime.

Surprise versus representational updating

This distinction is important because prediction error and internal updating are not necessarily the same quantity. Human language experiments have connected word surprisal to electrophysiological responses, particularly the N400,^[5] while P300 research has long considered whether unexpected events trigger a broader context-updating process.^[2]

There is also increasing interest in treating transformer hidden states as trajectories rather than isolated vectors. Recent work has examined how deviations from the preceding hidden-state trajectory can contain information that is not captured by token surprisal alone.^[6] That makes the analogy especially interesting: perhaps the relevant quantity is not simply how unlikely a token was, but how strongly that token forces the model away from the representational path it was previously following.

The upcoming experiment

The project will therefore construct controlled sequences in which an expected continuation can be replaced by an increasingly surprising one while keeping the surrounding context as similar as possible. For each condition, I will record the hidden states across both layers and token positions.

The main object of interest will be the resulting layer-by-token response surface. Does the effect appear immediately in shallow layers or only after deeper computation? Is it concentrated on the surprising token, or does it spill into the representations of later tokens? Does larger surprisal produce a larger representational disturbance? And, most importantly, is there evidence for a response that rises and then decays rather than simply remaining as a permanent difference between two sequences?

This would not establish that an LLM literally possesses a P300. Transformers and biological neural systems have very different computational substrates, and layer depth should not be identified with milliseconds of cortical processing. The aim is narrower: to use the P300 as a computational analogy for asking whether prediction violations generate transient, structured changes in the internal dynamics of a language model.

If that analogy is useful, the interesting object may not be a single activation at all. It may instead be the geometry of how an unexpected token perturbs representation over two computational directions: through the layers processing the current token, and through the future tokens that inherit the updated context.