A model can score as if it matches the brain even after the brain information has been destroyed. And a language pattern can look real when the test measuring it is failing. Similarity scores have become a common way to relate models to one another and to the brain.

They are also used to look for shared structure across languages. The tempting step is to treat measured correspondence as evidence of a meaningful property that could be useful. If that correspondence is real, it might be turned into training guidance that improves a model, or into a control that changes its behavior.

But that promise depends on the score measuring what people think it measures. The central warning is simple: in neither setting does the score mean what its size suggests. So in neither setting can the score be interpreted from its magnitude alone; its size does not tell us what the score means.

The audit has three parts. First, the reference is rebuilt so it keeps incidental properties but loses the correspondence itself. Second, the measuring instrument is checked: a test that fails in one language cannot support a claim about that language’s transfer, especially when its reliability changes along the same path as the result.

Third, the units are counted correctly, because pairwise designs can create many observations from only a few entities; the entities, not the pairs, must be used for the final check. A transfer result is only interpretable where the test being transferred from works.

In four languages, that test did no better than guessing in its own language. Those failures were not scattered at random: they followed a pattern across languages rather than appearing independently across the sample.

A language’s test became worse as it differed more from the rest of the sample, showing a strong negative association between difference and test performance. Reliability and the result being studied therefore changed together: probe reliability and the predictor of interest co-varied in the analysis.

When the analysis was restricted to languages whose tests worked at both ends, the explained variation was cut in half. The direction remained more convincing than the size of the effect. The intervention itself showed no trend with distance. Its raw benefit had no relationship with distance.

Random-direction and per-target baselines produced only a tentative apparent relationship, suggesting any remaining pattern might reflect general perturbability rather than a robust effect. After languages rather than language pairs were treated as the units, the corrected relationship did not survive: the correlation was minus zero point zero nine, with p equal to zero point seven zero four.

The brain-alignment result has the same kind of trap. Targets whose correspondence to the brain was destroyed still recovered between eighty-three and ninety-two percent of the brain-aligned score. The part specifically attributable to brain content was only the small increment above those controls, and the score was dominated by something else.

The measure also returned a large number when a shuffled target was compared with the real one, even though there was no correspondence at all. The score’s size therefore included a built-in overlap unrelated to the brain.

The visible pattern is a small blue-bar advantage over two nearly matching controls, while the dashed reliability limit remains clearly above all three. The practical test reached the same conclusion: brain alignment did not improve the model’s language performance.

It was not significantly above the ordinary training condition or either control. The result repeated at larger model sizes: brain alignment did not improve BLiMP, and it was never significantly above LM only in any tested scale.

Ordinary fine tuning did move performance, with BLiMP changing by three to six points, showing the training regime was active. Across both settings, the underlying phenomena may be real, but the raw scores do not support their size. In the cross-language case, the design cannot recover the magnitude at all.

The checks were inexpensive, and each changed what the score could honestly mean. Without them, a correspondence score is a number whose units are unknown. The safe question comes first: what does a correspondence measure read when the correspondence it is meant to detect is removed, and how much of the score remains?

A high correspondence score is not automatically evidence of meaningful shared structure. Before using one to guide a model or explain a result, test what score remains when the intended information is removed.