Figure 1: 2D projections of the representations learned through perception (UTX, left) or autoregressive (VTX, right) tasks based on OpenELM-450M and the Emotion dataset. In the perception space, joy and sad words are clearly separated, illustrating label-relevant semantic encoding; in the autoregressive space, these two emotion classes overlap, but syntactic categories become clearly clustered.
Abstract
Autoregressive pretraining has become the de facto paradigm for learning general-purpose representations in large language models (LLMs). However, linear probe performance across downstream perception tasks shows substantial variability, suggesting that features optimized for next-token prediction do not consistently transfer well to downstream perception tasks. We demonstrate that representations learned via autoregression capture features that may lie outside the subspaces most informative for perception. To quantify the (mis)alignment between autoregressive pretraining and downstream perception, we introduce the Next Token Perception Score (NTPS) — a score derived under a linear setting that measures the overlap between autoregressive and perception feature subspaces. This metric can be easily computed in closed form from pretrained representations and labeled data, and is proven to both upper- and lower-bound the excess loss. Empirically, we show that NTPS correlates strongly with linear probe accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270M to 8B parameters, confirming its utility as a measure of alignment. Furthermore, we show that NTPS increases following low-rank adaptation (LoRA) fine-tuning, especially in large models, suggesting that LoRA aligning representations to perception tasks enhances subspace overlap and thus improves downstream performance. More importantly, we find that NTPS reliably predicts the additional accuracy gains attained by LoRA finetuning thereby providing a lightweight prescreening tool for LoRA adaptation. Our results offer both theoretical insights and practical tools for analytically assessing LLM perception skills.
Contributions
(1) Systematically evaluating the benefits of autoregressive pretraining by comparing linear-probe performance on six pretrained models against identical architectures trained from scratch across 12 downstream perception datasets.
(2) Proposing the Next Token Perception Score (NTPS): a metric that quantifies the alignment between autoregressive pretraining and downstream perception tasks by measuring the overlap between their respective feature subspaces.
(3) Empirically validating the reliability of NTPS by demonstrating that it not only correlates strongly with linear probe accuracy across 12 diverse datasets and eight pretrained models, but also consistently increases after task-specific LoRA fine-tuning.
(4) Guiding LoRA finetuning with NTPS predictions by demonstrating that NTPS reliably forecasts the additional accuracy gains from LoRA finetuning, thereby providing a lightweight prescreening tool.
