The Verification Bottleneck: Toward Adaptive, Lossless Speculative Decoding for Autoregressive Language Models

Abstract

Speculative decoding accelerates autoregressive language-model inference by proposing tokens with a cheap draft model and verifying them in parallel with the target model, preserving the target output distribution exactly. Most of the literature has optimized the draft: better draft architectures, tree-structured proposals, and reduced draft vocabularies. A recent systematic measurement of speculative decoding on a production engine complicates that focus. It reports that verification by the target model, not drafting, dominates end-to-end execution, that acceptance length varies sharply across token positions, requests, and datasets, and that observed speedups fall well short of the theoretical bound. This survey takes that finding as its organizing question and maps the design space it implies. We formalize speculative decoding as a three-term latency model, separate draft-side from verify-side optimization, and place recent methods within it. On the draft side, we examine the vocabulary projection bottleneck and the move from static reduced vocabularies (FR-Spec, VocabTrim, EAGLE-3) to per-step vocabulary speculation (SpecVocab). On the verify side, we examine semi-autoregressive drafting and load-aware, confidence-scaling verification.