vLLM 0.33 adds FLy lossy speculative verifier, off by default

vLLM 0.33.0, released on PyPI on 2026-11-02, ships with FLy, a speculative decoding verifier that is available as an opt-in feature. FLy deliberately accepts some draft tokens that the target model would have rejected. The PR states plainly that the verifier does not preserve the target distribution and may reduce output quality. Standard verification remains the default setting, so users have to switch it on explicitly if they want FLy's behavior.

According to the PR, FLy delivers decode throughput of 1.041x to 1.310x compared with standard speculative decoding across all 48 model, platform, and batch combinations tested, with minimum accuracy retention of 96.3%. No combination in the reported data fell below that threshold.

In standard speculative decoding, a smaller drafter proposes K tokens first, and the target model verifies them in a single forward pass. Verification stops at the first rejected token, and the accept/reject rule is constructed so that the output distribution matches what the target model would have produced on its own. FLy loosens that constraint by deliberately admitting draft tokens the target would normally discard, trading strict distributional fidelity for higher decode throughput.

The feature ships in optional mode rather than as a default replacement, and the material provides no further comparative data on FLy versus standard verification beyond the figures stated in the PR. Whether broader benchmarks or real-world deployments behave differently remains unreported here.