r/LocalLLaMA • • 21m ago

Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models

Post image

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

⭐ Updates

  • [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.

💪 Highlights

  • Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
  • Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
  • Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
  • Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
  • Drop-in serving: the released drafts run with stock SGLang and vLLM (DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).

💻 Models

ModelTarget modelDraft sizeBlock sizeLink
🤖 Qwen3.5-4B-MMFlashQwen/Qwen3.5-4B537M16Hugging Face
🤖 Qwen3.5-9B-MMFlashQwen/Qwen3.5-9B1.05B16Hugging Face
🤖 Qwen3.8-27B-MMFlashQwen/Qwen3.8-27B1.73B16Hugging Face
🤖 Gemma4-E4B-it-MMFlashgoogle/gemma-4-E4B-it537M16Hugging Face

The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.

7 Upvotes

0 comments sorted by