Skip to main content

Speculative Decoding

Overview​

Speculative decoding (also referred to as assisted-generation) is a latency-oriented optimization for autoregressive text generation. A small, cheap drafter proposes one or more candidate continuations, and the larger target (main) model validates them in a single forward pass. Tokens that the target would have produced anyway are accepted; the first divergent token is replaced by the target's own sample, and drafting resumes from there.

Because validation is parallel over the candidate length, each accepted draft token replaces a sequential decode step of the target model with (a fraction of) one batched step. When acceptance is high, this yields end-to-end speedups on memory-bandwidth-bound decoding without changing the model output distribution in theory (the standard verification scheme is statistically equivalent to sampling from the target). In practice, the target scores several tokens in one pass, which can round differently from decoding one token at a time, so greedy output can differ from the target alone where two tokens are nearly tied.

OpenVINO GenAI implements five drafting strategies. They differ only in how candidates are produced — the target-side validation step is shared.

StrategyDrafterBest ForContinuous Batching (CPU/GPU)Stateful (NPU)
Speculative Decoding (Fast Draft)Smaller off-the-shelf LLMGeneral-purpose, robust speedup without retraining✓✓
Prompt Lookup DecodingN-gram match against the promptOutput copies large spans of the input (RAG, summarization, code editing)✓✗
EAGLE3Custom draft head trained on the target's hidden statesHighest acceptance rate; also supports tree drafting✓✓
DFlashBlock diffusion draft model trained on the target's hidden statesLong candidate blocks drafted in one forward pass✓✗
DSparkDFlash draft with a Markov head that makes each drafted token depend on the one beforeLong candidate blocks whose tokens depend on each other✓✗
info

All five strategies are available through the LLMPipeline Python and C++ APIs. Prompt Lookup Decoding, EAGLE3, DFlash and DSpark are additionally supported in VLMPipeline (e.g. Qwen3-VL, Qwen3.5, MiniCPM-V 4.7).

The Continuous Batching backend supports multi-request batching; the Stateful backend is single-request.

Every strategy is configured in two places: the drafter is selected when the pipeline is constructed, and the drafting behavior is tuned per request through GenerationConfig. The shared knob is num_assistant_tokens — the number of candidate tokens submitted to the target per iteration, defaulting to 5 when unset, max 100. The remaining knobs are strategy-specific and described in the corresponding sections below.

Measuring Effectiveness

Inspect result.extended_perf_metrics, which exposes main_model_metrics, draft_model_metrics and get_num_accepted_tokens().

The single most useful diagnostic is get_num_accepted_tokens() divided by the number of target iterations — the average accepted tokens per step. Aim for this to be well above 1; values close to 1 mean drafting is not paying for itself.

Prefix Caching with Speculative Decoding​

On the Continuous Batching backend, enable SchedulerConfig.enable_prefix_caching to reuse cached prefixes across requests with shared prompts. For hybrid-attention models, reuse also depends on linear-attention state checkpoints; see Hybrid Attention Model Cache Management for checkpoint granularity and memory-budget guidance.

With MTP, DFlash and DSpark drafting, the main and draft models must use the same enable_prefix_caching setting. The pipeline restores both models to a common reusable prefix before processing a request.

Speculative Decoding (Fast Draft)​

Fast Draft is the classic two-model setup: a smaller off-the-shelf LLM that shares the target's tokenizer drafts tokens autoregressively, and the target verifies them. It works with any target/draft pair you already have, without retraining. The speedup is bounded by how often the small model's distribution agrees with the large one.

Model Preparation​

Export both models with optimum-cli. The draft model must use the same tokenizer as the target:

optimum-cli export openvino --model Qwen/Qwen3-8B --weight-format int4 --task text-generation-with-past Qwen3-8B-ov-int4
optimum-cli export openvino --model Qwen/Qwen3-0.6B --weight-format int4 --task text-generation-with-past Qwen3-0.6B-ov-int4

Configuration and Usage​

In addition to num_assistant_tokens, Fast Draft supports:

  • assistant_confidence_threshold (float) — a dynamic-length stopping criterion. The drafter keeps proposing while the candidate token probability exceeds this threshold, then hands off to the target. It is mutually exclusive with num_assistant_tokens.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-0.6B-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "CPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "CPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5
# Or, on the Continuous Batching backend, dynamic-length drafting:
# config.assistant_confidence_threshold = 0.4

result = pipe.generate("The Sun is yellow because", config)
tip

Increase num_assistant_tokens until the accepted-tokens-per-step figure plateaus, then back off. Past the plateau, rejected draft tokens are pure overhead.

Backend Differences
  • Continuous Batching uses num_assistant_tokens as-is and runs the drafter and target as fully scheduled paged-attention pipelines, each with its own KV cache. The target and draft caches share the configured SchedulerConfig.cache_size, so increase it when the drafter is large.
  • Stateful uses num_assistant_tokens as the initial value and adapts it per step based on recent acceptance, falling back gracefully when acceptance is low.
warning

assistant_confidence_threshold is supported on the Continuous Batching backend only.

The Stateful backend requires NPU to be the execution device for at least one of the two models. For NPU deployments, the recommended configuration places both the target and the draft model on the NPU.

tip

Check out Python and C++ speculative decoding samples.

Prompt Lookup Decoding​

Prompt Lookup decoding replaces the draft model with a string match: the most recently emitted suffix is searched as an n-gram inside the prompt, and the tokens that follow each match are proposed as candidates. No second model is loaded.

This is highly effective for input-grounded generation — RAG, document QA, summarization, code editing, multi-turn chat — where the output frequently copies entity names, phrases or code chunks verbatim from the input. On these workloads it delivers speculative-decoding-class speedups at near-zero drafter cost. On workloads with low prompt/output overlap it degrades gracefully back to plain decoding, since the proposals are simply rejected.

Configuration and Usage​

Prompt Lookup is enabled at pipeline construction time with prompt_lookup=True; it cannot be switched on per request. In addition to num_assistant_tokens it supports:

  • max_ngram_size (size_t) — the maximum n-gram length matched against the prompt. Larger values produce longer, more confident continuations when they hit, but match less often.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"

pipe = openvino_genai.LLMPipeline(main_model_dir, "CPU", prompt_lookup=True)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5
config.max_ngram_size = 3

pipe.generate("The Sun is yellow because", config)
warning

Prompt Lookup is available on the Continuous Batching backend only.

The draft_model and prompt_lookup constructor options are mutually exclusive — Prompt Lookup uses no draft model.

tip

Check out Python and C++ prompt lookup decoding samples.

EAGLE3​

EAGLE-3 (Extrapolation Algorithm for Greater Language-model Efficiency) replaces the generic draft model with a small head — typically one transformer layer — trained to mimic the target's distribution conditioned on the target's hidden states, not just on tokens. This much tighter coupling raises the per-token acceptance rate substantially over Fast Draft on the same target, while keeping the drafter cheap.

EAGLE3 can produce candidates either as a linear chain or as a tree.

Model Preparation​

Export both the target and the matching EAGLE3 draft head with optimum-cli. The draft repository must be an EAGLE3 head trained against the same target family — the head exposes a d2t mapping table that OpenVINO GenAI picks up automatically:

optimum-cli export openvino --model Qwen/Qwen3-8B --weight-format int4 --task text-generation-with-past --trust-remote-code Qwen3-8B-ov-int4
optimum-cli export openvino --model AngelSlim/Qwen3-8B_eagle3 --weight-format int4 --task text-generation-with-past --trust-remote-code Qwen3-8B_eagle3-ov-int4
Chain Or Tree?

Start with chain drafting: it has a single knob, and on a well-matched EAGLE3 head it already captures most of the available speedup.

Switch to tree drafting when acceptance per step has plateaued but the target still has spare compute per forward pass — typically on GPU at small batch sizes, where the target's step time grows sublinearly with the candidate count. Tree drafting trades a larger validation batch for a higher accepted-token count per step.

Chain Drafting​

Chain drafting runs the EAGLE3 head autoregressively for num_assistant_tokens iterations and submits the resulting linear chain to the target for verification. It needs no configuration beyond num_assistant_tokens.

import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-8B_eagle3-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5

pipe.generate("What is OpenVINO?", config)

Tree Drafting​

Instead of a single linear chain, the drafter expands the top branching_factor continuations at each of tree_depth layers, scores the resulting tree, and submits the highest-scoring num_assistant_tokens candidates to the target in one packed verification step. With a single target forward pass the pipeline can therefore validate multiple alternative continuations and accept the longest matching one, compounding EAGLE3's already-high acceptance rate into longer accepted runs.

Tree drafting adds two GenerationConfig fields:

  • branching_factor (size_t) — number of top-k expansions retained per tree node and per tree layer.
  • tree_depth (size_t) — number of draft iterations, i.e. tree layers. Setting tree_depth > 0 switches the request from chain drafting to tree drafting.
How Tree Drafting Works
  1. The EAGLE3 head runs tree_depth iterations. At each layer it expands the top branching_factor children of every surviving node, producing up to branching_factor^2 * (tree_depth - 1) + branching_factor candidate tokens organized as a tree.
  2. Candidates are scored cumulatively along their tree paths; the top num_assistant_tokens are flattened into a single packed input together with a tree attention mask and tree position ids.
  3. The target processes the packed input in one forward pass and verifies all paths simultaneously. The longest path whose tokens match the target's own samples is accepted.
  4. Accepted tokens become the new prefix, the draft and target KV caches are advanced to match, and the next round begins.

Two constraints follow from this, and both are enforced by GenerationConfig::validate():

  • The tree must be able to supply the requested number of candidates: branching_factor^2 * (tree_depth - 1) + branching_factor >= num_assistant_tokens.
  • num_assistant_tokens >= tree_depth, since an accepted path is at most tree_depth tokens long.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-8B_eagle3-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 15  # top-K candidates verified per step
config.branching_factor = 8       # top-k expansions per tree node
config.tree_depth = 4             # number of draft layers

pipe.generate("What is OpenVINO?", config)
tip

The trio (branching_factor, tree_depth, num_assistant_tokens) is the main tuning surface. A reasonable starting point on a high-acceptance target is branching_factor=4..8 and tree_depth=3..4, with num_assistant_tokens chosen to match how many candidate tokens the target can verify per step on your hardware.

warning

Tree drafting is an EAGLE3-only feature; Fast Draft and Prompt Lookup ignore branching_factor and tree_depth.

It cannot be combined with beam search or multinomial sampling — set tree_depth=0, do_sample=false or num_beams=1 accordingly.

EAGLE3 Streaming

Streaming is restricted to batch size 1, with either greedy decoding or tree search. Beam search and multinomial sampling are not supported.

assistant_confidence_threshold must be 0 for EAGLE3 — dynamic-length drafting is a Fast Draft feature.

DFlash​

DFlash drafts with a small block diffusion model. Like EAGLE3, the draft is trained against a specific target and reads the target's hidden states. Unlike EAGLE3, it does not run once per candidate: it fills a whole block of masked positions in a single forward pass, so drafting more candidates costs little extra draft time.

Model Preparation​

Export the target and the matching DFlash draft with optimum-cli. The draft must be trained against the same target and runs with the target's tokenizer. OpenVINO GenAI reads the DFlash metadata stored in the exported draft and selects DFlash automatically:

optimum-cli export openvino --model Qwen/Qwen3.5-9B --weight-format int4 Qwen3.5-9B-ov-int4
optimum-cli export openvino --model z-lab/Qwen3.5-9B-DFlash --weight-format int4 --task text-generation-with-past --trust-remote-code Qwen3.5-9B-DFlash-ov-int4

Configuration and Usage​

Per request, DFlash needs no configuration beyond num_assistant_tokens, the number of candidates the draft fills in each step. Setting it to 0 runs the request on the target alone.

import openvino_genai

main_model_dir = "Qwen3.5-9B-ov-int4"
draft_model_dir = "Qwen3.5-9B-DFlash-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.VLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 7

pipe.generate("What is OpenVINO?", generation_config=config)

Choosing the Block Size​

A DFlash draft is trained with a fixed block size, which the model provider usually states. OpenVINO GenAI does not check num_assistant_tokens against the draft's block size, only against the general limit of 100, but staying within the trained block size is recommended.

tip

Tune num_assistant_tokens on prompts that resemble your use case, balancing the accepted tokens per step against the step latency of the target and the draft. As the block grows, the accepted tokens per step rise and then plateau, while the step latency keeps growing, so past some point a larger block slows generation down.

Runtime limits

DFlash uses the Continuous Batching backend on CPU and GPU; NPU and the Stateful backend are not supported.

  • Greedy decoding only, with one return sequence per request. Beam search, adapters, prompt lookup, and assistant_confidence_threshold are not supported.
  • Cache eviction (SchedulerConfig.use_cache_eviction) is not supported for the target or the draft.
  • With a VLM target and prefix caching, ChatHistory and image add_request() requests need num_assistant_tokens > 0.
  • Omni intermediate outputs (return_omni_outputs) are not supported.
note

DFlash drafts exported for OpenVINO GenAI 2026.4 or earlier do not load. Re-export them with optimum-intel 2.3.0 or later.

DSpark​

DSpark extends DFlash with a Markov head. A DFlash draft predicts every position of a block in one pass, without seeing the tokens it drafts at the other positions. When the context allows several continuations, such as "of course" and "no problem", it can combine them into sequences such as "of problem" that the target rejects, so acceptance drops at later positions of the block. The Markov head is a small sequential stage after the parallel draft: it biases every drafted position by the token drafted before it, so the tokens of a block follow from each other.

DSpark drafts run through the DFlash pipeline, so the DFlash section, including its runtime limits, applies to them as well.

Model Preparation​

Export the target and the matching DSpark draft with optimum-cli:

optimum-cli export openvino --model Qwen/Qwen3-4B --weight-format int4 Qwen3-4B-ov-int4
optimum-cli export openvino --model deepseek-ai/dspark_qwen3_4b_block7 --weight-format int4 --task text-generation-with-past Qwen3-4B-DSpark-ov-int4

The draft backbone is exported as openvino_model.xml and the Markov head next to it as openvino_markov_head_model.xml; draft_model reads both from the draft folder. --weight-format applies only to the draft backbone; the Markov head is not compressed.

Configuration and Usage​

As for DFlash, set num_assistant_tokens per request. A DSpark draft also takes the markov_head_top_k draft property; see Markov Head Modes.

import openvino_genai

main_model_dir = "Qwen3-4B-ov-int4"
draft_model_dir = "Qwen3-4B-DSpark-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 7

pipe.generate("What is OpenVINO?", config)

Markov Head Modes​

draft_model loads the Markov head next to the draft backbone and runs it on the draft device, in one of two modes selected by the markov_head_top_k draft property:

  • Top-K candidate scorer (the default): keeps the K best backbone candidates at every drafted position and scores every pair of neighboring candidates with the Markov head, in one inference for all scheduled requests. By default, K is the value stored in the exported Markov head: 64 for drafts exported with optimum-intel, or the draft vocabulary size if it is smaller. Set markov_head_top_k to another K between 1 and the draft vocabulary size to change it. The pipeline warns for a K of 1, where the Markov head has no effect, and for a K below 8, where fewer draft tokens may be accepted.
  • Dense Markov walk (markov_head_top_k = 0): corrects every drafted position with the Markov head, one inference per position.

The Top-K candidate scorer runs the Markov head once per step instead of once per drafted position, so it is usually faster. It drafts the same tokens as the dense walk up to floating-point rounding, except where the walk would pick a token outside the K candidates. The target verifies every drafted token either way, so the mode does not change the output. draft_model rejects markov_head_top_k for a draft without a Markov head.

# 16 candidates per drafted position; markov_head_top_k=0 selects the dense Markov walk of a DSpark draft
draft_model = openvino_genai.draft_model(draft_model_dir, "GPU", markov_head_top_k=16)

Candidate Selection​

  • With the Top-K candidate scorer, the draft walks the candidates position by position, taking the best-scoring candidate after the previous drafted token. When the request's penalties or min_new_tokens can change the best token, the draft sampler applies them to the candidate scores, and a drafted token outside the candidates ends the block.
  • With the dense Markov walk, the draft takes every drafted token from the backbone logits corrected by the Markov head, after the request's logit processors.