Speculative decoding for faster code completion

Most code completion feels slow not because the model is slow, but because you're waiting for a 70B-parameter beast to generate tokens one at a time while your cursor blinks. Speculative decoding flips that dynamic: a tiny draft model races ahead with plausible completions, then a large verifier accepts or rejects entire chunks in parallel. When it works, you get multi-token-per-step throughput with the accuracy of the big model. When it doesn't, you've added latency for no gain.

This post walks through how speculative decoding works for code, why it's harder than prose, and the engineering choices that determine whether you see 2× faster completions or a system that's slower than baseline.

How speculative decoding works

Traditional autoregressive sampling generates one token at a time. The model computes logits for position n, you sample a token, append it to the context, then feed the entire sequence back in to get position n+1. Memory bandwidth dominates: you're loading multi-gigabyte weight tensors from VRAM for every single token.

Speculative decoding introduces a draft-verify loop:

1. A small, fast draft model (1B–7B parameters) generates k candidate tokens autoregressively.

2. The large target model evaluates all k candidates in a single forward pass, producing logits at each position.

3. You compare draft probabilities to target probabilities at each step. Accept tokens while the target distribution would have sampled the same token (with some probability threshold). Reject at the first mismatch and resample from the target's corrected distribution.

4. Repeat from the last accepted position.

The key insight: verifying k tokens in parallel is faster than generating them one-by-one if the draft model is cheap enough and accurate enough. You're trading off draft model cost and acceptance rate.

For code, the economics get tricky. A 1B draft model might accept 60–70% of tokens on Python docstrings or boilerplate imports, but drop to 30–40% on gnarly async Rust or heavily templated C++. The target model's forward pass isn't free—if you're only accepting one or two tokens per verification, you've added overhead.

Why code breaks the usual assumptions

Most speculative decoding benchmarks cite 2–3× speedups on natural language tasks like summarization or chat. Code is harder:

Vocabulary and distribution skew. Code has long identifiers, punctuation-heavy syntax, and rare tokens (::, ->, @dataclass) that small draft models underrepresent. A draft model trained on broad code corpora might confidently suggest self.data when the target model expects self._internal_state_machine. The mismatch kills acceptance.

Context length and structural dependencies. Completing a function in the middle of a 2,000-line file means the draft model needs to track imports, class hierarchies, and prior method signatures. Smaller models with 2K–4K context windows will hallucinate plausible-but-wrong names. The target model with 16K context catches the error, rejects, and you've burned cycles.

Branching factor on syntax. Code has narrow valid continuations (after def foo(, you need a parameter name or ), not arbitrary prose). Draft models often agree on syntax tokens ((, ,, :) but diverge on semantic ones (variable names, types). High acceptance on boilerplate, low acceptance where it matters.

Temperature and sampling. Code completion often uses low temperature (0.2–0.4) or even greedy sampling to avoid hallucinations. Speculative decoding's acceptance criteria assume you're comparing probability distributions; at temperature near zero, a tiny logit difference between draft and target becomes a rejection. You need careful tuning or modified acceptance rules (e.g., accept if draft token is in target's top-k).

Picking a draft model

You want a model that's 5–10× faster than your target but still trained on code-heavy data. A few patterns we've seen work:

Distilled variants of the target. If your target is a 70B code model, a 7B or 13B distilled version of the same architecture shares vocabulary, tokenization, and training distribution. Acceptance rates stay higher because the draft isn't guessing from a different prior.

Architecture tricks for speed. Some teams use draft models with fewer attention heads, grouped-query attention, or shallower layers. A 3B model with GQA can hit 2–3× the throughput of a standard 3B dense transformer, which matters when you're generating k=4–8 draft tokens per step.

Fine-tuning on rejection examples. After running speculative decoding in production, collect (context, draft tokens, target tokens) triples where the draft was rejected. Fine-tune the draft model on these hard negatives. You're explicitly teaching it to mimic the target's distribution on cases where they disagree.

Avoid the temptation to use a general-purpose small model (a 1B chat model, for instance) as your draft. The vocabulary mismatch and lack of code-specific pretraining will tank acceptance rates below 40%, and you'll lose performance.

Tuning k and acceptance criteria

The draft length k is a trade-off curve, not a single magic number:

• k=2–3: You're barely amortizing the target model's forward pass. Acceptance rates need to be 70%+ or you're slower than baseline.
• k=4–6: Sweet spot for many code tasks. Even 50% acceptance gives you a multi-token win per step.
• k=8+: Only viable if the draft model is extremely well-aligned and you're completing predictable code (test boilerplate, schema definitions).

Run offline benchmarks on representative completions from your domain. Measure wall-clock time for 100-token completions at different k values. Don't optimize acceptance rate in isolation—optimize accepted tokens per second.

For acceptance, the standard rule is: accept token t if p_target(t) ≥ p_draft(t). In code, we've seen gains from relaxing to "accept if t is in the target's top-3 and p_target(t) > threshold". This tolerates small logit disagreements on equivalent variable names (data vs. buffer) without rejecting the entire draft.

Integration with compile-first workflows

At Goatfied, we run every completion through a plan → constrain → edit → validate loop. Speculative decoding slots into the edit phase, but the validate phase (compile, lint, test) becomes a second verification step.

Here's the flow:

1. Draft model proposes a 6-token completion: async def fetch_user(user_id: int.

2. Target model verifies and accepts 5 tokens, rejects : int and resamples ) -> User:.

3. The completion is inserted into the file and sent to the validation step: ruff check, mypy, and compilation.

4. If validation fails (e.g., User isn't imported), the agent loop retries with the error message in context.

Speculative decoding buys you faster sampling, but doesn't eliminate validation. In fact, faster sampling makes the cost of validation more visible—if you can generate 10 completions in the time it previously took to generate 3, the bottleneck shifts to linting and testing.

We've found that batching validations (run ruff once on 5 candidate completions rather than serially) and caching negative results (don't re-validate identical diffs) helps more than further tuning draft models once you're above 50% acceptance.

When speculative decoding isn't worth it

A few cases where the complexity outweighs the gain:

Short completions. If you're completing 10–20 tokens (a single line), the overhead of loading two models and running the draft-verify loop is comparable to just running the target. Use speculative decoding for multi-line blocks, function bodies, or file-level generation.

Highly exploratory sampling. If you're generating multiple diverse candidates (temperature 0.8+, top-p sampling), the draft and target will disagree frequently. Speculative decoding assumes you want a single high-probability continuation.

Memory-constrained environments. Running two models simultaneously (even if one is small) increases peak VRAM. On a single GPU, you might hit OOM or force smaller batch sizes, losing throughput. Profile carefully.

Cold-start latency. Loading both models adds initialization time. For one-off completions (e.g., CLI code generation), the amortization doesn't pay off. Reserve speculative decoding for interactive sessions where you're generating dozens of completions.

Measuring in production

Don't rely on synthetic benchmarks. Instrument your completion pipeline to log:

• Acceptance rate per draft attempt (rejected at position 1 vs. position 5 tells you different things).
• Wall-clock time from request to first token and to completion.
• Draft model inference time vs. target model verification time.
• Cache hit rates (if you're caching KV for the target model across steps).

We've seen acceptance rates vary by 20+ percentage points between Python data pipelines and Rust async code in the same codebase. Global averages hide the variance that matters.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose