Amazon Unveils P-EAGLE: New Parallel Speculative Decoding Boosts vLLM Performance 1.69x
- What: Amazon has released P-EAGLE (Parallel-EAGLE), a high-speed speculative decoding method for LLM inference.
- Performance: Achieves up to 1.69x speedup over the previous state-of-the-art EAGLE-3 method on NVIDIA B200 hardware.
- Availability: Integrated into vLLM starting with version 0.16.0 (PR#32887).
- Models: Pre-trained P-EAGLE heads are available for GPT-OSS 120B, GPT-OSS 20B, and Qwen3-Coder 30B.
Amazon has launched P-EAGLE, a parallel speculative decoding framework designed to break the throughput ceiling of Large Language Model (LLM) inference. By transitioning from sequential to parallel draft generation, the technology delivers up to a 1.69x performance gain over vanilla EAGLE-3 on real-world workloads using NVIDIA B200 GPUs. The feature is available immediately for developers using the vLLM serving framework via a single configuration toggle.
Breaking the Sequential Bottleneck in Speculative Decoding
Speculative decoding has emerged as a critical technique for reducing LLM latency by using a smaller, faster "drafter" model to predict multiple future tokens, which are then verified in parallel by the larger "target" model. While the original EAGLE method is widely considered the industry standard—offering 2x to 3x speedups over standard autoregressive decoding—it suffers from a hidden architectural bottleneck.
According to the Amazon announcement, standard EAGLE is inherently autoregressive. To produce $K$ draft tokens, the system must perform $K$ sequential forward passes through the drafter model. As drafter models become more sophisticated and aim for longer speculation depths, this overhead scales linearly, eventually eating into the performance gains provided by the target model’s speed.
P-EAGLE (Parallel-EAGLE) removes this limitation by transforming the drafting process. Instead of sequential steps, it generates all $K$ draft tokens in a single forward pass. This shift allows the system to speculate more aggressively without the "latency tax" associated with deeper speculation.
Technical Architecture: Multi-Token Prediction (MTP)
The core innovation of P-EAGLE lies in its two-step architecture. The process begins with a "Prefilling" stage where the target model processes the initial prompt and captures internal hidden states ($h_prompt$ and $h_context$). These states encode the model’s "knowledge" at each position, which then serves as a guide for the drafter.
In the second stage, the P-EAGLE drafter constructs inputs for all positions simultaneously. For the first position (Next-Token-Prediction), the drafter uses actual token embeddings and context from the target model. However, for subsequent positions where the required token embeddings do not yet exist, P-EAGLE utilizes two learnable parameters:
- A shared mask token embedding: A fixed vector that acts as a placeholder for unknown future tokens.
- A shared hidden state: A neutral placeholder learned during training.
These inputs pass through the transformer layers and the language model head in parallel, predicting all draft tokens (e.g., $t1, t2, t3, t4$) at once. This parallelization is what allows P-EAGLE to maintain high speeds even as the number of speculative tokens increases.
Integration and Benchmarks in vLLM
Amazon has worked to integrate P-EAGLE directly into vLLM, one of the most popular open-source LLM serving libraries. Starting with vLLM version 0.16.0, users can enable parallel drafting by adding a simple parameter to their serving pipeline configuration.
Benchmarks conducted on NVIDIA B200 GPUs demonstrate the efficiency of this approach. On real-world workloads—including MT-Bench, HumanEval, and SpeedBench—P-EAGLE achieved speedups ranging from 1.05x to 1.69x over vanilla EAGLE-3 when running GPT-OSS 20B.
The company also addressed the necessity of training these models on relevant data. Modern reasoning models often produce long outputs; for instance, GPT-OSS 120B shows a median sequence length of nearly 3,900 tokens. To be effective, Amazon trained P-EAGLE drafters on matching context lengths to ensure they remain accurate during long-form generation.
Impact on the AI Industry
For developers and enterprises, the release of P-EAGLE represents a significant shift in the economics of LLM inference. By reducing the number of forward passes required for drafting, P-EAGLE lowers the compute cost per token while simultaneously decreasing user-facing latency.
In a competitive landscape where frameworks like SGLang and NVIDIA’s TensorRT-LLM are constantly vying for speed supremacy, the integration of P-EAGLE gives vLLM a distinct advantage in speculation efficiency.
"P-EAGLE effectively removes the 'latency tax' of speculation depth, allowing models to think further ahead without slowing down," according to the technical details provided in the announcement. This capability is particularly vital for agentic workflows and complex reasoning tasks where long, accurate sequences are required quickly.
What’s Next for P-EAGLE
Amazon has made the artifacts for P-EAGLE publicly available to encourage immediate adoption. Pre-trained drafter heads are currently hosted on HuggingFace for several major model families, including GPT-OSS (20B and 120B) and the Qwen3-Coder 30B series.
Developers can start using P-EAGLE by updating to the latest version of vLLM and adjusting their SpeculativeConfig to include "parallel_drafting": true. As more pre-trained drafter heads are released, the industry can expect a broader range of models to benefit from this parallelized speed boost.
Sources
All technical specifications, pricing, and benchmark data in this article are sourced directly from official announcements. Competitor comparisons use publicly available data at time of publication. We update our coverage as new information becomes available.