Optimizing LLM Inference: Quantization, Speculative Decoding, and vLLM

Deploying open-weights LLMs like Llama-3, Qwen-2.5, or DeepSeek in production comes with strict latency and memory constraints. Unoptimized PyTorch models on raw GPUs quickly saturate VRAM and bottleneck token throughput.

Here is how we optimize LLM serving infrastructure to maximize tokens per second (TPS) while drastically reducing GPU hardware costs.

PagedAttention and vLLM

The primary memory bottleneck during LLM inference is the Key-Value (KV) cache. Traditional implementations allocate contiguous VRAM blocks for KV cache per request, leading to massive memory fragmentation.

vLLM solves this with PagedAttention, dividing the KV cache into virtual memory pages:

# Launching high-throughput vLLM server with FP8 quantization and tensor parallelism
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --quantization fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Quantization: FP8 vs AWQ vs GGUF

• FP8 (Floating Point 8): Native support on NVIDIA H100 and Ada Lovelace GPUs. Retains near 99.9% FP16 accuracy with 2x lower memory bandwidth usage.
• AWQ (Activation-aware Weight Quantization): Ideal for 4-bit edge or lower VRAM deployments, preserving accuracy by prioritizing salient weight channels.

Speculative Decoding for Latency Reduction

Speculative decoding pairs a small draft model (e.g. Llama-3-8B) with a large target model (e.g. Llama-3-70B). The draft model rapidly proposes token sequences, which the target model verifies in parallel in a single forward pass:

# Speculative decoding setup in vLLM
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    speculative_model="meta-llama/Meta-Llama-3-8B-Instruct",
    num_speculative_tokens=5,
)

Using speculative decoding yields a 1.8x - 2.4x speedup in generation latency without losing model performance.