Inference Optimizations
The techniques that make it possible to run powerful LLMs in a more efficient and cost-effective way.
Continuous Batching
Schedules requests at the iteration level instead of waiting for a full static batch, so finished sequences are evicted and new ones join immediately. Keeps the GPU saturated and delivers large throughput gains under concurrent load.
FlashAttention
IO-aware exact attention algorithm that reduces memory reads/writes by tiling and recomputation. Provides 2-4x speedup and enables longer context lengths without approximation.
KV Cache Offloading
Moves KV cache blocks from GPU memory to CPU RAM or local storage when they are not actively needed, then loads them back on demand. Frees up scarce GPU memory to support larger batches and longer contexts.
PagedAttention
Virtual memory-inspired KV cache management that stores cache in non-contiguous blocks to eliminate fragmentation. Enables near-zero memory waste, larger batches, and higher throughput, and is the foundation of vLLM.
Parallelism
Distributes a model and its computation across multiple GPUs and nodes. Data, tensor, pipeline, expert, and hybrid strategies trade off memory, communication, and throughput to serve models too large for a single device.
Prefill-Decode Disaggregation
Splits the compute-bound prefill phase and the memory-bound decode phase onto separate GPU pools so each can be scaled and optimized independently. Reduces interference between the two phases and improves both latency and throughput.
Prompt Caching
Reuses the KV cache of shared prompt prefixes across requests so common system prompts and few-shot examples are computed only once. Dramatically cuts time-to-first-token for workloads with repeated context.
Quantization
Reduces the numerical precision of weights and activations (FP16 to INT8, FP8, or 4-bit) to shrink memory footprint and speed up inference. Methods like AWQ, GPTQ, and GGUF make it possible to serve large models on smaller GPUs with minimal quality loss.
Speculative Decoding
Uses a smaller draft model to generate candidate tokens that the larger model verifies in parallel. Achieves 2-3x faster decoding without any quality loss.