Inference Engines

C/C++ inference of LLMs with minimal dependencies. Supports GGUF quantization, runs on CPU and Apple Silicon with optional GPU offloading. The backbone of local LLM inference.

Toolkit for compressing, deploying, and serving LLMs from the InternLM team. Its TurboMind engine delivers high request throughput with persistent batching, blocked KV cache, and weight-only quantization.

Modular's high-performance inference framework and serving platform. Pairs a Mojo-powered kernel stack with an OpenAI-compatible server for fast, portable LLM serving across NVIDIA and AMD GPUs.

Universal LLM deployment engine built on Apache TVM. Compiles and runs models natively across GPUs, browsers (WebGPU), iOS, and Android for high-performance inference everywhere.

Run LLMs locally with a simple CLI. Wraps llama.cpp with model management, an HTTP API, and one-command model downloads. The easiest way to get started with local models.

Fast serving framework with RadixAttention for automatic KV cache reuse across requests. Features a frontend language for complex LLM programs with parallelism.

NVIDIA's optimized inference library for LLMs on NVIDIA GPUs. Leverages TensorRT for kernel fusion, in-flight batching, and FP8 quantization for maximum throughput.

Production-ready inference server for LLMs. Features continuous batching, Flash Attention, quantization support (GPTQ/AWQ), and seamless Hugging Face Hub integration.

Speed-of-light LLM inference engine built for agentic workloads, pairing TensorRT-LLM-level performance with vLLM-level ease of use. Features a static compiler, an optimized MLA kernel for Blackwell, and FP4 inference on NVIDIA and AMD.

High-throughput LLM serving engine featuring PagedAttention for efficient memory management. Supports continuous batching, tensor parallelism, and OpenAI-compatible API.