The Open Source
LLM Stack
A curated list of the latest open source LLMs, inference engines, optimizations, and agentic frameworks.
Latest Open Source LLMs
2026-04
- Parameters
- 1.6T (49B active)
- Architecture
- MoE
- Context window
- 1M
- Modality
- Text
- License
- MIT
- Recommended Hardware
- 8× H2008× B200
Run it with
2026-04
- Parameters
- 1.6T (49B active)
- Architecture
- MoE
- Context window
- 1M
- Modality
- Text
- License
- MIT
- Recommended Hardware
- 8× H2008× B200
Run it with
Inference Optimizations
Continuous Batching
Schedules requests at the iteration level instead of waiting for a full static batch, so finished sequences are evicted and new ones join immediately. Keeps the GPU saturated and delivers large throughput gains under concurrent load.
FlashAttention
IO-aware exact attention algorithm that reduces memory reads/writes by tiling and recomputation. Provides 2-4x speedup and enables longer context lengths without approximation.
KV Cache Offloading
Moves KV cache blocks from GPU memory to CPU RAM or local storage when they are not actively needed, then loads them back on demand. Frees up scarce GPU memory to support larger batches and longer contexts.
PagedAttention
Virtual memory-inspired KV cache management that stores cache in non-contiguous blocks to eliminate fragmentation. Enables near-zero memory waste, larger batches, and higher throughput, and is the foundation of vLLM.
Parallelism
Distributes a model and its computation across multiple GPUs and nodes. Data, tensor, pipeline, expert, and hybrid strategies trade off memory, communication, and throughput to serve models too large for a single device.
Prefill-Decode Disaggregation
Splits the compute-bound prefill phase and the memory-bound decode phase onto separate GPU pools so each can be scaled and optimized independently. Reduces interference between the two phases and improves both latency and throughput.
Prefix Caching
Reuses the KV cache of shared prompt prefixes across requests so common system prompts and few-shot examples are computed only once. Dramatically cuts time-to-first-token for workloads with repeated context.
Quantization
Reduces the numerical precision of weights and activations (FP16 to INT8, FP8, or 4-bit) to shrink memory footprint and speed up inference. Methods like AWQ, GPTQ, and GGUF make it possible to serve large models on smaller GPUs with minimal quality loss.
Speculative Decoding
Uses a smaller draft model to generate candidate tokens that the larger model verifies in parallel. Achieves 2-3x faster decoding without any quality loss.
Continuous Batching
Schedules requests at the iteration level instead of waiting for a full static batch, so finished sequences are evicted and new ones join immediately. Keeps the GPU saturated and delivers large throughput gains under concurrent load.
FlashAttention
IO-aware exact attention algorithm that reduces memory reads/writes by tiling and recomputation. Provides 2-4x speedup and enables longer context lengths without approximation.
KV Cache Offloading
Moves KV cache blocks from GPU memory to CPU RAM or local storage when they are not actively needed, then loads them back on demand. Frees up scarce GPU memory to support larger batches and longer contexts.
PagedAttention
Virtual memory-inspired KV cache management that stores cache in non-contiguous blocks to eliminate fragmentation. Enables near-zero memory waste, larger batches, and higher throughput, and is the foundation of vLLM.
Parallelism
Distributes a model and its computation across multiple GPUs and nodes. Data, tensor, pipeline, expert, and hybrid strategies trade off memory, communication, and throughput to serve models too large for a single device.
Prefill-Decode Disaggregation
Splits the compute-bound prefill phase and the memory-bound decode phase onto separate GPU pools so each can be scaled and optimized independently. Reduces interference between the two phases and improves both latency and throughput.
Prefix Caching
Reuses the KV cache of shared prompt prefixes across requests so common system prompts and few-shot examples are computed only once. Dramatically cuts time-to-first-token for workloads with repeated context.
Quantization
Reduces the numerical precision of weights and activations (FP16 to INT8, FP8, or 4-bit) to shrink memory footprint and speed up inference. Methods like AWQ, GPTQ, and GGUF make it possible to serve large models on smaller GPUs with minimal quality loss.
Speculative Decoding
Uses a smaller draft model to generate candidate tokens that the larger model verifies in parallel. Achieves 2-3x faster decoding without any quality loss.
Why Optimizations Matter
Running LLMs at scale requires clever engineering. From quantization that shrinks model weights to attention mechanisms that reduce memory usage, these techniques make it possible to serve powerful models on commodity hardware.
Hottest Agentic Frameworks
Agent Development Kit
Open source toolkit from Google for building, evaluating, and deploying AI agents. Model-agnostic and optimized for Gemini, with multi-agent orchestration, tools, and tight Vertex AI integration.
Agno
Agno
High-performance, full-stack framework for building multi-agent systems with memory, knowledge, and reasoning. Model-agnostic with built-in tools, structured outputs, and a path to production (formerly Phidata).
AutoGen
Microsoft
Multi-agent conversation framework enabling agents to chat with each other to solve tasks. Supports customizable agents, human participation, and diverse conversation patterns.
CrewAI
CrewAI
Role-based multi-agent framework where you define agents with specific roles, goals, and backstories. Agents collaborate to complete complex tasks through delegation and tool use.
LangGraph
LangChain
Framework for building stateful, multi-agent applications as graphs. Supports cycles, persistence, human-in-the-loop, and streaming. The agent orchestration layer of the LangChain ecosystem.
LlamaIndex
LlamaIndex
Data framework for building LLM and agent applications over your own data. Provides data connectors, indexing, retrieval, and agent workflows for production RAG and knowledge assistants.
OpenAI Agents SDK
OpenAI
Lightweight Python SDK for building agentic AI apps. Features agent handoffs, guardrails, tracing, and tool integration with a minimal abstraction layer over the OpenAI API.
Pydantic AI
Pydantic
Agent framework built on Pydantic for type-safe AI applications. Structured outputs, dependency injection, and model-agnostic design from the team behind Pydantic and FastAPI.
smolagents
Hugging Face
Minimalist agent library focused on code agents that write and execute Python. Simple API with tool calling, multi-step reasoning, and tight Hugging Face Hub integration.