Major Milestones
Key moments that shaped the open source LLM ecosystem.
Transformer Architecture Published
Google Brain publishes 'Attention Is All You Need', introducing the Transformer — the architecture that would power all modern LLMs.
Learn moreGPT-2 Released
OpenAI unveils GPT-2, a 1.5B parameter language model whose surprisingly fluent text generation sparks debate over staged releases and the risks of large language models.
Learn morePyTorch Paper Published
The PyTorch team publishes 'PyTorch: An Imperative Style, High-Performance Deep Learning Library', detailing its define-by-run design. PyTorch becomes the dominant framework for LLM research and training.
Learn moreScaling Laws for Neural Language Models
OpenAI publishes 'Scaling Laws for Neural Language Models', showing that model performance improves as a predictable power law with compute, data, and parameters — the empirical foundation for the era of scaling LLMs.
Learn moreLLaMA Released by Meta
Meta releases LLaMA (7B-65B), proving smaller models trained on more data can compete with much larger ones. Catalyzes the open source LLM movement.
Learn morellama.cpp Launched
Georgi Gerganov releases llama.cpp, enabling LLaMA inference in C/C++ on consumer hardware including Apple Silicon. Democratizes local LLM access.
Learn morevLLM Introduces PagedAttention
UC Berkeley releases vLLM with PagedAttention, applying virtual memory concepts to KV cache. Achieves 2-4x throughput improvement and becomes the standard for LLM serving.
Learn moreLlama 2 Released
Meta releases Llama 2 with a permissive commercial license, including chat-tuned variants. First truly open model family that could be freely deployed in commercial products.
Learn moreMistral 7B Released
Mistral AI drops Mistral 7B via a torrent link with no fanfare. Outperforms Llama 2 13B on all benchmarks, introducing sliding window attention and grouped-query attention.
Learn moreSGLang Launched
The LMSYS team releases SGLang, a fast serving framework introducing RadixAttention for automatic KV cache reuse across requests. Becomes a leading high-performance inference engine alongside vLLM.
Learn moreLlama 3.1 405B Released
Meta releases Llama 3.1 including a 405B parameter model — the largest open-weight model at the time, competitive with GPT-4 class models on many benchmarks.
Learn moreAgentic Frameworks Surge
2024 sees an explosion of agentic frameworks — LangGraph, CrewAI, AutoGen, and others mature rapidly. The focus shifts from simple chatbots to autonomous multi-step agents.
DeepSeek-V3 Released
DeepSeek releases V3, a 671B MoE model trained for only $5.5M — shattering cost assumptions. Introduces Multi-head Latent Attention (MLA) and matches top proprietary models.
Learn moreDeepSeek-R1 — Open Reasoning
DeepSeek releases R1, the first open-weight reasoning model that rivals OpenAI's o1. Achieves chain-of-thought reasoning through pure RL without supervised reasoning traces.
Learn moreLlama 4 Released
Meta releases Llama 4 (Scout and Maverick), its first natively multimodal, mixture-of-experts model family with very long context windows. Pushes open models further into MoE architectures.
Learn moreDeepSeek Sparse Attention
DeepSeek introduces Sparse Attention (DSA) with DeepSeek-V3.2-Exp, selectively attending to the most relevant tokens to cut the cost of long-context inference while preserving quality.
Learn moreDeepSeek-V4 Released
DeepSeek releases V4-Pro and V4-Flash, trained on over 32T tokens and natively supporting a 1M-token context window. Built on Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to extend context and efficiency.
Learn moreGLM-5.2 Released
Zhipu AI releases GLM-5.2, a large MoE model under the permissive MIT license with a 1M context window, designed for long-horizon agentic tasks. It beats GPT-5.5 on several benchmarks and trail Claude Opus 4.8 slightly, at far lower cost.
Learn moreKimi K3 Released
Moonshot AI releases Kimi K3, the first open model to reach 2.8 trillion parameters. The native multimodal MoE model combines a 1M-token context window with Kimi Delta Attention and Attention Residuals.
Learn moreTech Industry Open Weight Letter
After Kimi K3 hits public download and stirs concern that policymakers may crack down on open AI systems, Jensen Huang joins Satya Nadella and dozens of tech companies on a letter warning against restricting open models.
Learn moreDeepSeek-V4-Flash-0731 Redraws the Cost Curve
DeepSeek retrains V4-Flash for agents and coding and ships it at the same price: $0.14 per million input tokens and $0.28 per million output tokens. Terminal Bench 2.1 jumps to 82.7 with 13B active parameters, putting an MIT-licensed model on the Pareto frontier at roughly 1% of the output price of the closed frontier.
Learn more