Parameters are the learned numbers inside a model — the weights and biases adjusted during training. They are where the model stores everything it has ever learned. A modern frontier LLM has between 70 billion and (rumoured) several trillion parameters; a small image classifier might have a few million. The parameter count is the single most cited size metric for any model.
For a transformer, parameters live in three main places: the embedding matrices (mapping tokens to vectors), the attention layers (query, key, value projections) and the feedforward layers (the per-token MLPs that do most of the heavy compute). In a typical dense transformer, feedforward layers consume roughly two thirds of the parameter budget.
Parameter count correlates loosely with capability. Bigger models tend to be smarter, but the relationship is messy:
- Training compute matters more than raw size — a small model trained on more tokens often beats a larger model trained on fewer (the Chinchilla scaling laws codified this).
- Mixture of Experts breaks the simple count — a 200B-parameter MoE that activates 20B per token costs roughly the same to run as a dense 20B model but performs closer to a 200B dense model.
- Quality of data matters more than either — a well-curated corpus beats raw scale.
For a US developer choosing a model in 2026, parameter count is a useful first filter but never the last word:
- Fewer than 10B parameters — runs on a high-end laptop or a single GPU. Good for on-device, low-latency, cost-sensitive apps.
- 10B–70B — mid-tier; runs on a workstation with a couple of GPUs. The sweet spot for many production fine-tunes.
- 70B–400B (open weights) / hundreds of billions to trillions (frontier closed) — datacentre-only. The capability frontier.
Closed providers (OpenAI, Anthropic, Google) rarely publish exact parameter counts, leaning on capability benchmarks instead. Open-weight providers (Meta, Mistral, DeepSeek, Alibaba) usually do publish, and the open-weight gap to frontier closed has narrowed dramatically — DeepSeek V3 and Qwen 3 are within striking distance of GPT-5 on many benchmarks while being free to download.