This estimates dense-transformer inference memory in GiB, not training memory or tokens per second. Weight size and KV-cache size are calculated separately. The extra allowance is an assumption you control, not a measured runtime overhead.
How to use this estimator
Enter the total parameter count and weight precision for your checkpoint. Find the layer count, number of key/value heads and attention head dimension in the model configuration. For grouped-query attention, KV heads can be fewer than query heads. Do not substitute the query-head count.
Cached tokens include the prompt and generated tokens retained by the runtime. Set concurrent sequences to the number whose caches are resident together. Weight quantization and KV-cache precision are separate: a 4-bit weight file does not imply a 4-bit KV cache.
The defaults describe a hypothetical 8-billion-parameter model with 32 layers, 8 KV heads and head dimension 128. They are an example, not a claim about every 8B model. The calculator works with JavaScript disabled by submitting the form; with JavaScript enabled, the calculation stays in the browser.
The memory formula
Weights (GiB) = parameters × bits per weight ÷ 8 ÷ 2³⁰. The input expresses parameter count in billions, so it is multiplied by 10⁹ first.
KV cache (GiB) = 2 × layers × KV heads × head dimension × cached tokens × concurrent sequences × cache bits ÷ 8 ÷ 2³⁰. The factor of two represents keys and values. This is a simplified full-cache calculation for a conventional transformer layout.
Total estimate = (weights + KV cache) × (1 + allowance / 100). The allowance stands in for unmodeled allocations. It should be adjusted using measurements from your runtime, not treated as a universal safety margin.
Worked example: 8B at 4 bits
With the default values, nominal weight storage is about 3.73 GiB. An 8,192-token cache at 16 bits, 32 layers, 8 KV heads and head dimension 128 adds 1.00 GiB for one sequence. A 25% allowance brings the estimate to about 5.91 GiB.
Doubling the cached-token count doubles the cache term in this simplified model; it does not double the weights. Increasing concurrent sequences also grows the cache term. Moving from 4-bit to 8-bit weights doubles nominal weight storage while leaving the selected cache precision unchanged.
What the estimate does not include exactly
- Quantization metadata and mixed precision: scales, zero points, padding and unquantized tensors change actual storage.
- Runtime allocations: activations, temporary buffers, attention kernels, graph capture, allocator behavior and display usage vary.
- Different architectures: sliding-window or hybrid attention, compressed caches and other layouts may not follow this formula.
- Mixture-of-experts: active parameter count is not total stored parameter count. Routing, placement and offloading need a more specific model.
- Multi-device inference: sharding and replication determine per-device memory. Dividing this total by the number of GPUs is not a reliable general rule.
- Training: gradients, optimizer state and saved activations are outside the scope of this inference estimate.
A result below advertised GPU capacity is not a compatibility guarantee. Verify the checkpoint format, runtime support, available rather than nominal memory, target context length and realistic batch size. Measure peak usage on the actual workload.
Sources and next steps
- Transformers: KV cache strategies — cache choices and memory tradeoffs.
- Transformers: quantization overview — formats and supported methods.
Before deployment, read open weights versus open source and follow open-model news. Report a calculation or documentation error.