Budget the memory around the weights.

Start with the artifact

Use the actual GGUF file size for weights. A label such as Q4 does not mean every parameter occupies exactly four bits; metadata, scales and mixed tensor types affect the file. For a sharded model, include every required shard. The estimator's example inputs describe an imaginary conventional transformer, not a named model or a tested hardware configuration.

Read the architecture

For this deliberately limited budget, KV bytes = 2 × layers × KV heads × head dimension × context tokens × bytes per KV element × concurrent sequences. Divide bytes by 2^30 to get GiB. The factor two represents key and value storage. This is an explicit full-context, equal-K/V-dimension approximation. Architecture metadata and runtime allocation behavior must be checked before using it.

Know when the formula does not apply

Do not use it for MLA, recurrent or state-space blocks, hybrid attention layouts, sliding-window savings, unequal K/V dimensions, compressed caches or runtime-specific paging behavior. It does not model split CPU/GPU placement. Select Unsupported / unknown architecture in the calculator to retain a null result. Mixture-of-experts active parameters do not replace total stored weights in the weight budget.

Add overhead and reserve

Runtime overhead and safety reserve are user-selected allowances in GiB, not measured constants. They cover different things and both remain visible. Available capacity means the memory left for this workload after the OS and other processes, not the device's advertised total. Unified memory is shared by the system; adding a separate GPU-memory number can double count the same physical pool.

Read the calculator honestly

An estimate below the capacity you entered means only that this arithmetic budget is below that capacity. An over-budget result is a reason to reduce context, concurrency or weights before testing. A blank field produces no estimate. Downloading the JSON saves your assumptions and the derived budget; measured peak memory, power and compatibility remain null.

Verify later, under a fixed configuration

On an approved test host, record baseline memory, load the exact artifact, exercise the full planned context and output policy, then capture peak memory with appropriate OS/runtime tooling. Log allocation failures and failed requests in the denominator. If the test fails, retain the failure and revise the budget with a new run identity. Do not edit the previous run to make it appear successful.

Recover from a failed attempt

Stop only the test process and confirm memory is released. Return to the last known working context and placement. Keep the artifact hash and error phase. A smaller model or a different KV type is a new condition that needs its own quality and compatibility check. This local preview performs only arithmetic in your browser.

Read measured memory at its actual prompt length

The M4 Pro Qwen3.5 report gives a 6.4 GiB peak for the 4B int4 row at a 4,096-token prompt, while its 68.0-token/s generation row uses 256 input and 100 output tokens. Keep those as two profiles. A measured peak also belongs to a specific framework and artifact; it cannot be subtracted from nominal RAM and promoted to a guaranteed concurrent-user count.

Do not reuse the conventional formula for every new model

Recent references include sparse, sliding-window, linear-attention, Mamba and compressed-attention designs. They may keep different state or caches per layer. The estimator is deliberately limited to a conventional Transformer KV model and leaves unknown inputs unresolved. For Qwen Flash-Next, GLM Flash, DeepSeek V4.1 or MiMo, inspect the selected runtime’s own cache layout and measured allocation instead of forcing the full-attention formula to fit.

Sources & change log

1. llama.cpp v0.7.0 build documentation ↗Checked 2026-10-17 · Pinned source tag, CPU, Metal and CUDA build paths; not tested locally.
2. Ollama FAQ ↗Checked 2026-10-17 · Local-only configuration, loopback binding, processor reporting and memory caveats.

17 Oct 2026 · Revision 2. Source-based task guide and downloadable resource. Source documentation inspected; environment reproduction not run. Moving documentation may change after review.