Ollama, llama.cpp, vLLM or SGLang: Choose a Serving Path

Start with the exact model artifact, the hardware backend that can execute it, and the workload you need to serve. Then shortlist a runtime whose documented interfaces and maintenance requirements fit. A tool name alone does not tell you whether a particular checkpoint, quantization and attention kernel will work together.

This is a documentation-informed selection guide. The four maintainer READMEs were checked on 18 October 2026 and pinned below. Run AI Here did not install or benchmark these revisions. The recommendations are editorial starting points; the feature statements are attributed to the maintainers.

Figure: a conceptual selection sequence, generated as SVG by the local Spark model and reviewed for publication. The four runtime names are options, with no speed scale or ranking.

A practical shortlist

These are ways to narrow the first experiment. Capabilities overlap. An OpenAI-compatible endpoint does not make model loading, tool parsing, cache behavior or scheduling identical across runtimes.

What the maintainers actually document

Ollama documents platform installers, application integrations, a model library and a REST API for running and managing models. Its README now describes local and cloud access. If local execution is a requirement, select and verify an explicit local model and endpoint rather than inferring locality from the client application. Record the installed version and actual model identifier. Pinned Ollama README.

llama.cpp documents C/C++ inference, Apple acceleration through Metal and related frameworks, CUDA, several other backends, and CPU/GPU hybrid execution. Its current quickstart uses a GGUF artifact and exposes CLI and OpenAI-compatible server entry points. It is a useful starting point when the artifact and backend controls are central to your decision. Verify architecture support and the exact binary build before planning offload or context capacity. Pinned llama.cpp README.

vLLM documents continuous batching, chunked prefill, prefix caching, several parallelism modes and API serving. Its quantization list includes GGUF as well as multiple other formats; its hardware list also includes plugin paths. These lists are a starting point for compatibility checks, not proof that every listed model, format and accelerator combination works. Evaluate its serving features against your real queue and prompt distribution. Pinned vLLM README.

SGLang describes serving, agentic and rollout workloads and links model/hardware recipes. Its supported-hardware table includes both DGX Spark and Apple Silicon through Metal/MLX. Avoid treating an older platform summary as the current compatibility boundary. Use the matching platform guide and cookbook recipe, then preserve the image/build, attention backend, KV dtype and speculative mode. A platform entry does not validate every checkpoint on that platform. Pinned SGLang README.

Keep document revisions separate from installed versions

The full revisions and SHA-256 checksums are in the source ledger and JSON. A repository revision identifies the reviewed document tree. It does not identify the package installed on your machine, the binary build options, or an executed container image. Record those separately when evaluating a runtime. Avoid treating a moving image tag or an unpinned installation command as an immutable reproduction recipe.

Compare a deployment contract before comparing a rate

Write a small acceptance contract before trying the shortlist. Name the checkpoint and revision, tokenizer and chat template, weight format, KV dtype, hardware and backend, engine/build, context cap and allocated cache. Add required API behavior: streaming, tool calls, structured output, embeddings or multimodal inputs, as applicable. A successful plain chat request does not validate all of these features.

Next, choose request shapes that resemble the application. Include a short interactive prompt and a representative longer prompt, with explicit output lengths and concurrency. Preserve prefix-cache and speculative-decoding state. Report first-token waiting and whole-request served rate separately; a fast decode segment can coexist with a long prefill. Use the metric guide and retain correctness checks alongside timing.

Memory must also be observed at the intended shape. A weight file that fits does not establish KV capacity, concurrency or long-context correctness. The KV and speculation guide shows why configured ceilings, allocated pools and active request demand need separate fields.

If the target is a pair of Sparks

Treat GB10 architecture support, the attention path, container transport and per-node memory as explicit requirements. A recipe for a different checkpoint or GPU generation is a lead to investigate. Before a long load, inspect what the chosen build supports; after launch, verify the transport and actual cache allocation. The GB10 incident guide explains how a dtype assertion, a backend rejection and a shared-memory limit can look like one generic serving failure while requiring different responses.

For AI readers, the adjacent JSON retains four runtime_options, each with documented capability, operator checks, source line and document digest. Its tested_engine_version is null. Keep the source fact and the editorial shortlist together when recommending a path.