# Ollama, llama.cpp, vLLM or SGLang: Choose a Serving Path

Start with the exact model artifact, the hardware backend that can execute it, and the workload you need to serve. Then shortlist a runtime whose documented interfaces and maintenance requirements fit. A tool name alone does not tell you whether a particular checkpoint, quantization and attention kernel will work together.

This is a documentation-informed selection guide. The four maintainer READMEs were checked on 18 October 2026 and pinned below. Run AI Here did not install or benchmark these revisions. The recommendations are editorial starting points; the feature statements are attributed to the maintainers.

![Conceptual selection sequence from artifact to backend to workload, with four runtime options.](/assets/a07-batch3-local-spark.webp)

Figure: a conceptual selection sequence, generated as SVG by the local Spark model and reviewed for publication. The four runtime names are options, with no speed scale or ranking.

## A practical shortlist

| Your first requirement | Start the evaluation with | What still needs proof |
|---|---|---|
| Packaged model workflow and application integration | Ollama | The selected model runs where you intend, and the application uses the right endpoint and prompt format |
| A GGUF artifact and direct control of a CLI or server | llama.cpp | That architecture and quantization work with your chosen build and backend |
| Concurrent API serving, batching and parallel execution | vLLM | The exact model, kernel, hardware and request shape are supported together |
| A serving or agent workload with a matching deployment recipe | SGLang | The recipe matches your artifact, platform, image and KV/attention settings |

These are ways to narrow the first experiment. Capabilities overlap. An OpenAI-compatible endpoint does not make model loading, tool parsing, cache behavior or scheduling identical across runtimes.

## What the maintainers actually document

**Ollama** documents platform installers, application integrations, a model library and a REST API for running and managing models. Its README now describes local and cloud access. If local execution is a requirement, select and verify an explicit local model and endpoint rather than inferring locality from the client application. Record the installed version and actual model identifier. [Pinned Ollama README](https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md#L90).

**llama.cpp** documents C/C++ inference, Apple acceleration through Metal and related frameworks, CUDA, several other backends, and CPU/GPU hybrid execution. Its current quickstart uses a GGUF artifact and exposes CLI and OpenAI-compatible server entry points. It is a useful starting point when the artifact and backend controls are central to your decision. Verify architecture support and the exact binary build before planning offload or context capacity. [Pinned llama.cpp README](https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md#L60).

**vLLM** documents continuous batching, chunked prefill, prefix caching, several parallelism modes and API serving. Its quantization list includes GGUF as well as multiple other formats; its hardware list also includes plugin paths. These lists are a starting point for compatibility checks, not proof that every listed model, format and accelerator combination works. Evaluate its serving features against your real queue and prompt distribution. [Pinned vLLM README](https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md#L28).

**SGLang** describes serving, agentic and rollout workloads and links model/hardware recipes. Its supported-hardware table includes both DGX Spark and Apple Silicon through Metal/MLX. Avoid treating an older platform summary as the current compatibility boundary. Use the matching platform guide and cookbook recipe, then preserve the image/build, attention backend, KV dtype and speculative mode. A platform entry does not validate every checkpoint on that platform. [Pinned SGLang README](https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md#L21).

## Keep document revisions separate from installed versions

| Maintainer document | Reviewed repository revision | Installed version tested here |
|---|---|---|
| Ollama README | `36a6e994d2f6` | Unknown; no installation test |
| llama.cpp README | `547208dca250` | Unknown; no installation test |
| vLLM README | `da9134bc8936` | Unknown; no installation test |
| SGLang README | `e5dc10e73b0c` | Unknown; no installation test |

The full revisions and SHA-256 checksums are in the source ledger and JSON. A repository revision identifies the reviewed document tree. It does not identify the package installed on your machine, the binary build options, or an executed container image. Record those separately when evaluating a runtime. Avoid treating a moving image tag or an unpinned installation command as an immutable reproduction recipe.

## Compare a deployment contract before comparing a rate

Write a small acceptance contract before trying the shortlist. Name the checkpoint and revision, tokenizer and chat template, weight format, KV dtype, hardware and backend, engine/build, context cap and allocated cache. Add required API behavior: streaming, tool calls, structured output, embeddings or multimodal inputs, as applicable. A successful plain chat request does not validate all of these features.

Next, choose request shapes that resemble the application. Include a short interactive prompt and a representative longer prompt, with explicit output lengths and concurrency. Preserve prefix-cache and speculative-decoding state. Report first-token waiting and whole-request served rate separately; a fast decode segment can coexist with a long prefill. Use the [metric guide](/guides/llm-speed-metrics/) and retain correctness checks alongside timing.

Memory must also be observed at the intended shape. A weight file that fits does not establish KV capacity, concurrency or long-context correctness. The [KV and speculation guide](/guides/kv-cache-speculative-decoding/) shows why configured ceilings, allocated pools and active request demand need separate fields.

## If the target is a pair of Sparks

Treat GB10 architecture support, the attention path, container transport and per-node memory as explicit requirements. A recipe for a different checkpoint or GPU generation is a lead to investigate. Before a long load, inspect what the chosen build supports; after launch, verify the transport and actual cache allocation. The [GB10 incident guide](/guides/gb10-long-context-failures/) explains how a dtype assertion, a backend rejection and a shared-memory limit can look like one generic serving failure while requiring different responses.

For AI readers, the adjacent JSON retains four `runtime_options`, each with documented capability, operator checks, source line and document digest. Its `tested_engine_version` is `null`. Keep the source fact and the editorial shortlist together when recommending a path.


## Original sources and conditions

- ollama: https://github.com/ollama/ollama/tree/d1ed3fd13ac73f60cff00bbead8af24a68c4200d
  - Document revision: d1ed3fd13ac73f60cff00bbead8af24a68c4200d
  - SHA-256: 9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b
- ggml-org: https://github.com/ggml-org/llama.cpp/tree/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924
  - Document revision: f74e2fa6a31cc33ab4ee4ae516641cfe62df3924
  - SHA-256: dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6
- vllm-project: https://github.com/vllm-project/vllm/tree/ce49c47cbb241b59a32ef110079a5261dbd08e49
  - Document revision: ce49c47cbb241b59a32ef110079a5261dbd08e49
  - SHA-256: 4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c
- sgl-project: https://github.com/sgl-project/sglang/tree/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e
  - Document revision: 2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e
  - SHA-256: 7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e
- ollama: https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md
  - Document revision: 36a6e994d2f6c4ab94b3555185154d069e5dba87
  - SHA-256: 9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b
- ggml-org: https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md
  - Document revision: 547208dca250ceb3e4c8615598d043f2b602a527
  - SHA-256: dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6
- vllm-project: https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md
  - Document revision: da9134bc89364465f68aa679564435936d157a14
  - SHA-256: 4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c
- sgl-project: https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md
  - Document revision: e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0
  - SHA-256: 7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e

Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export.

```json
{
  "facts": [],
  "reported_results": [],
  "conditions": {
    "comparison_type": "documentation-informed editorial shortlist",
    "installed_versions_tested": null,
    "model_benchmarks_run": 0,
    "speed_ranking_allowed": false
  },
  "runtime_options": [
    {
      "id": "ollama",
      "name": "Ollama",
      "evidence_status": "maintainer_documented_capability",
      "document_revision": "36a6e994d2f6c4ab94b3555185154d069e5dba87",
      "tested_engine_version": null,
      "shortlist_when": "A packaged model workflow, app integrations, and a model-management REST API are the main requirements.",
      "operator_checks": [
        "Select an explicit local model and endpoint if local execution is required; the README also offers cloud models.",
        "Record the actual Ollama version, model identifier and artifact digest.",
        "Validate the app API, prompt template and memory behavior with the intended request."
      ],
      "source_document_id": "ollama-readme-20261018",
      "source_document_sha256": "9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b",
      "source_line": 92,
      "evidence_url": "https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md#L92"
    },
    {
      "id": "llama-cpp",
      "name": "llama.cpp",
      "evidence_status": "maintainer_documented_capability",
      "document_revision": "547208dca250ceb3e4c8615598d043f2b602a527",
      "tested_engine_version": null,
      "shortlist_when": "You want a GGUF-oriented starting point, direct CLI/server control, or documented CPU, Metal and CUDA backend choices.",
      "operator_checks": [
        "Check architecture and quantization support for the exact GGUF artifact.",
        "Record the binary/build revision and selected backend.",
        "Measure GPU offload, KV allocation and request concurrency rather than inferring them from file size."
      ],
      "source_document_id": "llama-cpp-readme-20261018",
      "source_document_sha256": "dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6",
      "source_line": 72,
      "evidence_url": "https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md#L72"
    },
    {
      "id": "vllm",
      "name": "vLLM",
      "evidence_status": "maintainer_documented_capability",
      "document_revision": "da9134bc89364465f68aa679564435936d157a14",
      "tested_engine_version": null,
      "shortlist_when": "A serving workload needs documented batching, prefix caching, parallel execution, and compatible API surfaces.",
      "operator_checks": [
        "Verify the exact architecture, quantization, kernel and hardware combination; listed support is not a full compatibility matrix.",
        "Record the installed package/commit, wheel or image digest, and launch flags.",
        "Evaluate TTFT and served rate under representative concurrent requests."
      ],
      "source_document_id": "vllm-readme-20261018",
      "source_document_sha256": "4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c",
      "source_line": 32,
      "evidence_url": "https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md#L32"
    },
    {
      "id": "sglang",
      "name": "SGLang",
      "evidence_status": "maintainer_documented_capability",
      "document_revision": "e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0",
      "tested_engine_version": null,
      "shortlist_when": "A supported serving or agent workload has a matching model-and-hardware recipe you can maintain.",
      "operator_checks": [
        "Start from the matching cookbook/platform recipe and pin it.",
        "Its README includes Apple Metal/MLX and DGX Spark; confirm per-model/backend limitations.",
        "Record image/build, KV dtype, attention backend and speculative mode separately."
      ],
      "source_document_id": "sglang-readme-20261018",
      "source_document_sha256": "7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e",
      "source_line": 21,
      "evidence_url": "https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md#L21"
    }
  ]
}
```