{ "id": "A01", "path": "/guides/llm-speed-metrics/", "title": "LLM Speed: TTFT, Decode and Served Rate", "summary": "Understand the clock behind a speed claim with three attributed community examples, exact workloads and separate timing definitions.", "answer": "TTFT measures the first-token wait; decode describes delivery after that point; served rate includes the whole request window. Compare only results whose model, workload, conditions and denominator match.", "published_on": "2026-10-18", "modified_on": "2026-10-18", "source_checked_on": "2026-10-18", "sources": [ { "publisher": "weklund", "url": "https://github.com/weklund/mlx-coding-bench", "repository": "weklund/mlx-coding-bench", "platform": "github", "revision": "5ae9b491f7a0539d5abef949eded6bd57785a981", "document_path": "measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md", "pinned_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md", "report_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md", "document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "last_commit_at": "2026-07-01T19:16:36Z", "checked_on": "2026-10-17", "metadata_status": "checked", "record_id": "qwen35-m4pro-20260306" }, { "publisher": "tonyd2wild", "url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark", "repository": "tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark", "platform": "github", "revision": "d45538f67366c62da668c35ec1afd4fcd0e4c8b4", "document_path": "README.md", "pinned_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md", "report_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#benchmark-boot-10-the-serving-config", "document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7", "last_commit_at": "2026-09-19T13:16:08Z", "checked_on": "2026-10-17", "metadata_status": "checked", "record_id": "deepseek-v41-four-spark-boot10" }, { "publisher": "WeZZard", "url": "https://github.com/WeZZard/dgx-spark-bench", "repository": "WeZZard/dgx-spark-bench", "platform": "github", "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260", "document_path": "README.md", "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md", "report_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md", "document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "last_commit_at": "2026-09-06T00:35:29Z", "checked_on": "2026-10-17", "metadata_status": "checked", "record_id": "spark-pair-moe-served-rate" } ], "facts": [], "reported_results": [ { "label": "Qwen3.5-4B int4 first token", "metric": "ttft", "value": 6.14, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.25 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-4B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306" }, { "label": "Eight-category mean, C1", "metric": "per_stream_decode", "value": 43.12, "unit": "tokens/s", "statistic": "mean across eight categories; counting excluded", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 1, "denominator": "after first token" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7", "source_record_id": "deepseek-v41-four-spark-boot10" }, { "label": "DeepSeek V4 Flash 0731, long agentic", "metric": "served_rate", "value": 22.5, "unit": "tokens/s", "statistic": "median of three campaigns", "uncertainty": { "spread_ratio": 1.04 }, "conditions": { "system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-05", "output_tokens": 512, "ignore_eos": true, "prefix_cache": false, "warmup": "all workload/user shapes, full generation", "repetitions": 3, "concurrency": 1, "measurement_denominator": "whole request wall clock including TTFT", "workload": "agentic 33k context", "model": "DeepSeek-V4-Flash-0731", "kv": "fp8_ds_mla", "kv_pool_tokens": 517484, "draft": "DSpark k5 probabilistic" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "source_record_id": "spark-pair-moe-served-rate" } ], "conditions": { "cross_source_ranking_allowed": false, "one_shared_experiment": false, "independent_site_runs": 0 }, "generation": { "text_draft": "local Spark Qwen", "review": "Run AI Here with Codex", "image": "SVG authored by local Spark Qwen; reviewed and rasterized for the web", "local_first": true }, "image": { "url": "/assets/llm-speed-metrics-spark.webp", "alt": "Conceptual timeline showing the three intervals used for TTFT, decode and served rate.", "width": 1200, "height": 630, "sha256": "5e7448dbf9795d83f9127da60566315581af78183cff975aa1b6271c07cce5e4", "caption": "Conceptual SVG generated by the local Spark text model and reviewed by Run AI Here; no measured scale." }, "schema_version": "1.0.0", "record_type": "source_backed_explainer", "evidence_state": "source_reported", "site_reproduced": false, "metrics": null, "performance_ranking_allowed": false, "body_markdown": "# LLM Speed: TTFT, Decode and Served Rate\n\nTTFT tells you how long you wait for the first token. Decode rate describes output generation after that point. Served rate divides output tokens by the request's total elapsed time. Read all three with the workload and timing definition beside them: a fast stream can still follow a long wait.\n\nThe examples below come from separate community experiments. Each explains a metric in its own configuration. RunAIHere has not independently reproduced these measurements.\n\n![Conceptual timeline showing the three intervals used for TTFT, decode and served rate.](/assets/llm-speed-metrics-spark.webp)\n\n*Conceptual illustration generated as SVG by the local Spark text model and reviewed by RunAIHere. Not to scale; no benchmark values.*\n\n## Read the clock before the headline\n\nStart by identifying when the author starts and stops the timer. Does the reported interval include queueing, prompt processing or a cold start? Was the server already warm? Does an output-rate calculation include the first-token wait? Those choices determine what a reader can infer from a tokens-per-second figure.\n\nFor an interactive session, both the initial wait and subsequent delivery affect the experience. A batch-serving report also needs its concurrency and aggregation rules. Keep an aggregate throughput figure separate from a per-request or per-stream rate.\n\n## TTFT: waiting before the first token\n\nOne dated report used an Apple M4 Pro with 64 GB unified memory, MLX 0.31.0 and mlx-lm 0.30.7. For Qwen3.5-4B int4, the 4,096-token-input test reported mean TTFT of **6.14 seconds**, with standard deviation **0.25 seconds**, across 20 measurements after one warmup iteration. This describes that input shape and setup. The report's generation-rate test used a different input length, so keep those profiles distinct. [Original measurement row](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L40)\n\n## Decode: the generation phase\n\nA four-DGX-Spark report using TP4 and DSpark k5 reported **43.12 tokens/s** as mean per-stream decode at concurrency one, averaged across eight categories with counting excluded. Input and output shapes varied by category; the source reports one batch per cell. It concerns a DeepSeek V4.1 Flash launch checkpoint. The record does not establish performance for DeepSeek-V4.1-Flash-1017. [Original report and row](https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130)\n\n## Served rate: the whole request window\n\nA different operator defines served rate as output tokens divided by whole-request wall-clock time, including first-token waiting. Its two-Spark DeepSeek-V4-Flash-0731 agentic test at approximately 33k context reported **22.5 tokens/s**, the median of three campaigns. That cell used concurrency one, 512 output tokens with ignore-eos enabled, prefix caching off and full-shape warmup. The reported spread ratio was **1.04**; this is a descriptive spread across those campaigns, not a confidence interval. [Original method and result](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70)\n\nThe two DeepSeek examples use different checkpoints and hardware counts. Their values do not provide a controlled comparison between the machines or models.\n\n## A checklist before choosing a deployment\n\n- Match the exact checkpoint, quantization, runtime and hardware topology.\n- Record input length, output length, concurrency and stopping rules.\n- Keep cache, warmup and speculative decoding settings with each result.\n- Retain the statistic and repetitions; distinguish standard deviation, observed spread and confidence intervals.\n- Check missing conditions in the original report before treating a nearby configuration as equivalent.\n\nThe accompanying timeline is conceptual and has no measured scale. Its purpose is to show which interval each metric describes. The linked original reports remain the authority for their measurements.\n" }