# Kimi IQ1_S on Three Sparks: Scores, Retries and Missing Pins ![Publisher-labelled K3 / three Spark layer-split RPC / exact upstream checkpoint unpinned.](/assets/batch7-kimi-hero.webp) *Small source subsets and unequal cloud settings do not establish lossless quantization.* Small local quality reports can be informative when they preserve the questions tested, failure history and sampling controls. They become misleading when a composite score is presented as a first-pass result or matching percentages become a claim of lossless quantization. The [chishiki37 report](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md) provides deployment details and raw subset results for a publisher-labelled Kimi K3 IQ1_S on three Sparks. This note keeps its useful observations and the limits visible together. ## Start with the identity that is actually known The August 7 report calls the model Kimi K3/Kimi-Linear IQ1_S, with nine GGUF shards and an approximately 330 GB file total. It names a shard pattern but does not pin the exact upstream repository revision, weight digest or complete runtime commit. Run AI Here therefore retains the publisher's model label without certifying that it uniquely identifies a current official checkpoint. File size and a plausible filename cannot establish that identity. Three DGX Sparks supply 128 GB unified memory each. A pwilkin llama.cpp fork uses layer-split RPC with two remote workers. This is not tensor parallelism across three GPUs. The fabric overview says MTU9000, while the actual deployment configuration says MTU4096, inline316 and receive-depth24. Those are distinct source statements, and the JSON preserves both instead of quietly choosing the larger value. The configuration uses f16 KV, load mode `none` and one request slot. The source reports that Q8 KV is incompatible with its V-head dimension and block constraint in this build. That is a specific runtime/artifact condition, not a claim that every Kimi model or future llama.cpp version lacks quantized KV support. The original launch line is a reproduction reference; this site has not executed it. ## A configured million-token window is not a measured full input The configured window is 1,048,576 tokens. The performance summary reports approximately 2.9 tokens/s steady decode and the following prefill values. Exact per-run prompt counts, output lengths, timing traces and repeated-run dispersion are not supplied in that table. | Original input label | Reported prefill, tokens/s | |---|---:| | 8K | 79.6 | | 32K | 43.9 | | 131K | 48.5 | ![Kimi IQ1_S source prefill rates at 8K, 32K and 131K nominal prompt lengths.](/assets/batch7-kimi-prefill.webp) *A configured 1,048,576-token window is separate from these performance observations.* The longest listed performance label is 131K. A source statement that a server loads with a 1M window is not enough to establish a completed 1,048,576-token input, correct recall throughout that input or acceptable concurrency at that length. The selected sources do not supply the corresponding full-input payload and paired telemetry. Those fields stay unknown rather than borrowing a test from a different Kimi configuration. ## Read the GSM8K score as a fixed subset The [local GSM8K raw summary](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json) reports 49 correct of 50, or 98 percent, on the selected canonical test questions. The source's cloud Kimi and cloud GLM-5.2 comparisons also report 49/50 and the same failed item. These are source subsets, not full standardized leaderboard evaluations and not independent Run AI Here measurements. The controls are not matched across services. Local Kimi uses temperature0.2 and an 8,192-token output cap, while cloud Kimi forces temperature1 and uses a 2,048-token GSM8K cap. The GLM comparison raises its cap after two token-exhaustion failures. Exact API model versions and a matched unquantized local baseline are missing. Equal subset percentages under those conditions do not establish equivalence or isolate the quality cost of IQ1_S quantization. ## HumanEval: the first pass was 14/20 The [initial local HumanEval file](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json) records 14 passes out of 20, or 70 percent. Four requests time out at the 600-second client limit: IDs1,6,10 and17. Two other tasks, IDs2 and7, fail assertions. Timeouts are retained as first-pass failures rather than silently removed from the denominator. The [timeout-only rerun](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json) evaluates those four timed-out tasks with higher limits and passes all four. Its 4/4 result is a selected retry subset. Combined with the fourteen initial passes, it explains the README's final 18/20, or 90 percent. That composite is not a fresh first-pass evaluation of twenty problems and should never replace the initial 14/20 in a reliability discussion. ![Kimi local HumanEval subset rises from 14 of 20 initially to 18 of 20 after four timeout retries.](/assets/batch7-kimi-retry-scores.webp) *The four successful retries are a selected subset; final 18/20 must retain this history.* One retry takes 995.8 seconds and another 1,180.1. This matters operationally: raising a timeout can recover a correct answer while increasing how long a client or worker remains occupied. The source's final aggregate timing summary does not give a clean first-pass service-latency distribution. It is unsuitable for an application percentile claim, and the selected retries do not estimate the rate at which arbitrary future timeout failures recover. The remaining failures concern `truncate_number` and `filter_by_substring`. The README groups both under floating-point edge cases, but substring filtering is not itself a floating-point task. This article retains the assertion failures without adopting that blanket diagnosis. Raw failure IDs and errors are evidence; a common root cause would require further analysis. No upstream code or generated candidate was executed during this editorial review. ## Memory occupancy is approximate and lacks a peak method The source gives about 119–120 GB per node and about 357 GB total, with small reported headroom. It does not define a peak sampling method, sampling cadence, process-versus-host boundary or paired context length. The GB label also lacks a verified byte convention. Those observations are approximate occupancy reports, not a measured peak-RAM curve. ![The report labels approximately 119–120 GB per node without a telemetry method.](/assets/batch7-kimi-memory.webp) *Do not infer a 1M full-input pass, safe peak headroom or quantization equivalence.* The reported `vmstat` swap-in/out counters are zero during observation. That does not prove no swap was allocated, that no future workload can swap or that all three machines have safe transient headroom at the configured maximum. A warm-cache load time and a memory occupancy snapshot answer different questions. They should not be added to weight-file bytes or transformed into a universal hardware sizing guarantee. ## What these scores can help decide This report offers a concrete three-node deployment reference, a slow but measured generation path and a transparent small-suite failure history. It can help frame a reproduction: pin the actual artifact and runtime, retain per-task sampling and output caps, record timeout policy, and measure memory alongside the intended request lengths. It does not support a global ranking against cloud models or another quantization format. The useful score is therefore the one that keeps its denominator and history. Read 49/50 with the subset and unequal controls; read 18/20 with the initial 14/20 and selected four-task rerun. Keep configured context separate from completed input and approximate occupancy separate from peak memory. Those distinctions allow both readers and AI systems to use the original evidence without turning a limited experiment into a broader capability claim. ## Original sources and conditions - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590 - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053 - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/cloud-humaneval-logic-k3-cloud.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 8d42fd9033e794d5a8a7330d5c56707128b521f869c46be071a364c5f45fa22c - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-cloud.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 636e0ee2c31986d961ffe637927bcb95aa1d3d82d78dea08f7efe0807f29971b Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "steady decode", "metric": "decode_rate", "value": 2.9, "unit": "tokens/s", "statistic": "approximate source summary", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "steady decode", "input_length_source_label": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 43, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L43", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "8K prefill", "metric": "prefill_rate", "value": 79.6, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "8K prefill", "input_length_source_label": "8K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 44, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L44", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "32K prefill", "metric": "prefill_rate", "value": 43.9, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "32K prefill", "input_length_source_label": "32K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 45, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L45", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "131K prefill", "metric": "prefill_rate", "value": 48.5, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "131K prefill", "input_length_source_label": "131K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 46, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L46", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "GSM8K local subset", "metric": "accuracy", "value": 98.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "GSM8K canonical test subset", "correct": 49, "total": 50, "subset_ids": "50 source-indexed questions; fixed subset, not full benchmark", "timeout_seconds": 1800, "retry_policy": "GLM comparison changed output cap; local retries unreported" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra0", "source_document_sha256": "2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be", "source_line": 5, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json#L5", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval initial local subset", "metric": "accuracy", "value": 70.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 14, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": 600, "timeouts": 4, "retry_policy": "initial run; four timeouts retained as failures" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra1", "source_document_sha256": "29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json#L6", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval timeout retries", "metric": "accuracy", "value": 100.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval timeout-only subset", "correct": 4, "total": 4, "subset_ids": [ 1, 6, 10, 17 ], "timeout_seconds": null, "retry_policy": "rerun only four timed-out problems with higher limits; not fresh 20-problem pass" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra2", "source_document_sha256": "469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json#L6", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval local after selected retries", "metric": "accuracy", "value": 90.0, "unit": "percent", "statistic": "source composite score after retries", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 18, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": null, "retry_policy": "14 initial passes plus 4 successful timeout-only retries; not first-pass 18/20", "failed_ids": [ 2, 7 ] }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 83, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L83", "source_record_id": "kimi-k3-iq1s-three-spark-scores" } ], "conditions": { "independent_site_runs": 0, "full_1048576_input_test_proven": false, "unquantized_local_control": false, "sampling_matched_to_cloud": false, "quantization_equivalence_proven": false, "upstream_checkpoint_identity_verified": false }, "quality_comparisons": [ { "benchmark": "GSM8K", "subset_size": 50, "local_correct": 49, "cloud_Kimi_correct": 49, "cloud_GLM52_correct": 49, "local_temperature": 0.2, "cloud_Kimi_temperature": 1.0, "local_output_cap": 8192, "cloud_Kimi_output_cap": 2048, "GLM_cap_changed_after_exhaustion": true, "matched_accuracy_proves_equivalence": false }, { "benchmark": "HumanEval", "subset_size": 20, "initial_local_correct": 14, "initial_timeouts": 4, "retry_subset_ids": [ 1, 6, 10, 17 ], "retry_correct": 4, "final_local_correct": 18, "remaining_failure_ids": [ 2, 7 ], "first_pass_18_of_20": false } ], "memory_observations": [ { "kind": "approximate_per_node_occupancy_unknown_measurement_method", "source_range_GB": [ 119, 120 ], "unit": "GB (source label; byte convention unspecified)", "node_count": 3, "is_measured_peak": false, "peak_RAM": null } ], "long_context_cases": [ { "classification": "configured_context_not_completed_full_window", "configured_limit_tokens": 1048576, "longest_performance_source_label": "131K", "actual_full_window_prompt_tokens": null } ] } ```