Kimi IQ1_S on Three Sparks: Scores, Retries and Missing Pins

Small source subsets and unequal cloud settings do not establish lossless quantization.

Small local quality reports can be informative when they preserve the questions tested, failure history and sampling controls. They become misleading when a composite score is presented as a first-pass result or matching percentages become a claim of lossless quantization. The chishiki37 report provides deployment details and raw subset results for a publisher-labelled Kimi K3 IQ1_S on three Sparks. This note keeps its useful observations and the limits visible together.

Start with the identity that is actually known

The August 7 report calls the model Kimi K3/Kimi-Linear IQ1_S, with nine GGUF shards and an approximately 330 GB file total. It names a shard pattern but does not pin the exact upstream repository revision, weight digest or complete runtime commit. Run AI Here therefore retains the publisher's model label without certifying that it uniquely identifies a current official checkpoint. File size and a plausible filename cannot establish that identity.

Three DGX Sparks supply 128 GB unified memory each. A pwilkin llama.cpp fork uses layer-split RPC with two remote workers. This is not tensor parallelism across three GPUs. The fabric overview says MTU9000, while the actual deployment configuration says MTU4096, inline316 and receive-depth24. Those are distinct source statements, and the JSON preserves both instead of quietly choosing the larger value.

The configuration uses f16 KV, load mode none and one request slot. The source reports that Q8 KV is incompatible with its V-head dimension and block constraint in this build. That is a specific runtime/artifact condition, not a claim that every Kimi model or future llama.cpp version lacks quantized KV support. The original launch line is a reproduction reference; this site has not executed it.

A configured million-token window is not a measured full input

The configured window is 1,048,576 tokens. The performance summary reports approximately 2.9 tokens/s steady decode and the following prefill values. Exact per-run prompt counts, output lengths, timing traces and repeated-run dispersion are not supplied in that table.

A configured 1,048,576-token window is separate from these performance observations.

The longest listed performance label is 131K. A source statement that a server loads with a 1M window is not enough to establish a completed 1,048,576-token input, correct recall throughout that input or acceptable concurrency at that length. The selected sources do not supply the corresponding full-input payload and paired telemetry. Those fields stay unknown rather than borrowing a test from a different Kimi configuration.

Read the GSM8K score as a fixed subset

The local GSM8K raw summary reports 49 correct of 50, or 98 percent, on the selected canonical test questions. The source's cloud Kimi and cloud GLM-5.2 comparisons also report 49/50 and the same failed item. These are source subsets, not full standardized leaderboard evaluations and not independent Run AI Here measurements.

The controls are not matched across services. Local Kimi uses temperature0.2 and an 8,192-token output cap, while cloud Kimi forces temperature1 and uses a 2,048-token GSM8K cap. The GLM comparison raises its cap after two token-exhaustion failures. Exact API model versions and a matched unquantized local baseline are missing. Equal subset percentages under those conditions do not establish equivalence or isolate the quality cost of IQ1_S quantization.

HumanEval: the first pass was 14/20

The initial local HumanEval file records 14 passes out of 20, or 70 percent. Four requests time out at the 600-second client limit: IDs1,6,10 and17. Two other tasks, IDs2 and7, fail assertions. Timeouts are retained as first-pass failures rather than silently removed from the denominator.

The timeout-only rerun evaluates those four timed-out tasks with higher limits and passes all four. Its 4/4 result is a selected retry subset. Combined with the fourteen initial passes, it explains the README's final 18/20, or 90 percent. That composite is not a fresh first-pass evaluation of twenty problems and should never replace the initial 14/20 in a reliability discussion.

The four successful retries are a selected subset; final 18/20 must retain this history.

One retry takes 995.8 seconds and another 1,180.1. This matters operationally: raising a timeout can recover a correct answer while increasing how long a client or worker remains occupied. The source's final aggregate timing summary does not give a clean first-pass service-latency distribution. It is unsuitable for an application percentile claim, and the selected retries do not estimate the rate at which arbitrary future timeout failures recover.

The remaining failures concern truncate_number and filter_by_substring. The README groups both under floating-point edge cases, but substring filtering is not itself a floating-point task. This article retains the assertion failures without adopting that blanket diagnosis. Raw failure IDs and errors are evidence; a common root cause would require further analysis. No upstream code or generated candidate was executed during this editorial review.

Memory occupancy is approximate and lacks a peak method

The source gives about 119–120 GB per node and about 357 GB total, with small reported headroom. It does not define a peak sampling method, sampling cadence, process-versus-host boundary or paired context length. The GB label also lacks a verified byte convention. Those observations are approximate occupancy reports, not a measured peak-RAM curve.

Do not infer a 1M full-input pass, safe peak headroom or quantization equivalence.

The reported vmstat swap-in/out counters are zero during observation. That does not prove no swap was allocated, that no future workload can swap or that all three machines have safe transient headroom at the configured maximum. A warm-cache load time and a memory occupancy snapshot answer different questions. They should not be added to weight-file bytes or transformed into a universal hardware sizing guarantee.

What these scores can help decide

This report offers a concrete three-node deployment reference, a slow but measured generation path and a transparent small-suite failure history. It can help frame a reproduction: pin the actual artifact and runtime, retain per-task sampling and output caps, record timeout policy, and measure memory alongside the intended request lengths. It does not support a global ranking against cloud models or another quantization format.

The useful score is therefore the one that keeps its denominator and history. Read 49/50 with the subset and unequal controls; read 18/20 with the initial 14/20 and selected four-task rerun. Keep configured context separate from completed input and approximate occupancy separate from peak memory. Those distinctions allow both readers and AI systems to use the original evidence without turning a limited experiment into a broader capability claim.