THE SOURCE IN CONTEXT
The September NVFP4 report and October EXL3 correction are different deployment lines. The EXL3 author withdrew its old task scores after finding an unloaded router bias. Current evidence is a same-quant probability check; it does not revalidate old task scores or import the NVFP4 task panel.
Which scores remain historical NVFP4 observations?
The September 3 TP3+EP Marlin NVFP4 t10 campaign uses GMU 0.85, FP8 KV, DFlash2 k7 and a decimal-million configured window. Full MMLU is 14042 zero-shot loglikelihood questions with exact accuracy 0.8594217347956131 and harness SE 0.0028276391244761038. Tool-eval has eight trials over 88 hard scenarios, mean 87.8 and source SD 0.9. GSM8K is only 200 questions, 8-shot,188 correct. IFEval uses a custom unverified evaluator.
Source: NNNtrance · pinned document NNNtrance · pinned document NNNtrance · pinned document
Why did the EXL3 author withdraw its old scores?
The October 8 notice identifies missing e_score_correction_bias in the old cuda-exl3 plugin. Wrong expert routing invalidates that repository's old task panels. The new TP3 main stack loads the bias and fails boot if missing. TP2 remains unported and untested. This notice is scoped to the EXL3 plugin line, not proof of the same bug in the separate historical NVFP4 line.
Source: NNNtrance · pinned document NNNtrance · pinned document NNNtrance · pinned document
What was rerun on corrected EXL3?
The detailed KL table compares the same EXL3 checkpoint to exllamav3 1.5.4 on six author-written texts. Assistant KL is 0.1033 pre-fix and 0.00828 on production a4i. Perplexity 2.222 belongs to bias-fix a4g; production perplexity is not reported in that table. The reference is quantized, not BF16. Task benchmarks have not been rerun, and the prepared text set and per-position logits are not public.
Source: NNNtrance · pinned document NNNtrance · pinned document
How are failed attempts kept?
Historical NVFP4 MMLU initially hits a harness queue timeout, and needle tests initially exceed a 120s request cap; same-content reruns use longer timeouts with zero engine errors reported. These are harness retries, not model-accuracy zeros or engine recoveries. A separate model line's completed task scores cannot fill the corrected EXL3 missing scores.
Source: NNNtrance · pinned document NNNtrance · pinned document NNNtrance · pinned document
Reported figures and their conditions
Selected facts reported by the original author. Each row has its own configuration and measurement boundary. Run AI Here has not reproduced these tests.
| Configuration / workload | Metric | Reported value | Evidence |
|---|---|---|---|
| Tool-eval eight-trial mean | tool eval score | 87.8 pointsmean across eight trials; source SD 0.9, source CI [87.1,88.2] | Original row ↗ |
| Tool-eval Pass^8 | all trials pass rate | 76.1 percentsingle finite benchmark; no dispersion reported | Original row ↗ |
| GSM8K 200-question subset | accuracy | 94.0 percentsingle finite benchmark; no dispersion reported | Original row ↗ |
| IFEval prompt strict, custom evaluator | accuracy | 78.9 percentsingle finite benchmark; no dispersion reported | Original row ↗ |
| IFEval instruction level, custom evaluator | accuracy | 85.1 percentsingle finite benchmark; no dispersion reported | Original row ↗ |
| Needle four sizes x five depths | retrieval correct count | 20 correct probessingle finite benchmark; no dispersion reported | Original row ↗ |
| Full MMLU exact accuracy | accuracy | 0.8594217347956131 fractionsingle full MMLU, harness standard error 0.0028276391244761038 | Original row ↗ |
Tool-eval eight-trial mean · tool eval score · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | tool-eval-bench 2.6.1.dev39+gd3352edf5 |
| scenarios | 88 |
| trials | 8 |
| seed | 42 |
| max_turns | 8 |
| concurrency | 1 |
| source_trial_SD | 0.9 |
| source_CI95 | [87.1, 88.2] |
| CI_method_independently_verified | False |
Tool-eval Pass^8 · all trials pass rate · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | tool-eval-bench |
| scenarios | 88 |
| trials | 8 |
| concurrency | 1 |
GSM8K 200-question subset · accuracy · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | GSM8K |
| subset_size | 200 |
| full_test_size | 1319 |
| correct | 188 |
| shots | 8 |
| CoT | True |
| concurrency | Not specified |
IFEval prompt strict, custom evaluator · accuracy · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | IFEval |
| prompts | 541 |
| correct | 427 |
| concurrency | Not specified |
| evaluator | custom; not independently cross-checked |
| proven_lower_bound | False |
IFEval instruction level, custom evaluator · accuracy · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | IFEval |
| instructions | 834 |
| correct | 710 |
| concurrency | Not specified |
| evaluator | custom; not independently cross-checked |
| proven_lower_bound | False |
Needle four sizes x five depths · retrieval correct count · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | 0 |
| thinking | on, effort low except MMLU |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | single-fact NIAH |
| correct | 20 |
| total | 20 |
| max_actual_input_tokens | 997952 |
| concurrency | Not specified |
| initial_timeout_seconds | 120 |
| rerun_timeout_seconds | 3600 |
| identical_content_seed_on_rerun | True |
| multihop_validated | False |
Full MMLU exact accuracy · accuracy · conditions
| model | GLM-5.3-Flash |
|---|---|
| model_artifact | local-inference-lab/GLM-5.3-Flash-NVFP4 |
| model_revision | Not specified |
| engine | LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10 |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| system | three DGX Spark GB10, TP3 + EP |
| node_count | 3 |
| parallelism | TP3+EP |
| weight_precision | NVFP4 storage / Marlin MoE |
| kv_dtype | FP8 |
| draft | DFlash2 k7 |
| context_limit_tokens | 1000000 |
| memory_utilization | 0.85 |
| configured_max_requests | 8 |
| batch_tokens | 2048 |
| experiment_date | 2026-09-03 09:08-18:08 UTC |
| status | historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores |
| host_peak_RAM | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| input_tokens | Not specified |
| output_tokens | Not specified |
| cache_state | Not specified |
| benchmark | MMLU |
| subset_size | 14042 |
| subjects | 57 |
| shots | 0 |
| scoring | loglikelihood, no generation or sampling |
| harness | lm-eval 0.4.9 |
| concurrency | 8 |
| timeout_seconds | 36000 |
| max_retries | 6 |
| accuracy_stderr | 0.0028276391244761038 |
| stderr_is_SD | False |
| initial_harness_timeout_at_fraction | 0.81 |
| engine_errors_reported | 0 |
| rerun_same_content | True |
Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark.