Four of six advertised quant levels, including the default q4_k_m, only produce an f16 intermediate and still exit 0 (CR-A13-003) #408

Wave 1 · theme D1-5: Training, merging and export correctness · highest severity High · 4 finding(s) · effort small

Summary

A release script runs the README command, gets exit 0, and then uploads or ollama creates model-q4.gguf, which does not exist. Or the script ships the f16 file believing it is q4_k_m.

Primary finding

CR-A13-003 — Four of six advertised quant levels, including the default q4_k_m, only produce an f16 intermediate and still exit 0

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs

What happens. _OUTTYPE_MAP maps q2_k, q3_k_m, q4_k_m and q5_k_m to f16 (forgelm/export.py:41-52). _resolve_kquant_path redirects the output to a .f16.gguf sibling (export.py:266-292), and export returns success with manual_step_required and a followup_command. The CLI default is q4_k_m, so the documented command never creates the file named in --output. README, the site and the manual promise six quant levels.

Where.

• forgelm/export.py:36-52 (_OUTTYPE_MAP and _K_QUANTS at 41-52)
• forgelm/export.py:266-292 (_resolve_kquant_path)
• forgelm/export.py:464-483 (manual_step_required/followup_command/ExportResult)
• forgelm/cli/_parser.py:88-94 (confirmed)
• docs/usermanuals/en/deployment/gguf-export.md:8,12-22,46-54
• docs/usermanuals/en/deployment/gguf-export.md:44-69
• forgelm/export.py:41
• forgelm/export.py:266
• forgelm/export.py:468
• tests/test_export.py:182-205,334-362 (assert the substitution; test-gap)

Evidence.

• scr/export_t.py: export_model(..., 'out/model.gguf', quant='q4_k_m') -> success=True, output_path=out/model.f16.gguf, quant='f16', requested_quant='q4_k_m', manual_step_required=True. out/model.gguf does not exist.
• export.py:41-52, 266-292, 468-471; tests/test_export.py:182-205, 334-362.
• forgelm/export.py:41-47 (_OUTTYPE_MAP), :266-295 (_resolve_kquant_path), :468-478. Stub-converter run of all 6 quants. pyproject.toml:126-132 admits the manual step. README.md:65,159 and site/index.html:489 claim '6 quant levels'. The usermanual docs/usermanuals/en/deployment/gguf-export.md:8-31,46-51 shows false output. pyproject.toml:128 points at docs/guides/export.md, which does not exist.

Recommended fix. Make the default quant one the tool can produce (q8_0 or f16), or run llama-quantize when it is available. Otherwise exit non-zero, or require an explicit opt-in, when the requested quant cannot be produced. Fix the manual.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T08-002 — Export and serving targets: Quant levels offered by forgelm export (q2_k, q3_k_m, q4_k_m, q5_k_m, q8_0, f16), the f16 intermediate, and the 'converter only supports f16/q8_0' claim (aging)

severity Next release

Item. Quant levels offered by forgelm export (q2_k, q3_k_m, q4_k_m, q5_k_m, q8_0, f16), the f16 intermediate, and the 'converter only supports f16/q8_0' claim · verdict: aging · urgency: Next release

State in ForgeLM. forgelm/export.py:31 and forgelm/cli/_parser.py:91-93 offer six quants, default q4_k_m. _OUTTYPE_MAP (export.py:41-49) passes f16 and q8_0 straight through and maps every K-quant to --outtype f16. _resolve_kquant_path (export.py:266-292) writes a .f16.gguf intermediate and asks the user to run llama-quantize by hand. The adapter merge is forced to float16 (export.py:158-175). The comments at export.py:44 and :72-73 say the converter only supports or emits f16/q8_0 directly.

State of the ecosystem on 2026-10-01. convert_hf_to_gguf.py:60, both on master and vendored in llama-cpp-python 0.3.36, offers --outtype choices f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto, with default auto. K-quants and IQ-quants still need llama-quantize, whose CMake target is still llama-quantize (tools/quantize/CMakeLists.txt:17). The quantize README's reference flow is a bf16 GGUF ('--outtype auto ... also works well') followed by llama-quantize ... Q4_K_M. quantize.cpp lists, for Llama-3-8B, Q4_K_M at 4.58G +0.1754 ppl, Q6_K at +0.0217, Q8_0 at +0.0026 and Q2_K at +3.5199. For Llama-3.1-8B, IQ4_XS is 4.17 GiB / 4.46 bpw against Q4_K_M at 4.58 GiB / 4.89 bpw. llama-quant.cpp:856-874 requires an imatrix for IQ1_, IQ2_ and IQ3_XXS, and for Q2_K tensors in a Q2_K_S file. Ollama's import doc says Ollama does not quantize GGUF on import.

Impact. The quant names are all valid and Q4_K_M is still a sound default. The mechanism is stale. The f16/q8_0-only comment is wrong. The forced f16 merge and f16 intermediate lose the bf16 fidelity upstream now recommends. Four of the six levels are never produced automatically; this is flagged honestly in requested_quant, manual_step_required and followup_command. The menu lacks Q6_K, IQ4_XS and imatrix support, and the user manual shows quantised output the code never produces.

Recommendation. Use --outtype auto (or bf16) for K-quant intermediates, and keep f16 only when f16 is explicitly requested. Run llama-quantize automatically when it is found, keeping the intermediate as a fallback. Add q6_k and an opt-in --imatrix for iq4_xs and low-bit quants, and add a quality warning to q2_k. Fix the comments at export.py:44 and :72-73 and the manual.

Sources.

• llama.cpp convert_hf_to_gguf.py master (--outtype choices, :60) (2026-10-01)
• llama.cpp tools/quantize README (bf16/auto intermediate, llama-quantize, IQ4_XS vs Q4_K_M) (2026-10-01)
• Vendored llama.cpp: tools/quantize/quantize.cpp, tools/quantize/CMakeLists.txt, src/llama-quant.cpp (2026-10-01)
• Ollama import docs (no GGUF quantization on import; use llama-quantize) (2026-10-01)
• Hugging Face Hub docs: GGUF quantization types (2026-10-01T16:37:50Z)

DOC-D3B08-013 — docs/usermanuals/en/deployment/gguf-export.md: false claim — A manifest JSON with base_model, fine_tune_run_id, quants{q:{size_bytes, sha256}} and exported_at.

severity High

Where. docs/usermanuals/en/deployment/gguf-export.md:56-69 (Turkish mirror: docs/usermanuals/tr/deployment/gguf-export.md:61) · type: false claim

The document says. A manifest JSON with base_model, fine_tune_run_id, quants{q:{size_bytes, sha256}} and exported_at.

What the code does. No such file exists. The only record is an entry {file, format, quant, sha256, size_bytes} appended to exported_artifacts in model_integrity.json, and only if that file already exists in the export's model_path directory (export.py:207-239).

Code references. forgelm/export.py:207-239, forgelm/export.py:493-494

User impact. Users and automation look for a manifest that does not exist.

Proposed correction. Replace with the real model_integrity.json exported_artifacts entry and note it is updated only when model_integrity.json exists in MODEL_PATH (skip with --no-integrity-update).

Turkish mirror. the Turkish page has the same error; fix both in the same PR

DOC-D3B08-040 — docs/usermanuals/en/deployment/gguf-export.md: false claim — Export writes a manifest with base_model, fine_tune_run_id, quants{...}, exported_at

severity Medium

Where. docs/usermanuals/en/deployment/gguf-export.md:56-69 · type: false claim

The document says. Export writes a manifest with base_model, fine_tune_run_id, quants{...}, exported_at

What the code does. Only exported_artifacts[] (file, format, quant, sha256, size_bytes) appended to model_integrity.json when it exists.

Code references. forgelm/export.py:219-226

User impact. Consumers parsing the documented manifest find nothing.

Proposed correction. Document model_integrity.json exported_artifacts.

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}