feat(server): enable reasoning_effort on /v1/chat/completions - #2
Merged
Conversation
jason-fxz
changed the title
jason-fxz
force-pushed
the
feat/chat-reasoning-effort
branch
from
August 14, 2026 04:09
190747b to
4ffcc10
Compare
ezutfen
referenced
this pull request
in Zutfen-LLC/FreeToken
Aug 27, 2026
* bench: add reproducible InferSwarm Phase 0 baseline harness Tooling and instrumentation only for InferSwarm issue #2 (Phase 0 baseline). No distributed execution, and no benchmark results: this branch was developed without the target RTX 3060, so it claims no measurement. Harness (benchmarks/inferswarm_phase0/, entry point benchmarks/phase0_baseline.py) - Executes the exact B1-B5 sweep declared in the InferSwarm Phase-1 success criteria, passing every value the criteria fix explicitly: --nvfp4-backend is always stated (EngineConfig defaults it to "triton", not "auto"), and --moe-cache-auto is always stated (the CLI, not the dataclass, applies it). - CORRECTNESS_REFERENCE is a separate subcommand: fixed configuration, explicit resolved NVFP4 backend, fixed --moe-cache-size, --moe-cpu-layers 0, greedy, full output text retained. Never selected by speed, never a comparator. - Frozen workload manifest (JSON + schema) pinning per class the fixture, its sha256, output-token count, sampling, ignore_eos and chat-template settings. W1/W3/W4 fixtures come from issue #3; only a clearly-labelled smoke-test example ships here. - Precommitted protocol: 2 warmups + 10 measured generations per (arm, class), recorded execution order, distinct session ids, reversed traversal for a second session. Overrides require --dev-smoke and stamp the run NON-CANONICAL. - Provenance capture with explicit nulls-plus-reason, a refusal to start a canonical run on missing provenance, and a refusal of any model revision that is not a 40-hex commit SHA. - Raw artifacts: one JSONL line per generation (warmups tagged, never dropped) with full inter-token timings; run status derived from expected-vs-observed counts so a summary cannot hide a missing repetition. No ratio is computed and no baseline is selected by the runner. - Hardware profile subcommand (GPU identity, PCIe link gen/width, topology, ft bench bw) plus a single-expert NVFP4 decode-GEMV microbenchmark, marked diagnostic-only. Runtime instrumentation (the narrowest additions the above needs) - freetoken/engine/runtime_report.py: the engine's RESOLVED configuration, read back off the live engine rather than re-derived - resolved MoE backend, resolved NVFP4 backend and whether the flag was inert for the executing expert path, whether _auto_cpu_layers locked layers, resolved cache slots/bytes, and whether the Marlin 992-slot cap applies and whether it bound. Shipped on the readiness ack and served by a new read-only GET /v1/instrumentation. - Prefill is measured where prefill happens: CUDA events bracket the prefill model forward in Engine.forward_batch and are summed per request across chunked prefill, so prefill throughput is not prompt_tokens/TTFT. Off by default, behind FREETOKEN_INSTRUMENT_PREFILL; a multi-request prefill batch is marked shared rather than split. Tests: 156 new hardware-independent tests (subprocess/server/GPU mocked). Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: backfill the resolved expert weight format into the run artifact The format is only knowable once an engine has loaded the banks, so the provenance document said "server not started yet" for the life of the artifact. Backfill it from what the arms actually reported, and record per-arm disagreement explicitly: criteria section 3 rule 4 holds the weight format constant across arms, so a mismatch invalidates the campaign rather than being smoothed into one value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: gate the Phase-0 harness on the prerequisites it precommitted to Review found that the harness could still produce an apparently canonical, COMPLETE, MEASURED artifact while a precommitted requirement had failed. The invariant it now enforces: a canonical-looking Phase-0 artifact exists only when every prerequisite, held-constant rule, workload-shape rule, instrumentation requirement, provenance requirement and repetition requirement was satisfied. Anything else is explicitly INVALID, INCOMPLETE or NON-CANONICAL. Campaign validity is now a first-class answer, separate from completeness (inferswarm_phase0/validity.py). run.json carries execution_status (COMPLETE / INCOMPLETE) alongside validity (VALID / INVALID / NON_CANONICAL) and a structured campaign_invalidations list with stable reason codes; the bare `canonical: true` boolean is gone, because it was the field a reader would mistake for a verdict. MEASURED now labels an observation, never the campaign. SUMMARY.md and run.json both lead with one of VALID CANONICAL CAMPAIGN / INVALID CANONICAL ATTEMPT / NON-CANONICAL DEVELOPER RUN / INCOMPLETE RUN, keyed off the overall campaign state rather than the repetition protocol alone -- --allow-missing-provenance leaves the protocol untouched and still makes a run non-canonical. `ft bench bw` is a session-level prerequisite, not a B2-local side effect (inferswarm_phase0/bench_bw.py). B2 resolves its fetch split from the profile AND B3's --moe-backend auto reads the same profile, so the refresh runs once before the sweep traversal in either direction -- a reversed session runs B3 first and would otherwise consume a stale profile. --no-bench-bw is refused for a canonical sweep; a failed command, an unreadable profile, or a profile benched on another card aborts before any server starts. The record pins the command, both timestamps, the return code, the GPU UUID, the resolved path, the profile contents and its sha256. Physical-GPU provenance is proven, not asserted (inferswarm_phase0/gpu.py). --gpu is required for a canonical run and resolved to a stable UUID through freetoken.gpu_select -- no second selector policy -- with both the UUID and the nvidia-smi index recorded; the resolved UUID is what every child process gets. After startup the UUID the engine reports for itself is compared with it, and a mismatch invalidates. The nvidia-smi query now asks for the index, without which a numeric selector cannot be correlated with a row. Workload shape and output length are contract, not commentary. A prompt outside its frozen class rule and a completion length that is not the requested one invalidate the canonical block; the observation is preserved and the prompt is never rewritten. The "~16,000" / "~128" tolerances the criteria leave informal are frozen in version control and reproduced in every artifact. Prefill instrumentation is a validity gate. Records are attributed by request uid (chatcmpl-<uid> against the record's own uid) rather than "newest above a sequence floor"; where the id shape is unrecognized, more than one candidate is ambiguous rather than resolved by guessing. Unavailable, disabled, missing, ambiguous, shared-batch and unusable timings each carry a stable code and each invalidates a canonical measured repetition. Warmups are exempt: they are discarded by construction. CORRECTNESS_REFERENCE forces greedy request sampling on the same frozen prompt fixture and records the override. --sampling-defaults none was not sufficient: the manifest states sampling in every request body and a request-level value beats a server default, so a realistic performance sampling would have made the correctness reference sampled -- with no seed to make it reproducible. The performance sweep keeps the manifest's frozen sampling. The single-expert microbenchmark now measures a single expert: top_k=1, timed directly. The grouped top-k step is retained as a separately named diagnostic and is never divided by top_k, because experts inside a grouped call execute concurrently. Both it and the new device-memory-bandwidth benchmark bind the process through FreeToken's own gpu_select path and record the UUID they actually bound, refusing rather than misattributing. Device (VRAM) bandwidth is measured, which issue #2 asks for and `ft bench bw` does not provide -- its ceilings are host DRAM and the PCIe link. A device-resident D2D copy over a working set far beyond L2, CUDA-event timed after warmup, every repetition reported, byte accounting stated both ways, labelled MEASURED and diagnostic-only. Provenance refusals: a dirty FreeToken checkout cannot be reproduced from its commit SHA, so a canonical run refuses it and names the modified paths; --inferswarm-commit must be a full 40-hex SHA; a Hugging Face snapshots/<sha> path and its models--<org>--<name> cache entry are reconciled with --model-revision / --model-repository and a disagreement is a refusal, while a non-snapshot path records "cannot cross-check" rather than guessing. Resolved configuration is audited against criteria section 2.3, and the report gained the fields it was missing: the resolved --max-prefill-length and --cache-type (both on SchedulerConfig, and --cache-type is rewritten by _adjust_config) and the resolved hybrid fetch fraction. A missing or null required field invalidates, as does a held-constant value differing across arms or B3 resolving to a MoE backend -- or an NVFP4 expert path -- that coincides with neither B1 nor B2. --dry-run now reports what would refuse the run, so a plan cannot read "CANONICAL" while it would abort. Tests: 169 new hardware-independent tests (294 in tests/benchmarks/, up from 125), covering every refusal and every invalidating path above. Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbyUuzs7m7peuxDwfjALco * bench: enforce usable Phase-0 NVFP4 calibration * Enforce canonical Phase 0 model and GPU * test: isolate Phase-0 microbench fake torch --------- Co-authored-by: Claude <zutfen@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Aug 29, 2026
…tch end multi-minute first-request stalls Forensics (2026-08-28/29, ridge64-67): the 'minutes to first token' class had TWO stacked causes, both paid on the first request after boot/idle: 1. Cold Triton autotune: every novel extend length (fla chunk_fwd keys on BC) and KV-depth bucket sweeps whole autotune grids INSIDE the request's forward. Measured first-hit stalls: 46s/70s/129s/270s on lengths 575/266/ 255/505; identical replays 3-6s. The existing engine warmup covered only [80,128,256,512,1024] with hand-built batches -- no +/-1 divisibility shoulders, no decode steps, no depth walk. 2. Page-cache eviction of the mmap'd expert banks (checkpoint lives on a shared spinning disk): raw-socket sentinel probe stalled 135.5s while the worker faulted 81MB (0.6MB/s) behind concurrent desktop IO. Warmup depth walks ran disk-bound at ~250 tok/s vs ~2000+ page-cache-warm; the '416.6s greeting' maestro session and the 4m23s prefill->decode gap match this contention profile. Fixes: - scheduler/warmup.py + Scheduler._prefill_warmup: 33-class extend-length ladder (1..16, 2^k +/-1, max_extend) each with one decode step, plus a chunked depth walk to serving depth through the REAL request path (admission, GDN slots, drain/free) -- run in launch.py BEFORE the ready ack so clients see loading, never an idle SSE stream. Bounded, env-gated (FREETOKEN_PREFILL_WARMUP, FREETOKEN_WARMUP_MAX_DEPTH), exception-safe (failure logs and serving continues), emits boot progress. - utils/prefetch.py: posix_fadvise(WILLNEED) the whole checkpoint at boot + KeepResident daemon re-advising every 120s (FREETOKEN_PREFETCH_MODEL, FREETOKEN_KEEP_RESIDENT_S). Steady state costs no disk IO; restores evicted pages at background priority between requests. - scripts/stall_sentinel.py: 45s raw-socket pipeline probe with per-probe api/worker read_bytes deltas; any response >10s snapshots per-thread state/wchan + io (the kernel-level evidence py-spy cannot get at ptrace_scope=1). This is what caught cause FlashML-org#2 in the act. - mtp_probe reads in _process_last_data/_free_req_resources now tolerate engine-less scheduler shells (fixes 5 pre-existing test failures). - step-probe clock fix (wrapper-owned timer; was reading _forward's _t0). Verified on ridge67 (fresh boot, warm triton cache, prefetch filling): x14 503s->12.5s (cache still filling) -> subsequent hits ~3s; x13 46s->3.0s; para 70s->2.5s; ~8.5k-token prompt 75.6s(yesterday, 1.4k!)->11.4s for 8.5k (~750 tok/s wall-clock including queue+detokenize); depth walk 280s->149s. Known issue parked: full-depth (>=~50k) warmup walk hits an HSA hardware exception -- depth capped via FREETOKEN_WARMUP_MAX_DEPTH=40960 until the >=50k-context fault is diagnosed (never exercised by any traffic before). tests: 93/93 scheduler suite (6 new warmup tests + prefetch gate test).
sueichen
pushed a commit
to sueichen/FreeToken
that referenced
this pull request
Sep 2, 2026
…P4b-1) FT_VULKAN_PHASE_SEGMENTS=1 runs the decode step through one segment graph per FA boundary on the FIXED-SLOT per-layer banks (dc.banks[il], 16 slots) instead of the full-resident banks (slot == expert id, ~18GB at the real model's scale). The router ids must round-trip through the host (_assemble_route_decode: LRU slot assignment + expert_blit) between the router and the MoE, which share one DAG, so each segment graph executes twice per step: run FlashML-org#1 harvests the ids (the MoE runs over the stale slot map, outputs discarded), the host assembles the slots, run FlashML-org#2 produces the real step outputs (the attention is deterministic, so both runs agree). Step executes: 2*N_segments + 1 head -- 21 at the real model's 10 FA boundaries (vs ~92 per-graph), 5 on the mock (vs 11). Mock 6-step decode matches the per-graph path within 1e-3; [perf] line prints the count. Co-Authored-By: Claude <noreply@anthropic.com>
sueichen
pushed a commit
to sueichen/FreeToken
that referenced
this pull request
Sep 2, 2026
…review 1/5) The segmented phase path ran each segment graph twice per step and assumed the two runs' attention outputs agree. That held on the mock (sharp router margins) but not in general: the round-1 MoE consumes the STALE slot map, so its outputs propagate through the segment's x chain and the deeper layers' attention inputs -- and router ids -- differ between runs. The real model's narrow top-8 margins can flip, which would silently load the wrong experts in run FlashML-org#2. Fix (reviewer-probed, coordinator-decided): the run-2 MoE's slot map must match run-2's OWN router ids, so every segment now re-checks the FIXPOINT CERTIFICATE: the routers' ids of each round are compared to the previous round's; on a mismatch the slots are re-assembled and the graph re-run, capped at _PHASE_SEG_MAX_ROUNDS (3). A segment that still fails certifies falls back to the per-graph path for the step (self._phase_seg_fallback_count, [warn] line), which is correct by construction (its MoE consumes the ids and the slot map from the SAME attention fetch). The certificate's precondition (the map is always the slots of the harvested ids -- never forged) is tested by driving flips through CORRUPTED BANK CONTENT instead. Also (Minors): FT_VULKAN_PHASE_SEGMENTS no longer overrides an explicit _use_phase_graphs=False; the [perf] line is gated behind FT_VULKAN_TRACE_LAYERS / FT_VULKAN_PERF. Tests: near-tied routers (device-tensor injection) + corrupted stale slots flip the certificate deterministically on the mock -- the round-3 re-assembly converges to the per-graph path (1e-3, bit-level in practice); with the cap at 2 rounds the step falls back and still matches; the sharp-margin mock certifies at round 2 every step with zero fallbacks. 31 passed (p4_phases + p2_decode); wider vulkan regression green except the 2 pre-existing test_graph failures. Co-Authored-By: Claude <noreply@anthropic.com>
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Sep 4, 2026
…L-org#2 — emission protocol exonerated Per-replay eager-vs-graph rows probe (same entry state, same staged z): step 0 graph_rows==eager_rows (both the known-healthy [1156,14,1156] signature); by step 3 the graph returns [91,107,26,198] while the eager control on the SAME state returns sane-class [1156,1156,16,16] -- a divergence far outside the ~8.5 nondeterminism envelope. The [sd2] breadcrumbs show the graph's rows changing character from step 1: the corruption is INSIDE the captured replay from replay FlashML-org#2, not the emission/staging protocol, and not the fused kernel (layer-0 y-dump stayed 0.0000 throughout -- faithful). Next rung: compare the graph's trunk TAPS (depths 6/20/34/48/62, already written by the captured forward) against the eager control's taps -- the first diverging depth localizes the guilty captured kernel class (suspects: an in-pool accumulate-without-init intermediate in the extend attention / fla chunk workspace, only manifesting from the second replay when the pool holds prior-replay values).
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Sep 4, 2026
…ergence at depth 6, guilty class bounded to layers 1-6 EAB probe extended with per-depth tap diffs (graph taps vs the eager control's, depths 6/20/34/48/62): step 0: rows match, tapdiff = 0.0 at ALL depths (bit-exact replay FlashML-org#1) step 3: rows diverge far outside the nondeterminism envelope, tapdiff {6: 5.3, 20: 11.1, 34: 14.6, 48: 15.7, 62: 37.0} — first divergence at depth 6, growing monotonically. With layer-0's dense projection proven bit-exact (y-dump 0.0000), the wrong values enter in layers 1-6: the GDN chunk path or the layer-3 attention. Mechanism consistent with all evidence: an in-pool intermediate ACCUMULATED without init — correct on replay FlashML-org#1 (fresh capture pool, effectively zeroed pages), wrong from replay FlashML-org#2 when the pool holds the prior replay's values; eager passes never show it because their fresh torch.empty allocations land on allocator-cleared pages. This also explains why re-capture-per-step (the heal era) was coherent on every step. Next: name the exact kernel in the 1-6 window (fla chunk workspace / extend-attention scratch), init-on-replay fix, then the FUSED_M8 default flip (banked 2x).
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Sep 4, 2026
…te still garbles no-probe — the extra-replay correlation sharpens
(1) _spec_pending_from_last_row now takes the anchor from input_ids'
tail when the prefill already appended its sampled token (stream-ground-
truth rule applied to the anchor; kills the logits-rederivation that
produced the 'TheThe' boundary duplicate). (2) detokenize.py skips
orphan msgs for uids deleted earlier in the batch instead of crashing
the worker (spec emission sends multiple msgs per step; a finished msg
can precede a stale one) — KeyError worker crash eliminated.
Gate-3 (no-probe, FUSED+NODRAFT): P0 still garbage-class
('TheThe[=//:=、...') — while the PUREGRAPH probe boot (ridge410, with
the ZAB hybrid's EXTRA GRAPH REPLAY between production steps) was
coherent. The never-tried keep-alive is an extra IDENTICAL graph
replay per step: next experiment = replay-twice-per-step, discard the
first result (also tests whether replay FlashML-org#1-after-gap is wrong and FlashML-org#2
right, reconciling R12's step-0 bit-exactness).
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Sep 5, 2026
…fying the slot-staleness conviction); the captured GDN COMPUTE itself is wrong on replay>=2 FREETOKEN_SPEC_TWIN=1: before each verify replay, copy the live GDN slot (conv+recurrent) into the pool's padding twin and point linear_idx at it — a capture-known slot with freshly copied bytes. Result on the frozen harness: step-0 match=True (canonical rows), step-1/2 match=False with rows BIT-IDENTICAL to the no-twin control. Identical bytes at a different slot change nothing ⇒ the captured kernel's slot read is fully live (pointer + index + bytes). The GDN-block divergence (tapdiff layer1=8.3, stored-K bit-exact) is therefore INSIDE the captured GDN compute on its second and later replays — the in-graph fla chunk invocation is deterministically wrong from replay FlashML-org#2 with live inputs and (offline) deterministic kernels: the accumulate/self-corrupt class in the graph-pool intermediates the captured GDN path reads before writing (the fla fwd_h h-buffer written per-forward but potentially read-back stale across REPLAYS — h is pool-allocated at capture; if the captured chunk_o's read of h can see the PREVIOUS replay's h when the current fwd_h's stores are... in-graph ordered, so the sharper suspect is a buffer the captured path reads that is only initialized at CAPTURE by the eager warmups and never re-written inside the graph). NEXT: instrument the captured GDN path's global-memory intermediates (h, v_new) across two consecutive replays (fingerprint before/after each) — the buffer whose bytes fail to refresh on replay FlashML-org#2 is the self-corrupt site; fix = re-initialize it inside the captured region (zero-store kernel at region start) or route it through the per-replay staging. Then step-N stability, text-verified graphed spec-deep, the depth ladder, and 60+ @72k DFLASH LIVE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a typed top-level
reasoning_effortfield toChatCompletionRequestand maps it onto the per-family thinking toggle, closing the gap where/v1/chat/completionsaccepted-but-ignored the field while/v1/responses(reasoning.effort) and/v1/messages(thinking.type) already honor their protocol-native equivalents.Semantics mirror the existing Responses handling (vLLM-compatible):
reasoning_effort: "none"disables thinking viamodel_meta.think_toggle_kwargs(per-family mapping:enable_thinkingfor qwen3/glm/gemma4,thinking_modefor minimax_m3, no-op for always-on families).chat_template_kwargs.reasoning_effortfor templates that grade it (gpt-oss).chat_template_kwargsalways wins; absent effort leaves behavior unchanged.Why
Clients speaking the OpenAI reasoning dialect had no way to disable thinking on this endpoint. Concrete failure: a 64-token title-generation request burns its entire budget on reasoning and returns empty content with
finish_reason: "length".Testing
tests/server/test_openai_api.py: 3 new tests mirroring the Responses-side coverage (enable/disable/family routing/kwargs precedence);tests/server/suite green (66 passed).reasoning_effort:"none"+max_tokens:64→ noreasoning_content, real content,finish_reason:"stop"reasoning_effort:"high"→reasoning_contentpresentreasoning_effort:"none"+chat_template_kwargs:{"enable_thinking":true}→ kwargs wins, reasoning present