Repository navigation
Memory architecture M1 — Flat decode (Storage/Scope/TensorView, matmul via KernelRegistry, TraceSink) #1002
Description
Parent tracking issue for **M1 — Flat decode** of the SKEEP-003 memory architecture (umbrella: #932). Design docs: `docs/design/memory/memory-architecture-proposal.md` (proposal rev. 2, §-numbers below refer to it) and `docs/design/memory/memory-architecture-milestones-prd.md` (PRD, requirement IDs `Mx-Fy`/`Mx-Ay`) — both land with the SKEEP-003 docs PR; until merged they are attached to #932.
## Goal
Run a real 1B-class model with weights mapped outside the heap and a recycled `Forward` scope so memory over 1 000 decode steps is a flat line, and single-token decode on packed weights goes through the kernel registry with no special cases (#993/#991 class closed by construction).
## Sample app / acceptance vehicle
`skainet-decode` CLI + minimal Android activity (home decided in its own sub-issue: SKaiNET core vs SKaiNET-transformers `llm-apps`): prefill + N decode tokens, before/after table (peak RSS, RSS@1000, allocations/step, adapters/step, TTFT, tok/s, effective bandwidth), Perfetto trace.
## Consumes (proposal §9 phases)
P1 complete · P2 (`Storage` Heap/OffHeap/Mapped, `Owner`, `Scope` Model/Forward/Ambient, `TensorView`, `TensorData` façades, `TraceSink` + allocation events) · P3 for **matmul only** · the P2 prototype benchmark (decision #6) passed first.
## Acceptance criteria (PRD)
- M1-A1 flat RSS: RSS@1000 within 1 % of RSS@50 (JVM `-Xmx256m` and reference Android device)
-
M1-A2 peak RSS during load ≤ packed file + KV + Forward slab + 100 MB
-
M1-A3 zero Forward-scope allocations per decode step after warm-up
-
M1-A4 Quantized matmul dispatch skips rank-1 (single-token decode) activations, falls through to broken matmulGeneric #993 repro (rank-1 × packed weight) and Quantized matmul dispatch silently falls back to a broken generic kernel for rank>2 attention projections (ClassCastException: Byte cannot be cast to Float) #991 strict-subtype case run through the registry, finite output, no special-casing
-
M1-A5 decode tok/s and effective bandwidth ≥ develop − 3 %; matmul microbenchmarks within noise
-
M1-A6 packed matmul bit-identical to develop for all seven GGML encodings + ternary + TurboQuant
-
M1-A7 Perfetto trace: one track per scope, flat live-bytes counter, kernel spans labelled by TensorId
-
M1-A8 plan-vs-actual within 10 % (closes M0-A1)
-
M1-A9 develop green throughout;
TensorDatapublic API source-compatible (BCV dumps)Slices (sub-issues, one feature branch each)
Order: P2 spike (gates the Storage/Scope/TensorView slices) → TraceSink API → Storage (common+Heap, JVM off-heap+mapped, native/wasm) → Scopes → Layout+TensorView → TensorData façades (dense, packed) → trace exporters → memory debug mode → KernelKey + reference matmul → dispatch (common, JVM+packs) → plan-vs-actual → KV prealloc → decode sample (home decision first).
-
[S1.5] P2:
TraceSinkAPI + phase/kernel/adapter/allocation event types (no-op default) #1017 (S1.5) -
[S1.1c] P2: Kotlin/Native and WASM/JS
Storagebindings (malloc/mmap; WASM → Heap) #1020 (S1.1c) -
[S1.4a] P2: dense
TensorDataimplementations become façades overTensorView(bit-identical) #1023 (S1.4a)
The checklist is maintained by GitHub's sub-issue tracking on this issue.
Working rules (every slice)
- One slice = one branch
feature/<this-issue#>-<slug>fromdevelop= one PR (Closes #<this issue>), merged independently. developstays usable after merge: additive API or behind façade / opt-in; no default behaviour switch without golden-parity + benchmark evidence in the PR; deprecate withReplaceWith, never delete before a major (zero-consumer dead code may be removed with evidence in the PR). No new abstract members onTensorData/KernelProvider(defaults only); no data-class primary-constructor changes (secondary ctors only); BCV:apiDumpbothapi/jvmandapi/android,apiCheckclean.- Test gate before the PR (JDK 25,
scripts/pr-gate.shonce it exists):./gradlew jvmTest linuxX64Test assemble,./gradlew verifyNpmPins jsTest wasmJsTest wasmWasiTest,./gradlew apiCheck,./gradlew :skainet-test:skainet-test-java:test; storage/backend slices add:skainet-lang:skainet-lang-core:jvmBenchmarkand:skainet-backends:benchmarks:jvm-cpu-jmh:jmhbefore/after when a hot path changes. - Commits: conventional commit subject,
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>trailer. Do not editCHANGELOG.mdin slice PRs (parallel branches conflict on it; release notes are written at release time from the merged PRs).
Activity
TensorView/Storageaccess-path benchmark on JVM + Android, flat-RSS loop — report only #1016TraceSinkAPI + phase/kernel/adapter/allocation event types (no-op default) #1017Storagesealed type —Owner,StorageId,StorageClosedException,Heapon all targets #1018Storage.OffHeap(FFM MemorySegment / direct ByteBuffer) andStorage.Mapped(FileChannel.map) #1019Storagebindings (malloc/mmap; WASM → Heap) #1020ScopeModel/Forward/Ambient — JVM Arena bump allocator,reset(),retain(), escape check; ExecutionContext opt-in #1021Layout+ newTensorView(view(),materialize(format, scope), decodingget()) — the only thing kernels receive #1022TensorDataimplementations become façades overTensorView(bit-identical) #1023TensorDataimplementations (Q4_0…Q8_0, Q4_K/Q5_K/Q6_K, ternary, TurboQuant) become façades overTensorView#1024SKAINET_MEMORY_DEBUG=1— allocation-site tags, use-after-close with closing stack, adapter log, ExecutionContext hooks, leak check #1026KernelKey+ rank normalization + reference matmul via defaultedKernelProvider.kernelFor(key); #993/#991 through the registry; kernel sample #1027DefaultCpuOps.matmul(commonMain) through the registry + adapters;is-ladder becomes fallback, then removed #1028DefaultCpuOpsJvmmatmul arms → registered kernels; provider packs (reference always present, capabilities in the key) #1029MemoryPlan, printed; CI assertion (> 10 % fails) #1030Scope.Modelto ctx (KvCacheStore), TurboQuant store included #1031skainet-decode(SKaiNET core vs SKaiNET-transformers), then build the CLI + Android activity #1032michalharakal commentedon Aug 23, 2026
M1 — Flat decode: wrap-up
All 17 slices are merged (#1016–#1032, plus the follow-up #1077 that #1032 uncovered). What M1 set out to build exists in
developand is exercised on every commit:Storage / Scope / TensorView (P2) —
Storagesealed withHeapeverywhere, JVM/AndroidOffHeap(FFMMemorySegment, directByteBuffer) andMapped(FileChannel.map), K/Nativemalloc/mmap, WASM→Heap (#1018–#1020).Scope.Model/Forward(pre-sized slab, bump-allocated,reset()per step,retain()the only escape) /Ambient(#1021).Layout+ the newsk.ainet.lang.memory.TensorView, the only thing kernels receive, with a decodingget()andmaterialize()as the single copy point (#1022). Every dense, narrow-float and packedTensorDatais now a zero-copy façade over a view (#1023, #1024).Dispatch (P3, matmul) —
KernelKey(op, operands[(Format, LayoutClass)], placement, capabilities), rank normalised once, adapters visible rather than implicit;DefaultCpuOps.matmuland the JVM arms route through the registry via a defaultedKernelProvider.kernelFor(key), with theis-ladder kept as fallback (#1027–#1029).Observability —
TraceSink+ phase/kernel/adapter/allocation events (#1017), Perfetto/JFR/android.os.Traceexporters (#1025),SKAINET_MEMORY_DEBUG=1(#1026), plan-vs-actual with a >10 % CI assertion (#1030). KV preallocated inScope.Modelto ctx (#1031), and the cache now declares itsFormatso the planner reads the KV width instead of guessing it (#1077).Acceptance criteria
DecodeAcceptanceTest.m1a1…— forward scope is 0 bytes between steps, per-step forward use is a single distinct value after warm-up, every reset ends at 0 live bytesm1a3…— 0 FORWARD allocations between steps 4→12; 0 MODEL allocations after step 1scripts/pr-gate.sh --golden) over all seven GGML encodings + ternary + TurboQuant, the merge condition for every encoding-touching slicem1a7…— one track per scope, kernel spans labelled byTensorId, live-bytes counter returning to"forward":0at each resetm1a8…—PlanVsActual.withinTolerance, adapter bytes 0@ExperimentalMemoryApi, BCV dumps (api/jvm+api/android) additive-onlyWhat M1 found, and fixed
The synthetic decode harness (#1032) was worth building for the two real defects it surfaced, both fixed before this tracker closed: adopted/mapped weights were invisible to plan-vs-actual (
ModelScope.adoptnow emits anAllocationwith site"adopted", balanced by aFreeon close), and the planner assumed bf16 KV against an FP32 ring — understating a dense cache by 2× (KvCacheMode.FP32as the stopgap, then #1077 making the store declare its ownFormat).Carried forward, deliberately
skainet-decode(SKaiNET core vs SKaiNET-transformers), then build the CLI + Android activity #1032 chose option (c): the harness lives in core (it runs on every target, every commit), the realskainet-decodeCLI + Android activity live in SKaiNET-transformers, where Llama/Qwen/Gemma and the decode loop actually are. That sample closes M1-A2, the tok/s half of A5, and produces the Perfetto screenshot. Tracked cross-repo, not here.DefaultCpuOpsJvmmatmul arms → registered kernels; provider packs (reference always present, capabilities in the key) #1029): they need block-major bytes while a packedTensorViewdescribes row-major file order. That contract is Packed-quant byte-order (block layout) is an unwritten, contradictory contract across the engine and downstream converters #973's subject and is a prerequisite, cross-referenced there.Closing. M2 (#1003) starts with the ternary encodings (#1033).