Skip to content

feat: admit owned read and decode allocations - #11072

Draft
BubbleCal wants to merge 207 commits into
mainfrom
yang/t210-miniblock-input-lifetime-20261114
Draft

BubbleCal wants to merge 207 commits into
mainfrom
yang/t210-miniblock-input-lifetime-20261114

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

A request's I/O credit can be released while shared Bytes/Arrow allocations remain live. This draft adds opt-in read, decode and assembly APIs that admit a complete operation workspace before dependent work and retain each physical allocation's charge through its final alias. Refused attempts release inputs before replanning; dependent stages never wait for additional payload admission.

Implementation and scope

  • Exact aligned allocations account for backing owners, scratch and reallocation overlap. Standard/lite readers fill reserved buffers. Scheduler credit and payload ownership remain separate, including detached I/O/CPU cancellation.
  • MiniBlock decoding preserves perf(encoding): push down MiniBlock range decode and O(1) bitpack take #10909 range decode. Supported FullZip/Constant byte/string pages, List/LargeList nesting, structural codecs and raw/LZ4 dictionaries use the original workspace. Windowed traversal validates complete domains while retaining selected data.
  • Charged concat, interleave, struct validity/splice and newest-first overlay assembly include input/output overlap. FileReader, fragment and Dataset APIs compose actual base/overlay files, deletions, inline identifiers/versions and supported missing columns.
  • Flat and single-list raw/LZ4/Zstd outputs materialize final Arrow buffers directly, preflighting final nodes and codec scratch in the original workspace. Shared frame parsing and the initialized writer preserve framing, signed offsets, nullability and UTF-8 checks. Zstd writes into a bounded uninitialized suffix, publishes only the verified written length and reuses one charged context per decode invocation. Independent frames cannot inherit preceding output as a dictionary. Other shapes/codecs retain the admitted block path. Unique consumed compact storage returns to the same workspace; live aliases retain its complete charge.
  • Small independent same-file reads can share dispatches; direct credit refunds avoid waking an empty queue. Dataset cursors commit complete batches only, reuse accepted prefixes and back off unsuccessful growth probes. Cancelled/failed calls do not commit probe state. Small shallow raw decodes can run inline; larger/compressed work keeps pure CPU dispatch.
  • into_prefetched_batches reserves one output slot before another attempt and shares the same payload pool. A stalled consumer cannot create an unbounded queue. Dropping the stream aborts its producer; dispatched work and retained aliases keep their charges. Panic, decode error and minimal-workspace refusal remain errors rather than EOF.

The latest change retains a dataset cursor's successful prefix size and growth backoff across fragment boundaries. The measured implementation restarted the same oversized startup and growth attempts in each of twenty files. Only complete output commits the hint; cancellation and errors preserve it. Workspace changes still reset it, and periodic growth remains bounded so narrower fragments can recover. Every attempt still uses full admission. No payload cache, extra pool, or increased budget is added.

The preceding harness change exposes optional trailing ATTEMPT_BYTES for controlled comparisons, independently of I/O and shared-pool budgets. Defaults are preserved; ordinary mode rejects overrides.

Ordinary scanner/take defaults, Sophon dependencies and file/wire formats are unchanged. Sophon's scan still calls ordinary scanner.try_into_stream(); it does not adopt these APIs. This PR remains draft.

Validation

Current head 121b625bb8a651fd730c7f698a513e319ca265cb passed 101 targeted tests, 5 API examples, and strict Lance/example Clippy. Exact-source and no-test-util feature guards passed. The eight V2_2/V2_3 × Standard/lite × execution-order cases verify strict read reduction under identical cache conditions, complete output across uniform/narrower/wider fragments, growth recovery and final credit release. Existing boundary cancellation, workspace replanning and cursor/prefetch composition passed. All 43 checks passed, including Java attempt 3.

The initial read-byte fixture failure was corrected by disabling its shared metadata cache and testing both execution orders, with output/budget/recovery assertions retained. The preceding scoped job reached its 45-minute limit during final lint after tests passed; the current commit only raised that limit to 60 minutes. Java's earlier failures were Maven dependency-resolution errors, also observed on main, followed by matrix cancellations; successful attempt 3 required no source/dependency change. Original failure, cancellation and retry logs remain retained. No local tests/builds ran.

Retained head cbad8b4d969467b91ed15fd765e7815f6c6863cb passed 38 harness tests and strict example Clippy. The exact-source and no-test-util feature guards passed, and root checks passed. Coverage includes original-command defaults, argument rejection, actual multi-fragment complete drains with explicit equal attempts, too-small-attempt refusal and credit cleanup, plus dictionary/default-codec and existing harness cases. Local fmt, diff checks and commit fmt/typos hooks passed; no local tests or builds ran.

The preceding f93677e test build failed because the out-of-line dictionary fixture omitted attempt_workspace. Setting it to None preserves its original mode defaults, budgets and assertions. That original failure is retained; that head's successful test execution validates the repair.

Retained head 6673325bdfd1a9d72ff2f29a01d5449b488f7abf passed 729 targeted tests, 5 API examples and strict encoding/file/Lance/example Clippy, plus root platform checks. Those checks cover the retained decoder/file implementation and earlier cursor behavior; they do not validate the new cross-fragment hint or harness arguments. Coverage includes raw/compressed Arrow oracles, exact/one-byte-short budgets, empty/null selections, full physical alias charges, malformed frames, replay cleanup, invalid UTF-8, independent-frame history rejection, actual cross-page files, multi-column FileReader composition and fragment/Dataset cancellation/prefetch. The feature guard excludes lance-io/test-util from benchmark builds.

The actual-file matrix verifies stored codecs rather than assuming hints dictate layout: V2_2 deliberately stores Zstd even with an LZ4 hint for released-reader compatibility; V2_3 honors LZ4. All 16 version/hint/scheduler/shape combinations remain covered. Earlier failed fixture expectations and their diagnostics are preserved.

Performance status

Current head 121b625bb completed the managed-VM comparison against immutable cbad8b4. All four processes / twelve complete outputs passed identity, feature, whole-output hash, resource, no-OOM and final-credit gates.

Scenario / metric, lower is better Baseline cbad8b4 This PR 121b625bb Benefit
Prefetched / group wall 70.195 s 68.169 s 1.030x median speedup
Prefetched / child CPU 71.573 s 70.791 s 1.011x lower median CPU
Prefetched / peak process RSS 205.516 MiB 204.611 MiB 1.004x lower median RSS

Two runs per artifact in reversed order on the same n2-standard-16 Cascade Lake VM, cold guest caches, readonly SSD, 4 CPU / 4 GiB / no swap. Both artifacts use the same feature graph without test-util, prefetched mode, explicit 8 MiB attempts, 32 MiB shared pool, 16 MiB I/O/request, 128-row ceiling and three zero-delay requests, each returning 163,840 rows / 18,732,810,240 bytes. Every request produced 5,120 batches. Parent/current ranges: wall 67.857–72.532 / 67.472–68.867 s; child CPU 71.340–71.805 / 70.103–71.478 s; RSS 204.051–206.980 / 194.320–214.902 MiB. These small descriptive differences have overlapping ranges and do not establish a stable performance benefit. Cgroup peaks reached 4 GiB including page cache. Successful drain_all checks final credit; no independent serialized credit counter exists.

The exact-head build used Rust 1.98.1, configured mbx and release-with-debug. Candidate binary SHA256: b4855a7d375e79f374af673d102eec29358b2485f5cd956d17c1412f67a9f567; feature SHA256: 9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a. Comparison archive SHA256: ee1710eb7e746f0e75442da5b0c439c7c86e947919ec6964b15c1c5f41322947.

One bounded current-artifact prefetched read-geometry capture is now running, using the accepted 600 s / 256 MiB limits and retained cbad trace. It will check whether repeated startup/growth reads decreased; instrumented timings will not be treated as throughput acceptance. No rebuild or attempt-size sweep is involved. A current-head ordinary baseline, S3 and server acceptance remain incomplete. The following measurements belong to explicitly named retained revisions.

The exact cbad8b4 optimized build and same-artifact fixed-attempt diagnostic are complete. All eight processes / twenty-four complete outputs passed source, binary, feature, output/hash, resource, no-OOM and final-credit gates.

Scenario / metric, lower is better Baseline admitted cbad8b4 Prefetched cbad8b4 Benefit / cost
8 MiB attempt / group wall 88.130 s 66.983 s 1.316x speedup
8 MiB attempt / child CPU 74.162 s 74.356 s 1.003x more CPU
8 MiB attempt / peak process RSS 140.500 MiB 200.916 MiB 1.430x higher RSS
16 MiB attempt / group wall 73.409 s 73.343 s 1.001x; no clear benefit
16 MiB attempt / child CPU 68.884 s 70.613 s 1.025x more CPU
16 MiB attempt / peak process RSS 178.896 MiB 277.879 MiB 1.553x higher RSS

Both modes use the same immutable artifact and feature graph, n2-standard-16 Cascade Lake VM, readonly SSD, 4 CPU / 4 GiB / no swap, 32 MiB shared pool, I/O 16 MiB/request, 128-row ceiling, three complete 18,732,810,240-byte results and zero sink delay. Two reversed orders per treatment, cold guest caches before each fresh process. At each fixed attempt size, actual batches/request match: 5,120 at 8 MiB and 2,560 at 16 MiB. Wall ranges: admitted/prefetched 85.023–91.236 / 66.812–67.154 s at 8 MiB; 73.172–73.646 / 73.062–73.625 s at 16 MiB. All cgroup peaks reached 4 GiB including page cache. Two samples are descriptive; this diagnostic has no ordinary-mode baseline and does not clear the retained ordinary/server acceptance gaps.

Prefetch's measured benefit depends on attempt size; a larger attempt did not improve wall time here. The finite prefetched 4/8 MiB follow-up is also complete. All four processes / twelve outputs passed the same gates.

Scenario / metric, lower is better Baseline 8 MiB cbad8b4 4 MiB cbad8b4 Benefit / cost
Prefetched / group wall 66.581 s 68.533 s 1.029x slower
Prefetched / child CPU 73.850 s 74.938 s 1.015x more CPU
Prefetched / peak process RSS 223.734 MiB 139.414 MiB 1.605x lower RSS

Same artifact, features and fixed settings as above; two reversed orders, fresh processes and cold readonly SSD. Wall ranges were 66.498–66.665 / 67.755–69.311 s, and actual batches/request were 5,120 / 10,380. Lower RSS did not translate into lower wall time. These two-sample observations do not justify a production budget change or attribute all added CPU to batch count. No further attempt-size sweep is planned from this evidence.

The bounded cbad8b4 ordinary/prefetched syscall diagnostic completed: both processes and all six full outputs passed identity/hash/resource gates. Prefetched read 59,741,673,579 data-file bytes versus ordinary 56,229,844,218, with the same covered file regions: 3,511,829,361 additional bytes. All twenty files show the same 128/64-to-32 startup and subsequent 64-row growth-attempt cadence. This matches the source's per-fragment hint reset and refusal/retry path; it does not attribute the complete wall-time gap. The candidate addresses that repeated reset while retaining bounded growth recovery. One capture per mode used 600 s / 256 MiB caps, neither reached. No failed or partial data reads occurred; four futex termination parse exceptions are retained. Instrumented timing and overlapping syscall/futex intervals are not acceptance or critical-path proof. Archive SHA256: 58319d33850f09bebccf00867487ea62c41678bf3f453992583882790ead10d3.

The 4/8 MiB archive SHA256 is 07ce6c35f1e9f2a160802819954802d8b139067bef73bb563e729477df91481c.

Retained cbad8b4 binary SHA256: 1437c5c0061193041905cce5259f989bf302fdf3dfa65ce7f6516a84a63927f3; feature SHA256: 9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a; fixed-attempt archive SHA256: 425864dd783725b8043f12b07e3709526ac2bceb05c3f015f6071f90f08d3505. The remaining measurements below belong to retained 6673325, not current head 121b625bb. Accepted baseline artifacts remain unchanged.

The retained direct-parent comparison used admitted mode for both artifacts. All four processes / twelve complete outputs passed source, binary, feature, whole-output hash, resource, no-OOM and final-credit gates.

Scenario / metric, lower is better Baseline 2d3c9d1 Retained 6673325 Benefit
Wide / group wall 78.039 s 72.233 s 1.080x speedup
Wide / child CPU 82.097 s 66.404 s 1.236x lower CPU
Wide / peak process RSS 218.707 MiB 178.443 MiB 1.226x lower RSS

Both artifacts ran on the same n2-standard-16 Cascade Lake VM and readonly SSD, limited to 4 CPU / 4 GiB / no swap. Each process drained three complete 18,732,810,240-byte results. Shared pool 32 MiB, attempt/I/O 16 MiB, row ceiling 128 and zero sink delay were fixed. Medians use two runs per artifact in reversed order, with cold guest caches before each fresh process. All requests produced 2,560 batches. Baseline/current ranges: wall 75.630–80.448 / 71.723–72.742 s; CPU 82.025–82.168 / 65.772–67.035 s; RSS 214.727–222.688 / 171.582–185.305 MiB.

The same-head ordinary/admitted/prefetched comparison is now complete. All twelve processes / twenty-four outputs passed the same gates, but requested ordinary-path performance remains unmet:

Workload / metric, lower is better Ordinary 6673325 Admitted 6673325 Prefetched 6673325
Wide / group wall 58.968 s 73.460 s 66.097 s
Wide / child CPU 122.032 s 66.832 s 71.031 s
Wide / peak process RSS 1046.598 MiB 182.662 MiB 214.164 MiB
Short / child CPU 0.192733 s 0.185849 s 0.238016 s

Wide admitted/prefetched wall remains 1.246x / 1.121x ordinary; short prefetched CPU remains 1.235x ordinary. Lower CPU/RSS does not clear the wall-time gap. Wide ordinary RSS ranged 853.934–1239.262 MiB. Wide uses three requests, 32 MiB shared pool and readonly SSD; short uses one 15,682,496-byte result, 16 MiB pool and original HDD. Each workload fixes artifact, features, 4 CPU / 4 GiB / no swap, cold caches, I/O 16 MiB/request and 128-row ceiling. Admitted/prefetched attempts were 16/8 MiB: overlap and admission granularity were not isolated. Actual wide batches/request were 1,280/2,560/5,120; short was 1,024 for every mode.

Two samples per treatment are descriptive, not statistical significance or broad regression coverage. All wide cgroup peaks reached 4 GiB including page cache. Successful drain_all checks final payload credit; there is no independently serialized final-credit counter. Finite progress with per-batch yields does not prove general fairness.

Retained artifact SHA256: 4794258366af3a3dbad28a7ed8fd65ea814eab38a41daf198b5df57dbb834158; feature SHA256: 9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a. Archived direct-parent / modes evidence SHA256: b52c6e6c3c7f3b2ac768d48446b2e1b4937c5914eb969d19324058db86c6a058 / 7cb2224bae64bf9f47999588811fc008d5e8429553395529a5d0b68498353f06. Historical negative matched-row/pool diagnostics remain retained; they do not authorize raising production budgets.

Remaining acceptance conditions

General stored-struct/ancestor decoding, sparse visible leaves and unsupported codecs, spilled identifier/version inputs and missing children under stored structs remain incomplete. Metadata/task/channel descriptors, transport/cache and allocator overhead are outside the payload bound. Complete index/dictionary traversal may reread input for each selection.

Process memory, ordinary-path performance, concurrent cancellation, S3 cumulative results larger than RAM and distributed PE/RPC/QN/output acceptance remain unfinished. Low process RSS or payload credit returning to zero does not establish a process-wide memory bound. Sophon #9869 retains its performance and acceptance hold; #11119 remains a separate draft. No Sophon adoption, Ready/merge, release or deployment is part of this change.

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer A-ci CI / build workflows enhancement New feature or request labels Nov 14, 2026
@BubbleCal BubbleCal changed the title feat(io): retain allocation credit through owned read payloads feat: admit owned read and decode allocations Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-deps Dependency updates A-encoding Encoding, IO, file reader/writer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant