Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A request's I/O credit can be released while shared Bytes/Arrow allocations remain live. This draft adds opt-in read, decode and assembly APIs that admit a complete operation workspace before dependent work and retain each physical allocation's charge through its final alias. Refused attempts release inputs before replanning; dependent stages never wait for additional payload admission.
Implementation and scope
into_prefetched_batchesreserves one output slot before another attempt and shares the same payload pool. A stalled consumer cannot create an unbounded queue. Dropping the stream aborts its producer; dispatched work and retained aliases keep their charges. Panic, decode error and minimal-workspace refusal remain errors rather than EOF.The latest change retains a dataset cursor's successful prefix size and growth backoff across fragment boundaries. The measured implementation restarted the same oversized startup and growth attempts in each of twenty files. Only complete output commits the hint; cancellation and errors preserve it. Workspace changes still reset it, and periodic growth remains bounded so narrower fragments can recover. Every attempt still uses full admission. No payload cache, extra pool, or increased budget is added.
The preceding harness change exposes optional trailing
ATTEMPT_BYTESfor controlled comparisons, independently of I/O and shared-pool budgets. Defaults are preserved; ordinary mode rejects overrides.Ordinary scanner/take defaults, Sophon dependencies and file/wire formats are unchanged. Sophon's scan still calls ordinary
scanner.try_into_stream(); it does not adopt these APIs. This PR remains draft.Validation
Current head
121b625bb8a651fd730c7f698a513e319ca265cbpassed 101 targeted tests, 5 API examples, and strict Lance/example Clippy. Exact-source and no-test-util feature guards passed. The eight V2_2/V2_3 × Standard/lite × execution-order cases verify strict read reduction under identical cache conditions, complete output across uniform/narrower/wider fragments, growth recovery and final credit release. Existing boundary cancellation, workspace replanning and cursor/prefetch composition passed. All 43 checks passed, including Java attempt 3.The initial read-byte fixture failure was corrected by disabling its shared metadata cache and testing both execution orders, with output/budget/recovery assertions retained. The preceding scoped job reached its 45-minute limit during final lint after tests passed; the current commit only raised that limit to 60 minutes. Java's earlier failures were Maven dependency-resolution errors, also observed on main, followed by matrix cancellations; successful attempt 3 required no source/dependency change. Original failure, cancellation and retry logs remain retained. No local tests/builds ran.
Retained head
cbad8b4d969467b91ed15fd765e7815f6c6863cbpassed 38 harness tests and strict example Clippy. The exact-source and no-test-util feature guards passed, and root checks passed. Coverage includes original-command defaults, argument rejection, actual multi-fragment complete drains with explicit equal attempts, too-small-attempt refusal and credit cleanup, plus dictionary/default-codec and existing harness cases. Local fmt, diff checks and commit fmt/typos hooks passed; no local tests or builds ran.The preceding
f93677etest build failed because the out-of-line dictionary fixture omittedattempt_workspace. Setting it toNonepreserves its original mode defaults, budgets and assertions. That original failure is retained; that head's successful test execution validates the repair.Retained head
6673325bdfd1a9d72ff2f29a01d5449b488f7abfpassed 729 targeted tests, 5 API examples and strict encoding/file/Lance/example Clippy, plus root platform checks. Those checks cover the retained decoder/file implementation and earlier cursor behavior; they do not validate the new cross-fragment hint or harness arguments. Coverage includes raw/compressed Arrow oracles, exact/one-byte-short budgets, empty/null selections, full physical alias charges, malformed frames, replay cleanup, invalid UTF-8, independent-frame history rejection, actual cross-page files, multi-column FileReader composition and fragment/Dataset cancellation/prefetch. The feature guard excludeslance-io/test-utilfrom benchmark builds.The actual-file matrix verifies stored codecs rather than assuming hints dictate layout: V2_2 deliberately stores Zstd even with an LZ4 hint for released-reader compatibility; V2_3 honors LZ4. All 16 version/hint/scheduler/shape combinations remain covered. Earlier failed fixture expectations and their diagnostics are preserved.
Performance status
Current head
121b625bbcompleted the managed-VM comparison against immutablecbad8b4. All four processes / twelve complete outputs passed identity, feature, whole-output hash, resource, no-OOM and final-credit gates.cbad8b4121b625bbTwo runs per artifact in reversed order on the same n2-standard-16 Cascade Lake VM, cold guest caches, readonly SSD, 4 CPU / 4 GiB / no swap. Both artifacts use the same feature graph without test-util, prefetched mode, explicit 8 MiB attempts, 32 MiB shared pool, 16 MiB I/O/request, 128-row ceiling and three zero-delay requests, each returning 163,840 rows / 18,732,810,240 bytes. Every request produced 5,120 batches. Parent/current ranges: wall 67.857–72.532 / 67.472–68.867 s; child CPU 71.340–71.805 / 70.103–71.478 s; RSS 204.051–206.980 / 194.320–214.902 MiB. These small descriptive differences have overlapping ranges and do not establish a stable performance benefit. Cgroup peaks reached 4 GiB including page cache. Successful drain_all checks final credit; no independent serialized credit counter exists.
The exact-head build used Rust 1.98.1, configured mbx and release-with-debug. Candidate binary SHA256:
b4855a7d375e79f374af673d102eec29358b2485f5cd956d17c1412f67a9f567; feature SHA256:9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a. Comparison archive SHA256:ee1710eb7e746f0e75442da5b0c439c7c86e947919ec6964b15c1c5f41322947.One bounded current-artifact prefetched read-geometry capture is now running, using the accepted 600 s / 256 MiB limits and retained cbad trace. It will check whether repeated startup/growth reads decreased; instrumented timings will not be treated as throughput acceptance. No rebuild or attempt-size sweep is involved. A current-head ordinary baseline, S3 and server acceptance remain incomplete. The following measurements belong to explicitly named retained revisions.
The exact
cbad8b4optimized build and same-artifact fixed-attempt diagnostic are complete. All eight processes / twenty-four complete outputs passed source, binary, feature, output/hash, resource, no-OOM and final-credit gates.cbad8b4cbad8b4Both modes use the same immutable artifact and feature graph, n2-standard-16 Cascade Lake VM, readonly SSD, 4 CPU / 4 GiB / no swap, 32 MiB shared pool, I/O 16 MiB/request, 128-row ceiling, three complete 18,732,810,240-byte results and zero sink delay. Two reversed orders per treatment, cold guest caches before each fresh process. At each fixed attempt size, actual batches/request match: 5,120 at 8 MiB and 2,560 at 16 MiB. Wall ranges: admitted/prefetched 85.023–91.236 / 66.812–67.154 s at 8 MiB; 73.172–73.646 / 73.062–73.625 s at 16 MiB. All cgroup peaks reached 4 GiB including page cache. Two samples are descriptive; this diagnostic has no ordinary-mode baseline and does not clear the retained ordinary/server acceptance gaps.
Prefetch's measured benefit depends on attempt size; a larger attempt did not improve wall time here. The finite prefetched 4/8 MiB follow-up is also complete. All four processes / twelve outputs passed the same gates.
cbad8b4cbad8b4Same artifact, features and fixed settings as above; two reversed orders, fresh processes and cold readonly SSD. Wall ranges were 66.498–66.665 / 67.755–69.311 s, and actual batches/request were 5,120 / 10,380. Lower RSS did not translate into lower wall time. These two-sample observations do not justify a production budget change or attribute all added CPU to batch count. No further attempt-size sweep is planned from this evidence.
The bounded
cbad8b4ordinary/prefetched syscall diagnostic completed: both processes and all six full outputs passed identity/hash/resource gates. Prefetched read 59,741,673,579 data-file bytes versus ordinary 56,229,844,218, with the same covered file regions: 3,511,829,361 additional bytes. All twenty files show the same 128/64-to-32 startup and subsequent 64-row growth-attempt cadence. This matches the source's per-fragment hint reset and refusal/retry path; it does not attribute the complete wall-time gap. The candidate addresses that repeated reset while retaining bounded growth recovery. One capture per mode used 600 s / 256 MiB caps, neither reached. No failed or partial data reads occurred; four futex termination parse exceptions are retained. Instrumented timing and overlapping syscall/futex intervals are not acceptance or critical-path proof. Archive SHA256:58319d33850f09bebccf00867487ea62c41678bf3f453992583882790ead10d3.The 4/8 MiB archive SHA256 is
07ce6c35f1e9f2a160802819954802d8b139067bef73bb563e729477df91481c.Retained
cbad8b4binary SHA256:1437c5c0061193041905cce5259f989bf302fdf3dfa65ce7f6516a84a63927f3; feature SHA256:9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a; fixed-attempt archive SHA256:425864dd783725b8043f12b07e3709526ac2bceb05c3f015f6071f90f08d3505. The remaining measurements below belong to retained6673325, not current head121b625bb. Accepted baseline artifacts remain unchanged.The retained direct-parent comparison used admitted mode for both artifacts. All four processes / twelve complete outputs passed source, binary, feature, whole-output hash, resource, no-OOM and final-credit gates.
2d3c9d16673325Both artifacts ran on the same n2-standard-16 Cascade Lake VM and readonly SSD, limited to 4 CPU / 4 GiB / no swap. Each process drained three complete 18,732,810,240-byte results. Shared pool 32 MiB, attempt/I/O 16 MiB, row ceiling 128 and zero sink delay were fixed. Medians use two runs per artifact in reversed order, with cold guest caches before each fresh process. All requests produced 2,560 batches. Baseline/current ranges: wall 75.630–80.448 / 71.723–72.742 s; CPU 82.025–82.168 / 65.772–67.035 s; RSS 214.727–222.688 / 171.582–185.305 MiB.
The same-head ordinary/admitted/prefetched comparison is now complete. All twelve processes / twenty-four outputs passed the same gates, but requested ordinary-path performance remains unmet:
667332566733256673325Wide admitted/prefetched wall remains 1.246x / 1.121x ordinary; short prefetched CPU remains 1.235x ordinary. Lower CPU/RSS does not clear the wall-time gap. Wide ordinary RSS ranged 853.934–1239.262 MiB. Wide uses three requests, 32 MiB shared pool and readonly SSD; short uses one 15,682,496-byte result, 16 MiB pool and original HDD. Each workload fixes artifact, features, 4 CPU / 4 GiB / no swap, cold caches, I/O 16 MiB/request and 128-row ceiling. Admitted/prefetched attempts were 16/8 MiB: overlap and admission granularity were not isolated. Actual wide batches/request were 1,280/2,560/5,120; short was 1,024 for every mode.
Two samples per treatment are descriptive, not statistical significance or broad regression coverage. All wide cgroup peaks reached 4 GiB including page cache. Successful
drain_allchecks final payload credit; there is no independently serialized final-credit counter. Finite progress with per-batch yields does not prove general fairness.Retained artifact SHA256:
4794258366af3a3dbad28a7ed8fd65ea814eab38a41daf198b5df57dbb834158; feature SHA256:9815c08c1c3f59da37531d89791b3834ed77927b880d1bad932b90305251e65a. Archived direct-parent / modes evidence SHA256:b52c6e6c3c7f3b2ac768d48446b2e1b4937c5914eb969d19324058db86c6a058/7cb2224bae64bf9f47999588811fc008d5e8429553395529a5d0b68498353f06. Historical negative matched-row/pool diagnostics remain retained; they do not authorize raising production budgets.Remaining acceptance conditions
General stored-struct/ancestor decoding, sparse visible leaves and unsupported codecs, spilled identifier/version inputs and missing children under stored structs remain incomplete. Metadata/task/channel descriptors, transport/cache and allocator overhead are outside the payload bound. Complete index/dictionary traversal may reread input for each selection.
Process memory, ordinary-path performance, concurrent cancellation, S3 cumulative results larger than RAM and distributed PE/RPC/QN/output acceptance remain unfinished. Low process RSS or payload credit returning to zero does not establish a process-wide memory bound. Sophon #9869 retains its performance and acceptance hold; #11119 remains a separate draft. No Sophon adoption, Ready/merge, release or deployment is part of this change.