Load and unload a table as pages, without std - #5
Merged
Merged
Conversation
added 6 commits
September 6, 2026 12:39
let bytes = table.unload();
let table = LinearTable::load(&bytes)?;
Rows live in a `Vec` while the table is in use and are pages only at rest, so
everything between a load and an unload runs at `Vec` speed because it is a
`Vec`. The format is a codec at the two ends rather than a storage engine
underneath, which is the thing this crate exists not to be.
**No I/O traits.** A load takes `&[u8]` and an unload returns `Vec<u8>`.
Nothing seeks and nothing reads incrementally, so the question of which
read/seek/write abstraction to adopt does not reach this far, and the crate
stays `no_std` with no runtime and no ecosystem attached. Whoever holds the
bytes decides how they got there.
The bytes are DataBucket data pages: a `GeneralHeader` per page, bodies
concatenating into one rkyv archive of the row vector. Not a WorkTable space
file, because there is no schema here to write a `SpaceInfoPage` about, and the
module says so rather than implying compatibility it does not have.
**A fingerprint page, because rkyv will not refuse.** Loading
`Vec<(u64, String)>` bytes as `Vec<(u64, u64)>` first succeeded and returned
correct-looking keys with values like 18446743180901052274, which is a
`String`'s relative pointer read as an integer. Nothing errored. That is the
reinterpretation DataBucket's own migration doc warns about, and validation
cannot catch it because a `(u64, u64)` archive has no invalid bit patterns. So
page 0 now carries a hash of the row type and a mismatch is refused by name.
The hash is FNV-1a over `type_name`: not stable across compilers and not
unique, which is why it fails toward refusing a load rather than toward
accepting one.
The body comes back in an `AlignedVec`. rkyv reads an archive in place and
needs it aligned, and a `Vec<u8>` is aligned to 1; the round trip passing
before this was the allocator being generous, not a guarantee.
The index is derived rather than stored, so a load rebuilds it in one pass
instead of carrying a second thing on disk that can disagree with the rows.
**Cannot merge yet:** `data_bucket_format` is unpublished, so the dependency is
a path into a sibling checkout. It is behind the optional `hydrate` feature,
and the default build is untouched.
The first version of this reached for a page format in another crate, and building that crate was the wrong answer to the question. It is gone. Hydrate is functions in this crate, and it depends on nothing but a serializer. That is also the only option that works. The container `data_bucket` provides is `std`, tokio for file access and eyre through its signatures, and a table that is no_std and alloc-only cannot take that on just to serialize a `Vec`. So the pages are this module's own and deliberately modest: a fixed 24 byte header of little-endian integers, then a body. No rkyv in the header, because rkyv puts an archive's root at the end of its buffer and a header found at a fixed offset should not have to care. Rows use rkyv, where it earns its place. Every page names the row type rather than only the first, so a file spliced onto another is refused at the page where they stop agreeing instead of being concatenated into nonsense. These are not WorkTable space files and the module says so. A WorkTable space opens with a page carrying a name, a schema and a primary key list, and a `Vec<(K, V)>` declares no schema to put there.
table.write(&mut FromStd::new(File::create(path)?))?; // a real file
let table = LinearTable::read(&mut FromStd::new(File::open(path)?))?;
table.append(&mut FromStd::new(appending), from_row)?;
The crate stays no_std. `embedded-io` supplies the two traits, and it already
implements them for `&[u8]` and `Vec<u8>`, so memory needs no adapter and a
`std::fs::File` needs one line of one. There is no second code path for the two
cases and no std feature gating them.
**Every page now stands alone.** It was one rkyv archive split across page
bodies, which meant a single damaged page destroyed every row in the file and
an append rewrote everything. A page now holds an archive of exactly the rows
that fit in it, so damage is one page's problem and appending is writing more
pages onto the end.
**Every header field is checked**, which was not true before: `page` and
`pages` were written and never read, and a field that reads like a guarantee
and is never validated is worse than no field. Gone, replaced by fields that
are: a row count verified against what the body decoded to, a body length
bounded by the page, and a CRC-32.
The checksum is the one rkyv cannot do for us. Its validation says an archive
is structurally sound, which is not the same as saying these are the bytes that
were written: a flipped bit inside a u64 validates perfectly and reads back as
a different number. There is a test that flips one.
Measured on a real file, 200k rows and 12.7 MiB over 814 pages:
flush 67.3 ms 189.0 MiB/s 336 ns/row
open 52.2 ms 243.5 MiB/s 261 ns/row
The first version of the writer took **2343.8 ms**. Filling a page binary
searched over the whole remaining slice, re-serializing every row still to be
written on every probe, for every page: quadratic, and 350x slower than rkyv
encoding the same rows. Now each page starts from the previous page's row count
and walks, so a probe serializes about a page rather than a file. 35x, found by
measuring rather than by reading it.
`read` was wrong. It hydrates a `Vec`, which is a load, and the vocabulary is
the point of the module. So `_to` and `_from` say where and the verb stays the
same everywhere:
unload() unload_to(sink) append_to(sink, first)
load(bytes) load_from(source)
`unload_from(first)` becomes `unload_appending(first)`, because `from` had
started to mean two things in one API: which row, and which source.
`ReadError` becomes `HydrateError` for the same reason.
The module doc was also still advertising `flush(path)` and `open(path)`, an
API that never existed at all.
The workflow installed a target and ran a check against it to prove the crate is no_std. That proves something narrower and less useful: it pins a word size, and a dependency using AtomicU64 fails it for a reason that has nothing to do with no_std. `#![no_std]` enforces itself. A crate carrying the attribute cannot compile a `std::` path on any target, including the host, so the existing `--no-default-features` clippy and test steps already prove it.
I left this half-changed and the crate did not build. That is the first thing this fixes. The header is now DataBucket's `GeneralHeader` byte for byte: seven little-endian u32s in declaration order, which is what `rkyv::to_bytes` of that struct produces, since it has no relative pointers and pads `page_type` from u16 to four bytes. Verified against data_bucket 0.5.7 and pinned by `the_header_is_databuckets_layout`, because a layout reproduced in two places drifts and something has to notice. It is reproduced rather than imported because `data_bucket` is std, through tokio for file access and eyre in its signatures, and this crate is not. **Version 3 is the version that has a row directory.** WorkTable writes 2, and a 2 page carries no directory, which is exactly why a WorkTable data page cannot be read without its index: rows are bump allocated with `data_length` as a high water mark and no delimiters. Eight bytes at the tail of every page fix that here, a row count and a CRC-32, and `every_page_declares_its_own_rows` holds it: the pages account for every row with no index in sight. The count and checksum live in the directory rather than the header because the header is not this crate's to extend. That is the slotted page shape, and it is what makes the format readable by something that did not write it. The `space_id` field carries the row type fingerprint, since a `Vec<(K, V)>` belongs to no space. A WorkTable reader meets a space id it does not know, which is the honest outcome. Unchanged by any of it: flush 67.0 ms, open 52.0 ms over 816 pages.
The example needed the page size to split a file into pages, and reached for a `hydrate_page_size()` that does not exist. `PAGE_SIZE` is already a `pub const`; the module is private, so nothing outside could see it. Export it rather than adding a function that returns it. Page decode is the only embarrassingly parallel part of a load, so this is the measurement that says whether threading it is worth anything at all.
pathscale
force-pushed
the
feat/hydrate
branch
from
September 7, 2026 09:53
958ed5f to
1803f22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4. Review that one first; this branch's diff against it is the seven commits below.
A table you cannot put on a disk is a benchmark, not a component. This adds the whole persistence interface as two operations —
unload()to pages,load()back — and nothing else.What it is
PAGE_SIZEis 16 KiB, each page stands alone, and a page carries a row directory (format version 3, matching DataBucket) so a reader can find row boundaries without decoding the page before it.hydratetakesembedded-io, notstd::fs. A caller with an OS wraps aFilethroughembedded-io-adapters; a caller without one plugs in whatever it has. The crate stays#![no_std]on the real path, and only the tests linkstd, so they can put a page run on an actual disk and show the traits reach one.rkyvandembedded-ioare optional and pulled in only by thehydratefeature, because serialization is the one thing here that needs a serializer.Where an open actually spends its time
examples/where_time_goes.rs, 200,000 rows, 816 pages, median of 5, release:Decode is not part of a load, it is the load — the two measurements agree inside noise. That is the useful result: page decode is embarrassingly parallel and everything around it is free, so threading the decode is worth roughly the full core count. Nobody had to guess.
The example needed the page size to chunk a file and reached for a
hydrate_page_size()that never existed.PAGE_SIZEwas alreadypub constbehind a private module, so it is re-exported rather than wrapped in a function.Checks
no_stdis no longer gated on a CPU in CI — it was a target property being asserted about a runner.