Repository navigation
feat: compact atomic.redb through libatomic maintenance - #241
Merged
Merged
Conversation
vinceblock99
marked this pull request as ready for review
October 7, 2026 23:55
geekgonecrazy
added a commit
that referenced
this pull request
Oct 8, 2026
* feat(contract): add libatomic, the canonical Atomic protobuf contract One crate owns the Atomic contract: the canonical protobuf sources (36 files, package atomic), protox + tonic-build codegen (no system protoc), the generated clients/servers in the atomic module (aliased proto), and check_contract.py — the descriptor contract gate that compiles the tree and enforces the declared invariants: scope, caller, effect and capability annotations on every method; read/write discipline and replay metadata echo; retired fields stay reserved; sandbox callers never receive broad authority; generation/snapshot fencing. PASS today at 97 RPCs across 15 services, contract revision 2. Status: DRAFT for alignment — add-only after review and the first tagged release. Purely additive: no consumer wiring in this change. * docs(contract): drop draft framing; state the add-only discipline * style(contract): rustfmt build.rs; clippy-allow the generated module * feat(libatomic): add the reference handlers — the service layer over the contract The daemon module implements the contract's service surface: the repository registry with per-repository serialization gates (one writable database handle at a time; read-only opens with a bounded wait so concurrent readers coexist with short-lived writers), the tonic service impls and their plain inner functions (query, mutation, vault, attestation, knowledge, the provenance journal port, maintenance, view), the provenance journal core and its in-process sink, and the domain/protobuf converters. The handlers are transport-neutral: a serving process wires them onto any transport through the generated servers, and any Atomic-speaking consumer can call the same handlers in-process — one implementation. Carries two small atomic-repository additions the handlers require: in-place revise (reword) and stash support, plus the vault entity scaffolds for CreateVaultEntity. * style(libatomic): rustfmt * feat(contract): wire extensions for full routing parity (add-only) ChangeRef gains a prefix arm (whole-store case-insensitive resolution; the target operation's own membership/guard errors speak). GetChange carries the versioned change bundle (atomic.change.v3) plus the provenance ledger graphs (atomic.prov.graph.v1) and the view sequence. Sessions carry the ledger bundles (record+turns, domain-shaped JSON), vault-derived intent counts, turn-to-intent resolution, and the session manifest data. Status reports needs_reindex; Log filters by paths/tags and includes inherited entries; AddFiles gains dry-run/force/directory/ no-recursive with a tolerant per-path report; Record carries an explicit author or named identity; Remove/Move gain dry-run and force; Insert gains the apply_dependencies escape hatch; CreateView gains the parent (anchored, unseeded) base; SwitchView carries an explicit dirty-check bypass; SplitView gains a dry-run analysis mode; Repair gains REINDEX_WORKING_COPY; KnowledgeGraph enrichment scopes to explicit changes; vault listings filter by identity. The unrecord preview now runs the SAME domain dry-run as the mutating unrecord — one guard surface (membership, dependents, emptiness). * feat(vault): rank memory context handler-side + memory identity filter GetVaultContext grows the retrieval seeds (query/intent/files/budget/ include_body, add-only) and a ranked ContextItem list: the gather+rank behind `atomic vault context` now lives in the handler (one ranking recipe, both transports), with the CLI rendering md/candidates/JSON from the wire items. ListVaultEntries' Memory arm honors the identity filter — only memories whose fresh attestation verifies under that identity list. * docs(contract): escape an HTML-looking angle bracket in a vault comment * feat(contract): wire the remaining listing, show, KG, attest, and revise surfaces (add-only) Extend the wire (add-only; 97 RPCs unchanged, checker PASS) so every remaining flag-form of the wired commands carries its data through the handlers: - VaultEntry carries the listing-surface columns: manifest kind, the attestation fresh/stale/none state, the DID-match-then-verify token, the memory kind/status/about columns, the whole-vault type/size/date columns, the goal started_at/turns, and the intent update report fields. ListVaultEntries gains the path-prefix, entry-type, and raw status filters; kind-absent is the whole-vault listing (the handler serves every entry, including the previously-unimplemented Goal arm). - GetVaultEntry resolves by vault-relative path and carries the versioned entry bundle (schema atomic.vault.entry.bundle.v1): the stored entry's domain serialization plus the RAW attestation sources (the tracked entry and the legacy sidecar), so clients run their existing lift/classification code over wire-carried inputs. - ViewInfo carries the own/inherited change-count split (view list JSON). - KGNode/KGEdge carry the full domain fields (source, metadata); KgSearchQuery gains the kind filter and the candidate pool; NeighborsQuery gains the traversal depth. - ListAttestations resolves one attestation by hash/prefix (the atomic.attestation.detail.v1 bundle: domain payload + per-view coverage) and serves the AI provenance summary (atomic.provenance.summary.v1) for the --summary/--pending modes. - GetChange reconstructs per-file before/after content server-side (the legacy no-file_ops diff fallback and the FileOps context padding). - ReviseRequest carries the content-modification mode (message, author override, positional paths) and the reword-form author; the domain gains revise_content — the unrecord/record/re-apply stack surgery. - InsertChanges honors apply_dependencies across all source arms (the --deps=false opt-out); UpdateVaultEntity gains the composed multi- field update arm applied atomically. - Repository::find_root accepts sandbox working trees (the .atomic- sandbox pointer), so service-layer resolution handles sandboxed cwd invocations the same way the CLI's root finder always has. * feat(contract): wire the insert dry-run/tag-from-view/multi-pick and restore single-file tails (add-only) Extend the wire (add-only; 97 RPCs unchanged, checker PASS) so the last local-dispatch tails of the wired commands route through the handlers: - InsertChangesRequest gains dry_run (the handler previews every arm's plan with the same domain call the local bodies make), tag_from_view (the `insert tag --from-view` source override), the promote arm PromoteCurrentView (the bare promotion resolves the current view, its parent, the missing set, and the shared-to-shared confirm gate server-side — the CLI holds no repository handle), the change_set arm (multi-pick: raw change references resolved handler-side with the insert resolver's full-parse/prefix/ambiguity semantics, applied with cherry-pick), and the single_ref arm (a raw single reference, applied with the single-insert semantics honoring --deps=false). - InsertChangesResponse carries the report fields the local renders need: the resolved source/target views, the confirm gate, the resolved change, the post-insert state, the skipped count, the materialize counts, the outcome conflict flag, and the still-on-disk conflict listing with kind/line/sides (per record, in file order). - PreviewRestoreRequest gains content_path: the single-file dry-run's pristine-bytes dump — the handler reads the file's content with the same domain call; PreviewMutationResponse carries pristine_content (absent = no pristine content, the CLI's not-found error). * feat(contract): wire the tag/provenance/remote/push-pull/sandbox/vault/triage-candidates/session and KG surfaces (add-only) Tags (TagService handlers: create/delete/list/show with the full record on the wire), provenance trace/show (ExportProvenance with the resolved change + per-graph trace bundles), the remote registry (ListRemotes + ManageRemotes with the add-default flag), push/pull (the network sync runs handler-side with the client's auth headers; the structured reports + stage errors carry the local renders), the sandbox trees (create/stage/seal), the vault surfaces (InitVault, ExportVault, DeleteVaultEntity, LinkVaultEntities, goal start/stop/resume, the memory write, the tool-result previews), triage candidates (the view resolution + candidate-set domain call), session fork/rebuild, and the KG extras (graph/callers/plan/ask/embed over QueryGraph + MaintainKnowledgeGraph, plus the CandidateSet Deserialize derives). New RPC: ExplainTurns (fully annotated). Checker PASS (98 RPCs). * chore: workspace lockfile for the libatomic service deps * feat(contract): adopt the #230 merged-database APIs in the service layer - state.change_store: ensure_database (the legacy-layout merge) then open_existing on the merged atomic.redb — redb refuses a second concurrent handle, so the journal-port handlers keep the sequential open/use/drop discipline under the gate. - DirectJournalSink holds NO handle: each journal operation opens the merged database, runs, and drops it (the orchestrator opens the same file; the two can no longer coexist as concurrent handles). - atomic-repository: sync the #230 database module (pub database mod for the CLI's root-finder predicate) + the CandidateSet/BaggageEntry/ Coverage Deserialize derives for the triage wire. * fix(cli): find #230 merged-database repositories with the root finder #230 merged pristine.redb and changes.redb into a single atomic.redb, so a fresh init no longer writes pristine.redb — but the CLI's root finder still required it and could not discover #230-created repos at all. The finder now applies the same predicate Repository::find_root uses (has_database: atomic.redb, or the legacy pristine.redb before the merge), and the discovery integration test asserts the merged database file. * feat(contract): route the full CLI command surface through the service layer The complete routing story the contract carries: every command family routes through the libatomic handlers (local in-process backend by default, reactor RPC when ATOMIC_SERVICE=reactor), with the render parity kept in the CLI over wire data. - atomic-cli: the rpc.rs hooks + src/service dispatch for every wired command (tags, provenance, remote registry, push/pull, sandbox, top-level split, agent explain, intent delete/link, memory write, vault materialize/summaries/init, vault goal lifecycle, triage candidates, session fork/rebuild, query KG extras, project init) - owner.rs and its integration suite removed: the database-owner command is dismantled by this port - routing parity tests for the newly wired surfaces - provenance_rpc_sink: resolve the Reactor checkout via ATOMIC_REACTOR_ROOT, and skip the suite when no Reactor checkout exists (CI runners) — the daemon binary is not buildable from this workspace alone - atomic-client: the socket-path crate the CLI service routing shares - workspace Cargo.toml/lock: the atomic-client member * fix(client): allow result_large_err on the transport crate The tonic client surface returns tonic::Status (176 bytes) from nearly every method — same opt-out the libatomic daemon surface carries. * fix(cli): allow result_large_err on the Status-returning stream collectors Same rationale as the transport crate: the collectors propagate tonic::Status (176 bytes) — the wire type every reactor call returns. * fix(client): gate the unix-domain transport so non-unix targets compile connect_channel is the socket connect: unix-only by construction. The non-unix arm keeps the crate's API shape (callers compile) and refuses connects with an explicit unsupported error. * fix(client): gate the socket transport imports unix-only * style(client): import order after the unix gating * docs(cli): log --all is real, not a hidden no-op The draft-view model filters inherited entries from plain 'atomic log' by design (a session draft shows the work it added), but the --all flag was still documented as 'currently a no-op' and hidden from help — stale from before the view-filter restructure. Unhide it and document what it actually does: the full ancestor chain. * feat: compact atomic.redb through libatomic maintenance (#241) * atomic-wasm * feat(canonical): sign attestations with a key held elsewhere `attest_value` splits into `prepare_attestation` (author, content hash and the exact bytes to sign) and `attach_proof`, so a signer that keeps its key outside this process — a browser's WebCrypto, a hardware token — produces the same `eddsa-jcs-2022` attestation the CLI does. `attest_value` is now those two halves around a local `Signer`. `atomic-canonical-wasm` exposes that to the browser: prepare, attach and verify attestations, plus the DID and canonical JSON of a document, all over JSON strings and byte arrays. `build.sh` builds it with wasm-bindgen; `smoke.mjs` signs with a non-extractable WebCrypto key, verifies, and checks tampering is caught. (cherry picked from commit e46c8a8) * feat(intent): attest with a key held elsewhere `atomic intent attest --prepare` prints the document an attestation signs and the exact bytes to sign, needing only the identity's public key. `--signed <file>` records an attestation a key holder produced from it: it must be signed by `--identity`'s key and attest the intent as it is now, so a signature over a stale or altered intent is refused. This lets a sandbox that holds only an agent's public identity attest as that agent, with the key kept by a signing service outside it. (cherry picked from commit c643ed7) * feat(token): name the server a token is for (aud) Self-signed request tokens now carry `aud`: the bare server URL they were minted for, normalised. A server that checks it refuses a token minted for somewhere else, so one leaked from one server can't be replayed at another within its five minutes. Verifiers that ignore unknown claims are unaffected. (cherry picked from commit d06dee6) * feat(repository): render a view's tree without touching disk `Repository::materialize_view_entries` hands each entry of a view — path, inode, kind, mode, bytes, content hash, conflict-marker line — to a sink, in path order with directories first, plus the view's Merkle state. Read-only: no working tree, stat cache or conflict state is written, so it runs beside other readers. It serves a remote sandbox its tree and baseline. The per-file rendering is now one function, `render_view_file`, shared with `materialize_parallel`, which keeps writing to disk as before. (cherry picked from commit 238384f) * Never treat a sandbox's pointer as untracked `.atomic-sandbox` showed up as untracked inside a sandbox, so `record --all` would record it; a remote pointer carries a token. Ignore it, and the sandbox's local cache `.atomic-sandbox.d`, like `.atomic`. (cherry picked from commit ac56c30) * Draft views: render them as themselves A draft's structural changes (files it adds, moves, deletes) wait in the deferred tree journal until the view is checked out; TREE is the checked-out view's. Rendering any other view — materialize_view_entries, so every caller of it — now projects that view's journal in a write transaction that is thrown away. Before, a file added on a draft view never appeared on it. The vault-indexing half of the original commit served the remote-sandbox cache and is not carried over. (cherry picked from commit 081e6eb) * Stage and seal a view as it renders materialize_view_to (behind `sandbox stage` and `seal`) listed files from TREE, which is the checked-out view's: a file another view added was missing from its image. It now writes materialize_view_entries, which projects the view's own tree. (cherry picked from commit 76aef1d) * docs(storage): the endianness rule, written down at the encoders A key that gets range-scanned is big-endian, so its bytes sort in the same order as the numbers they encode; a value that is only ever looked up by its exact key can be little-endian. The two rules meet in files that handle both, so decode an id with that type's own `from_bytes` rather than slicing bytes by hand — a mis-decoded id is indistinguishable from an absent one at runtime. Recorded at the encoders in `pristine/tables.rs` and as a section in AGENTS.md. Also carries two fixes to the in-memory view render from the original commit: `view_tree_items` fell through to `TREE` — the *current* view's — when it could not open a write transaction to project another view's, so a read-only repository answered a question about one view with another's tree; it now refuses. And its test. (cherry picked from commit 3e7ca25) * refactor(wasm): the wasm crate is atomic-wasm atomic-canonical-wasm named the crate it wraps twice over — atomic, twice — and it is the only wasm crate in the workspace, so the extra precision cost a word and bought nothing. atomic-wasm says the same thing in half the length, and the module a page imports becomes atomic_wasm. atomic-canonical itself keeps its name: that is the crate this one wraps, and it is a dependency, not a rename of it. Verified: builds for the host and for wasm32-unknown-unknown (787 KB artifact at the path build.sh expects), full workspace suite 9088 tests across 57 binaries, 0 failures. (cherry picked from commit 9b7defc) * test(repository): is the pristine's file lock exclusive across processes? One test binary, two roles: the parent holds a Repository open and re-runs itself as the child; the child tries to open the same repository and reports what happened. Whether the lock is cross-process-exclusive is an assumption every single-writer consumer (the CLI's bounded open-wait, any daemon-side gate) relies on, so the answer is recorded as a test. Carried over from the remote-sandbox branch, where measuring how far the owner could go under concurrent sandboxes made the question load-bearing. (cherry picked from commit 677d300) * chore: workspace lockfile for the external-signing and view-render deps * feat(libatomic): attest through the service with a key held elsewhere PrepareAttestation was unimplemented and RecordAttestation loaded a secret key and signed server-side, so an identity whose key is held elsewhere (a browser, a hardware token, a signing service) could not attest over the service layer — and any caller could mint an attestation under the daemon's default human key. The CLI's `intent attest --prepare/--signed` routed there, so the external-signer feature the surface was built for failed on the service path. - PrepareAttestation (intents): returns the document and the exact bytes a signer elsewhere must sign — the same preparation the local body computes — with only the identity's public key, gated on the pre-attest violations signing cannot fill. - RecordAttestation (intents): with a caller-supplied signature, no secret key is loaded. The signature must verify against the target AS IT IS NOW (a stale or altered intent hashes differently, so an old signature cannot land), the identity must be named — an unbound caller cannot borrow the daemon's default human key — and the proof attached is exactly the one the signature earned. Without a signature, plain attest still signs locally, as before. - The CLI routes `--prepare`/`--signed` through the service layer and prints the same JSON the local path prints. - `intent|memory validate --json` over the service path prints the same top-level report shape the local body prints (the wire carries each violation as one message string, so an entry is its message). This is the contract's attestation semantics (CONTRACT "Sandbox and vault writes": preparation returns public signing bytes under a read capability; recording verifies the supplied signature against the current target). --------- Co-authored-by: Vincent <vince@atomic.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
redb reuses pages freed by updates and deletes, but does not automatically shrink the database file. Add
atomic compact [--repository PATH] [--json]so users can explicitly return unused space inatomic.redbto disk and see the before/after sizes and reclaimed bytes.The command calls libatomic's new
MaintenanceService.CompactDatabase: in-process in local mode, or over the socket in Reactor mode. This keeps database maintenance behind the shared service implementation introduced in #239. Reactor mode checks support and fails without a local fallback if the service is unavailable or incompatible.Compaction takes the canonical repository gate and redb's exclusive lock, with a bounded lock wait controlled by
ATOMIC_DB_LOCK_WAIT_MS. Cancellation cannot release the gate while blocking work continues. It preserves history, views, provenance, change files and dirty working files; persistent savepoints are retained and reported as a blocker when redb cannot compact. Repeating the command is safe and may reclaim zero bytes; the contract documents that physical maintenance does not create a durable logical-mutation replay receipt.This is explicit space reclamation, not history pruning, change-file deduplication or database migration. Automatic compaction and hosted Storage cache coordination are outside this PR.
Draft dependent on #239, currently targeting
feat/libatomic-contractso the diff contains only this change. Retarget todevand revalidate after #239 merges. Supersedes the integration proposed in closed #238.Validation on macOS: full workspace tests exited successfully (Cargo reports 9,143 passed, 0 failed, 243 ignored; three reported passes self-skip without a private Reactor checkout). All 15 compact-specific tests passed, including real-handler socket coverage. The CLI semantic harness passed 23 checks. A fresh-repository CLI run reclaimed 5.78 MB (29.8%) while preserving history, views, change files and dirty main/sandbox files; subsequent restore, record and doctor checks passed. Protocol checks, formatting and strict libatomic Clippy passed. Strict workspace Clippy encounters six existing warnings in unchanged
rpc.rs/stash.rs; allowing those two lint classes yields a passing workspace run. Linux/Windows CI and deployed Reactor verification remain outstanding.