Skip to content

Git causal bridge: staged Git projection, verifiable checkpoint journaling, and redb 4.2 sync (RFC §1–§22 + CB follow-ups) - #207

Open
geekgonecrazy wants to merge 55 commits into
devfrom
feat/atomic-sidecar-for-git
Open

geekgonecrazy wants to merge 55 commits into
devfrom
feat/atomic-sidecar-for-git

Conversation

@geekgonecrazy

@geekgonecrazy geekgonecrazy commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Git causal bridge (RFC-ATOMIC-GIT-CAUSAL-BRIDGE)

This branch implements the complete Git causal bridge on top of the current dev (bcbe2c5a, includes #198, #195, redb 4.2.0 + the pinned redb_2_6 shim). Merge-base is dev's tip; a squash or merge here lands cleanly.

What ships

RFC §1–§22 core (CB-1A → CB-10B)

  • Shadow Git pipeline (CB-1/2/3): colocated projection of Atomic views onto Git refs with content-addressed change storage (V3 format, FastCDC chunking).
  • Verifiable bridge checkpoint (CB-4/5B/5C): signed RFC §7.3 Anchors bound to the Ed25519 binding key, journaled Recorded preimage recovery, workspace-txn entry/exit protocols with remediation (Git-owned sequences, conflicted index, missing/stale checkpoints all fail closed with exact remediation reports).
  • Hook observation (CB-9A/9B): post-checkout, reference-transaction, and pre-commit advisory dispatchers journal external Git transitions as immutable evidence (git-events.jsonl → captured events → commit-time capture bindings), with commit interpretation closures and capture-token bindings for prepared ref ops.
  • Import path (CB-9A): plain-git commits import as foreign history, absorbed into the bridge's causal ledger.
  • Bridge watch daemon (CB-13D R7): bounded external-transition watch, suppressible with attribution marker (.atomic/bridge/suppressed-operation.json).
  • Diagnostics: atomic git bridge status/verify/reconcile/census, divergence notices, forensic status (read-only, no reconciliation).

CB follow-up cycle (::12–::25, all vault-attested)

  • ::16-::23 — agent turn provenance hardening (review R1-R7): dirty-turn Git-transition classification (RFC §10.2), exact managed-commit capture (ManagedGitCommitCaptured), durable refusals for unexplained transitions, journal-backed multi-agent concurrent hooks (ordered ledgers without external serialization), owner-scoped provenance checkpoint state machine (reserve/append/bind/acknowledge across process crashes).
  • ::24 — the full CB-13D follow-up: R1 (budgeted branch-only refresh, ephemeral-view-creation deferral, sandbox fencing), R2 (LockScanProof across 10 lock namespaces, fail-closed walkers), R3 (MetadataHint::verified_baseline), R4 (bounded Watchman probe + truthful Fallback + suppression attribution), R5 (crash-during-effect fixtures, linked-worktree equivalence), R6 (crashed-consumer claim replay exactly-once), R7/sweep (owner waiver recorded verbatim in vault).
  • ::25 — RFC §22 documentation sweep, §19 policy snapshot, closure audit, label audit, cross-stream matrix suite (§22.5).

dev sync (this PR's merge commit)

  • redb 2.6.3 → 4.2.0 with the in-place upgrade_legacy_database path (databases written by pre-4.2 binaries upgrade on open; both upgrade tests ported and green).
  • Ours' CB-12A/13D semantics restored on top of dev's journal machinery:
    • Git-only turns still classify (RFC §10.2); clean-turn refusals persist durably; watch notices drain before the next tool call.
    • Journal commits fail closed when an owner sink is present; without one, no provenance is written (dev's contract) and the session JSON remains the crash-safe evidence.
    • Observation-only turns publish through the fast gate with completion events; idle duplicate Stops return early without minting turns; programmatic journal turns (tool hooks without a visible turn start) still activate and publish.
    • Bare sandbox pointers resolve canonical identity on read-only opens; provisioned sandboxes keep their own working-copy records.
    • status/diff/hooks wait for transient writers (typed DatabaseBusy retried inside the opener, 10s) instead of failing fast; terminal hooks acquire turn-publication.lock before the boundary guard so a killed Stop releases it by process death alone.
  • Test-seam reconciliation: redb 4.2's graceful-shutdown marker is not a mutation signal (size-based pristine snapshots in cb0c/cb5c/doctor tests); scoped manifest + IDE JSON + git-JSON share one --json flag; empty drafts hide by default in view list (dev UX).

Verification

Suite Result
atomic-core lib 3654 passed, 0 failed
atomic-repository (lib + integration) 1509 passed, 0 failed
atomic-agent (lib + integration + doc) 1369 + 27 passed, 0 failed
atomic-cli (all suites, --test-threads=1) 2409 passed, 0 failed

The full journal multi-agent suite (database_owner_integration_test, 23 tests) is green end-to-end: concurrent hooks commit lossless envelopes, killed Stops release the publication lock and retry, crashed owners resume exactly-once, and scoped concurrent Stops never record sibling files.

Vault / provenance

All work is intent-driven and attested in the Atomic vault (.vault/intents/): ::12 through ::25 are done with all criteria met; review ::3's canonical signed artifact survives as the AC-3 record per owner disposition (option b). Pushed to the Atomic storage remote (origin) and GitHub.

Ops notes

  • Databases written by pre-4.2 binaries upgrade in place on first open of this build (one-way).
  • redb is pinned to 4.2.0; the redb_2_6 companion (same crate, v2.6.3) exists solely for the legacy upgrade path.
  • Every commit fsyncs (redb 4.2 removed Durability::Eventual); the historical write-vs-write_txn_immediate split collapses onto durable commits, which only strengthens the receipt contract.

leefaus and others added 19 commits September 2, 2026 16:21
All ten Git causal bridge follow-ups remediated and attested:
- ::22 CB-13B: cutover readiness/resume/CLI entry + effect-bearing undo
- ::23 CB-13C: journal hardening, consent leases, truthful outcomes,
  synthesis/resurrection separation, measured 100k evidence
- ::24 CB-13D: budget fencing, lease proof, verified-truth baseline,
  bounded watcher, crash-safe notices, linked-worktree equivalence
- ::25 final certification: sweep table, §19 snapshot, closure audit,
  cross-stream matrix suites (sha256/reftable/sparse recorded as honest
  libgit2 failures), AC-3 dispositioned per owner decision (option b)

Recorded through change #349 on feat/atomic-sidecar-for-git; everything
attested and pushed to atomic-storage.
Brings the branch onto dev's redb 4.2.0 engine with the redb_2_6
companion shim, so the branch binary opens the live repository the
installed dev-lineage binary upgraded, and keeps the branch mergeable.

Merged semantically (ours = CB bridge lineage, theirs = dev fixes):
- pristine: redb-4.2 upgrade-aware opens + legacy in-place upgrade tests
  grafted with the CB-13D capability fence and comprehensive inode scan
- agent provenance: dev's journal-sink state machine is authoritative;
  the CB-12A accumulator fallback stays sink-less only; EmptyTurn still
  publishes ledger turns; git-less repos classify clean turns as
  ObservationOnly; scoped/programmatic journal turns survive the fast
  gate; idle duplicate Stops no longer mint turns
- CLI: stop/session-end hooks acquire the turn-publication lock before
  the boundary guard; guard, status, and diff wait out transient
  writers; bare sandbox pointers resolve the canonical identity on
  read-only opens
- restore ours' CB-12A/13D semantics on top of dev's journal machinery:
  git-only turns still classify, clean-turn refusals persist, watch
  notices drain before tool calls; journal commits fail closed when an
  owner sink is present and skip provenance without one
- gate accumulator fallback writes (goal/reasoning/response) on sink
  absence so the journal is the single authority when owned
- publish observation-only turns through the fast gate with completion
  events; idle duplicate stops return early without minting turns
- bare sandbox pointers resolve canonical identity on read-only opens
- status/diff/hooks wait for transient writers (waiting workspace opens)
- git-less repositories classify clean turns as ObservationOnly and keep
  read-only/retry semantics (no phantom git-only turns)
- test seams: redb 4.2's graceful-shutdown marker is not a mutation
  signal (size-based pristine snapshots in cb0c/cb5c/doctor tests);
  scoped manifest + JSON status + git-json share one --json flag
@geekgonecrazy geekgonecrazy changed the title Feat/atomic sidecar for git Git causal bridge: staged Git projection, verifiable checkpoint journaling, and redb 4.2 sync (RFC §1–§22 + CB follow-ups) Sep 21, 2026
…ss all targets

- cargo clippy --fix + targeted fixes: unused imports/vars/mut, needless
  borrows, casts, next_back, derivable Default, collapsible ifs
- CliError: crate-level result_large_err allow (inline refusal payload
  by design; boxed costs more than the one-shot return)
- doc: dedup macro-doc fragments (/// → //), argv[1] links, private-item
  links, ambiguous Hash alias link
- allow-listed seam helpers (budgeted-with seam, instrumentation flags,
  projection/repair fns) with intent comments
- cargo fmt applied
…ymlink/dispatcher tests, cfg-gated journal opens
…for the crash window (daemon may win the reconcile race on slower CI)
…path test to linux (macOS FS refuses EILSEQ)
@geekgonecrazy
geekgonecrazy marked this pull request as ready for review September 21, 2026 18:08
@geekgonecrazy
geekgonecrazy force-pushed the feat/atomic-sidecar-for-git branch 3 times, most recently from 93cd069 to 873f088 Compare September 22, 2026 15:51
…s, libgit2 Advapi32 link

Windows:
- unix-gate raw-byte path handling (OsStrExt/OsStringExt) and unix-only
  tests/fixtures: cb9c chmod fixtures, cb9b non-UTF-8 env (constant ASCII
  stand-in), cb4a projection fixture (whole module), cb13d corpus mode
  delta, attribute_tests module, symlink-effect locals, rename_and_sync
  fsync (bare rename), prepared/staging references, watcher shims,
  orchestrator chmod-denial test
- RepoPath::from_native/to_native are exact for UTF-8 on windows instead
  of refusing (status/record were refusing every tracked path)
- vendored libgit2 1.8 references Advapi32 (GetNamedSecurityInfoW,
  RegOpenKeyExW) without linking it: each binary-crate build script emits
  cargo:rustc-link-lib=advapi32 (flags from unused wrapper crates are
  dropped from the link graph)

macOS:
- init_with_view stores the canonicalized root so init and open agree
  behind /var -> /private/var; ELOOP raw code accepted as 62
- tempdir path comparisons canonicalized on both sides

All: fmt/clippy/rustdoc clean under -D warnings; status strict
fail-closed behavior preserved
* feat: log every command to a daily file with tracing

Replace env_logger with tracing-subscriber. The terminal keeps
env_logger's behaviour; every command also appends info and above to
~/.atomic/logs/atomic.YYYY-MM-DD.log, one line per event with the pid
and spans. git import logs each commit in a commit{n, of, sha} span
with its duration, and the write loop's decisions.

* feat: log the git shim's decisions, hook evidence and bridge events

Add spans to bridge reconcile, switch, import_head, export, verify and
pre-push, with the reconcile observation and direction and each switch
step. Copy the evidence Git's hooks journal, every consented bridge
event, and the CLI's status messages into the log file. None of these
change what the commands do or print.
… processes (#225)

A plain `git import` now verifies, for each single-parent commit before
the last one, only the paths that commit touches. The last written
commit and every merge still verify the whole tree. Bridge reconcile
keeps the full-tree check on every commit.

Resolving the Git admin layout now runs one `git rev-parse` instead of
four, and `validate_working_copy` resolves it once instead of three times.
… bridge (#212)

A `.git` directory next to `.atomic` made every ordinary command (status,
add, record, diff, stash, tag, agent turn-end, ...) refuse with UnbornHead
or MissingCheckpoint until the workspace was anchored, and made add, tag and
status write or read Git index/ref state without consent. RFC §2 and CB-13C
scope the bridge to explicitly configured workspaces ([git.bridge] enabled,
default false).

- Repository::bridge_workspace_active(): the explicit opt-in, or a verified
  checkpoint written by an explicit bridge command (reconcile, import,
  anchoring, clone bootstrap).
- Ordinary workspace boundaries observe Git only for bridge workspaces; the
  repair boundary (begin_remediation_txn) is unchanged.
- The stash guard, `add` intent-to-add, `tag create` Git export and status
  Git overlays use the same predicate.
- Unanchored MissingCheckpoint/UnbornHead reports name the exact commands
  that resolve them.
- The operating guide documents the opt-in contract.

Tests: new git_bridge_opt_in_test (6, written first and failing), four
workspace_txn tests, and record_stale_conflict_cleanup_test now opts in
explicitly because it models an enrolled workspace with a missing checkpoint.
…, like Git (#213)

External filter drivers were run with `%f` replaced by the raw path, so a
path containing spaces or shell metacharacters was split or interpreted by
the shell and the driver received the wrong argument (e.g. "with space.txt"
arrived as "withspace.txt").

- Unix: substitute the path quoted exactly as Git does for filter drivers
  (sq_quote_buf in quote.c), so it always reaches the driver as one literal
  argument.
- Windows (cmd.exe): refuse to run a driver with a path that contains cmd
  metacharacters instead of substituting it.

Tests: driver_receives_the_path_verbatim_as_one_argument (written first,
failing on the unquoted substitution) and sq_quote_matches_git.
Brings in change signing, agent identity for hook recording, diff --json,
and the 0.19.0 version bump.

Conflicts resolved by keeping both sides:
- record options: signing identity added next to the existing fields
- change format: SIGNATURE section parsed alongside the strict V2 reader
- agent turn recording: the turn is signed, then recorded with the
  working-copy id
- session end: identity fields on the in-flight flush
- Cargo.lock: re-resolved from this branch; jcs-admit pinned to 0.1.0

Follow-up fixes needed to build and pass tests after the merge:
- atomic-core/Cargo.toml: ed25519-dalek was listed twice
- diff: default view label now comes from the workspace transaction,
  as the cb5b routing test requires
- new signing tests use the working-copy record/add signatures
Two tests in the shared test binaries could fail depending on which
other tests ran at the same moment:

- The projection failpoints and inside-window injections are armed
  through the process environment, which every test in the binary
  shares. A test publishing in the same window picked up a sibling's
  injected writer ("debug injection detached HEAD: object not found").
  Arming now takes a lock and names the arming test's repository root
  in ATOMIC_TEST_INJECT_ROOT; other repositories ignore the injection.
  With the variable unset the instrumentation behaves as before, so
  subprocess harnesses are unchanged.

- The shallow-clone test spawned git from the process cwd, which other
  tests move into temp dirs that are deleted when they finish ("Unable
  to read current working directory"). It now runs git from its own
  source repository.
Windows:
- lifecycle: fsync the refusal file through a writable handle
  (read-only handles cannot flush on Windows)
- materialize: expect the directory mode the platform reports
  (Windows has no permission bits)
- mark Git bridge projection tests ignored on non-unix; the
  projection refuses there by design. One test needs the file
  name `foo.`, which Windows cannot create.

macOS:
- compare the sandbox database path against the canonical root
Use the shared publication wait budget for the CLI's outer hook lock so
it honors the same timeout as the agent orchestrator. Exercise a timeout
and a successful retry after contention beyond the former 10-second limit.

Bound checkpoint-crash recovery through the subprocess helper and verify
that failpoints fire and duplicate Stops preserve one recorded turn.
Use observed filesystem mode in the interrupted snapshot-retention fixture.

Validated on draft PR #240: all eight CI checks passed, including Linux,
macOS, and Windows tests. Consolidates d2258ae and a692c24.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants