Repository navigation
feat(eval): private repos in place, persistent public clones, --clean - #14
Merged
Merged
Conversation
- Private repositories (eval/local/) are run where they are on disk: no copy, the eval follows their code as it changes, and each run records the commit it saw and whether the tree was dirty. Only `.astrograph/` is written there (it ignores itself), so `git status` stays clean. - The skill is loaded with --plugin-dir instead of being copied into the repository, so the run with Astrograph writes nothing else either. - Public clones stay in ~/.cache/astrograph-eval between runs; `bun run eval --clean` deletes them to free space. Part of #3 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3
What
eval/local/<name>.jsonis now{ path, stack, tasks }(nocommit). The eval runs in the working copy itself: no copy on disk, and it follows the code as it changes. Each run inruns.jsonrecordscommitanddirty, so a result can still be traced to the code it saw..astrograph/is written into a repository. The skill for the arm with Astrograph is now loaded with--plugin-dir(a plugin generated undereval/results/<run>/plugin) instead of being copied into.claude/skills/of the repo..astrograph/ignores itself (.gitignorewith*), sogit statusis unchanged, and rg/Grep skip it, so the.git/info/excludeworkaround is gone.~/.cache/astrograph-eval(one shallow clone per pinned commit, reused by the eval and bybench).bun run eval --cleandeletes them and prints the space freed; private repositories are never touched.Why
Owner feedback: private repos already live on disk and change over time, so copying them wastes space and freezes stale code; public samples should stay ready between runs but be easy to drop.
Verified
bun test,bun run typecheck,bun run checkgreen.bun run eval nestjs-bko-callerson a private repo: ran in place,git status --porcelainempty before and after,runs.jsonhascommit: 7f4a8b2e2b12, dirty: false, init event listsastrograph-eval:astrographskill in the arm with.--cleanagainst a temporaryASTROGRAPH_EVAL_CACHE: removes it and reports the size; a second call reports nothing to clean.🤖 Generated with Claude Code