Skip to content

Diff the baseline's stored file checksums when git cannot reach it - #174

Merged
Ibrahimrahhal merged 12 commits into
mainfrom
cursor/incremental-scan-file-checksum-manifest-de65
Sep 20, 2026
Merged

Ibrahimrahhal merged 12 commits into
mainfrom
cursor/incremental-scan-file-checksum-manifest-de65

Conversation

@Ibrahimrahhal

@Ibrahimrahhal Ibrahimrahhal commented Sep 15, 2026

Copy link
Copy Markdown
Member

Incremental scans currently need git diff against the baseline's commit, which rules out every run where that commit is unreachable: a shallow clone, a detached HEAD, a directory unpacked from a tarball. Those projects analyze every file on every run, forever. This computes a checksum manifest over the archive as it is packaged, uploads it with the scan, and diffs against the manifest the baseline scan stored.

Releases as 1.15.0 — a minor, matching how the previous feature releases were numbered (1.12.0 incremental scanning, 1.13.0 mcp install). Cargo.toml is the only place the version lives; npm and pip take it from there at release.

What changed

Manifest. src/manifest.rs builds a sorted digest path list over the archive's own entries, hashed to a root and uploaded gzipped. Because it describes the archive rather than the repository, ignored paths, excluded globs and uncommitted edits cannot make the changed-file list disagree with what the scanner will read. decode verifies the body against the root it was advertised with and caps the decompressed size, so a truncated or corrupted download is refused rather than read as a smaller tree — which would report every lost file as deleted and drop its findings.

Diff. src/incremental.rs prefers the baseline's checksums and falls back to git diff when the baseline stored none, stored a version this client cannot read, or the download fails. The checksum diff names the baseline by scan id rather than commit, since the scan that wrote the manifest may have had no commit at all. It also covers the working tree by construction, so a dirty tree no longer needs --ignore-dirty-worktree to scan incrementally.

Baseline search order. The branch being scanned is asked about first, then origin/HEAD, then main, then master. A branch that has been scanned before has a nearer ancestor than trunk does, so diffing against trunk would list everything the branch changed before that scan on every run until it merges. Trunk is reached only when the branch has no scan of its own. Scanning trunk itself collapses to the previous behaviour.

Two-pass lookup. The first walk cannot ask the server for clean scans only — that is how a git-less scan reports itself, and those are the ones carrying checksums. The second walk asks for exactly those once no checksums have been found, so a clean scan sitting behind a page budget's worth of dirty ones is still reachable.

Reporting both refusals. When the checksum diff does not apply and git cannot answer either, the run prints both reasons rather than only git's. A message naming just the dirty tree sends someone looking for a bug in the dirty-tree rule when what actually happened is that the baseline stored no checksums — the state every project is in until its first scan under this change, and every project stays in against a deployment that does not store them.

A directory entry is not always spelled like one

The two implementations agreeing had been argued rather than measured, so scripts/differential_manifest_check.py (in the doghouse PR) now measures it: this CLI packages fifty trees of adversarially-spelled paths — separators that sort against their own characters, combining marks, emoji, quotes, tabs, case-only pairs, names on the hashing chunk boundary — records the root it computed for each, and doghouse rebuilds every archive. Eight disagreed.

All eight were one bug, and it is worse than a disagreement. The zip writer appends the slash that marks a directory only when the name does not already end in / or \:

let name_with_slash = match name_as_string.chars().last() {
    Some('/') | Some('\\') => name_as_string,
    _ => name_as_string + "/",
};

That second case is a separator on Windows and an ordinary filename byte on Unix. A directory named slash\ was stored as slash\, a name nothing reading the archive can tell from a file's — so extracting it writes an empty file where the directory belongs and the first entry beneath it fails with NotADirectoryError. The upload was lost, not one path; the manifest disagreement was the smaller half. Naming the entry slash\/ here leaves the writer nothing to decide. The manifest never held directories, so no root changes: the same tree hashes to what it hashed to before.

Neither repo's CI can run the sweep, so what it found is pinned on both sides — here as directory_entry_name, which is a plain function and runs unignored.

Review fixes

Four ways a file could go missing from a manifest without anything having looked at it. Each is a dropped finding rather than a slower scan, because a path in the baseline and not in the new manifest reads as a deletion.

A subdirectory was passing as the whole project. Packaging walks . and keys entries relative to it, so a scan from backend/ (or an Actions working-directory) uploaded {app.py, …} where the next root-level scan reports {backend/app.py, …}. Subtracting those says every file was deleted and every file was added. get_repo_info already refuses to label such a zip with the parent HEAD; the manifest gate now uses the same is_at_repo_root check. A tree with no git at all is not a subset of anything and keeps its manifest — that case is the point of the change.

Zip entry names carried OS separators. to_string_lossy on a Windows path yields src\app.py, and ZipWriter::start_file stores the string verbatim. A Windows laptop scan followed by Linux CI described every file under a name the other did not have. Entry names are now built by joining path components with /, which also leaves a Unix file literally named foo\bar as one component — a blind replace would not.

A path could be spelled in a way the format cannot hold. A Unix filename may contain a newline and the canonical form is one entry per line, so a\n<64 hex> b is one archived file that reads back as two. Refused whole rather than per-entry: dropping the one bad path is exactly the deletion being avoided. decode rejects the same characters, and splits on \n rather than using lines, which silently drops a trailing carriage return.

A newer baseline without checksums hid an older one with them (thanks @leenk7991). The two kinds are not interchangeable — a manifest can be diffed from any clone, a commit needs history this one may not have — so a manifest-less scan from an hour ago sent the shallow checkout back to a git diff it cannot run. Checksums now win over recency within the page.

Also: --disable-incremental no longer SHA-256s every file it packs for a manifest that is then discarded.

Two reported issues were assessed and left alone, with the reasoning written into the code rather than the thread. A bundled corgea-image-scanning-*.tar is excluded from the manifest because docker save is not byte-reproducible, and that does not leave a new image unscanned: fusion scans a bundled archive whether or not the changed-file list mentions it (test_bundled_archive_is_scanned_when_no_container_source_changed). And a release that adds an exclude glob resolves correctly rather than quietly — a path in the baseline and not in the new archive is reported as changed, so its findings are retired instead of carried over a file nothing will scan again, which is the state a full scan under the new exclude set would leave.

The three SSRF reports on config.get_url() are not valid here. That URL is the Corgea deployment the operator configured, self-hosted installs depend on it being arbitrary, and every other API call in the CLI reads it the same way.

Compatibility

Against a deployment without the manifest endpoints, the new response fields parse as absent, no candidate advertises checksums, and every run takes the git diff path it takes today. Every refusal in this module falls through to the full scan the run would otherwise have done.

Open in Web Open in Cursor 

cursoragent and others added 2 commits September 15, 2026 17:52
An incremental scan needs to know what changed, and the only answer the CLI
has today is a git diff in the clone it is run from. A shallow checkout cannot
reach the commit the last scan covered, a detached HEAD names no branch, and a
pipeline unpacking a tarball has no .git at all, so each of those analyzes
every file on every run.

Hash each file on its way into the archive instead. The digests describe what
was uploaded rather than what the repository holds, which is also the more
accurate question: ignored paths, uncommitted edits and untracked files all
make the two disagree, and the archive is what the scanner reads.

Only a whole-project archive gets one. A --target, --exclude or
--only-uncommitted run packs a subset, and a manifest of a subset would read
to the server as every other file having been deleted.

Co-authored-by: ibrahim <ibrahim@corgea.com>
The manifest travels with the chunk that completes the archive, since that is
the request the server registers the scan from; sending it with every chunk
would re-upload it once per 50 MB for a server that discards all but the last
copy. Its root travels beside the bytes rather than inside them, so a
truncated manifest costs a full scan instead of reading as a tree that shrank.

The scope verdict now comes back on the upload response, and it supersedes
what this run worked out beforehand. A manifest is diffed against a baseline
the client does not hold, so the local decision can be both confident and
wrong; it is held back and printed only when the response says nothing, which
covers a deployment predating the field and an upload that deduplicated onto a
scan already running.

--disable-incremental withholds the manifest, since the server would otherwise
scope the scan from it.

Co-authored-by: ibrahim <ibrahim@corgea.com>
@Ibrahimrahhal
Ibrahimrahhal marked this pull request as ready for review September 15, 2026 19:16
Comment thread src/manifest.rs
Comment on lines +168 to +177
pub fn encode(&self) -> Option<EncodedManifest> {
if self.entries.is_empty() || self.entries.len() > MAX_ENTRIES {
return None;
}
let canonical = self.canonical();
let root = format!("{:x}", Sha256::digest(&canonical));
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(&canonical).ok()?;
let body = encoder.finish().ok()?;
Some(EncodedManifest { body, root })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Quality - The encode method suppresses I/O/compression errors by converting them into None via .ok()?. This makes it impossible for callers to distinguish “manifest intentionally not produced” (empty or too many entries) from “manifest failed to encode” (e.g., gzip writer failure), which can cause incorrect fallback behavior and makes diagnosing failures harder. Consider returning a Result&lt;Option&lt;EncodedManifest&gt;, io::Error&gt; (or a custom error enum) so error cases are observable while still allowing “no manifest” as a valid outcome. View in Corgea ↗

More Details
🪄Fix Explanation: Manifest encoding now propagates I/O failures instead of silently converting them into absence. This preserves the distinction between an invalid/empty manifest and an encoding failure, improving error handling and diagnosability.
<bullet_point>"encode" returns "Result<Option<EncodedManifest>, io::Error>", exposing compression and write failures to callers. <bullet_point>"Ok(None)" still represents empty or oversized manifests, while "Err" represents an actual I/O failure. <bullet_point>Replacing ".ok()? " with "?" prevents "write_all" and "finish" errors from being silently discarded. <bullet_point>Explicit error propagation improves observability and lets higher-level code choose appropriate recovery, logging, or user-facing behavior.</bullet_point>

💡Important Instructions: Update every encode caller to handle the new Result, propagating or reporting io::Error while preserving the existing None handling.
Suggested change
pub fn encode(&self) -> Option<EncodedManifest> {
if self.entries.is_empty() || self.entries.len() > MAX_ENTRIES {
return None;
}
let canonical = self.canonical();
let root = format!("{:x}", Sha256::digest(&canonical));
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(&canonical).ok()?;
let body = encoder.finish().ok()?;
Some(EncodedManifest { body, root })
pub fn encode(&self) -> Result<Option<EncodedManifest>, io::Error> {
if self.entries.is_empty() || self.entries.len() > MAX_ENTRIES {
return Ok(None);
}
let canonical = self.canonical();
let root = format!("{:x}", Sha256::digest(&canonical));
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(&canonical)?;
let body = encoder.finish()?;
Ok(Some(EncodedManifest { body, root }))

Comment thread src/manifest.rs
Self::default()
}

/// Record one archived file. `path` must be the zip entry name, byte for

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Quality - Manifest canonicalization writes path directly into a line-based format without escaping. If a path contains a newline, carriage return, or other control characters, the canonical form becomes ambiguous (it can look like multiple entries) and the computed root can represent something different than intended. This is a correctness and maintainability problem because callers are told “byte for byte” matching matters, yet the encoding doesn’t enforce or validate that the bytes are safe for this format. Consider validating/rejecting paths containing \n/\r (and possibly NUL), or switching to a length-prefixed/binary encoding. View in Corgea ↗

More Details
🪄Fix Explanation: Added validation to reject paths containing control characters in both insertion and finalization steps, preventing ambiguous or incorrect manifest states and ensuring data integrity.
"insert()" now checks for control chars "\n, \r, \0" in paths before insertion, avoiding invalid data entry.
A "debug_assert!" warns developers about invalid paths to catch issues early during development.
The "finalize()" method rejects manifests containing any such invalid paths, preserving canonical form correctness.
These checks avoid subtle bugs from malformed paths and ensure the algorithm processes only well-formed data, improving robustness.

💡Important Instructions: Update any external path generation or input sanitization to avoid control characters, ensuring compliance with new manifest constraints.
diff --git a/src/manifest.rs b/src/manifest.rs
index da3b280..8004d94 100644
--- a/src/manifest.rs
+++ b/src/manifest.rs
@@ -141,6 +141,11 @@ impl Manifest {
     /// and against stored findings, and a path spelled differently in either
     /// place is a file whose old findings are carried forward untouched.
     pub fn insert(&mut self, path: String, digest: String) {
+        let invalid = path.chars().any(|c| c == '\n' || c == '\r' || c == '\0');
+        debug_assert!(!invalid, "manifest path contains control characters (\\n, \\r, or NUL): {}", path);
+        if invalid {
+            return;
+        }
         self.entries.insert(path, digest);
     }
 
@@ -169,6 +174,10 @@ impl Manifest {
         if self.entries.is_empty() || self.entries.len() > MAX_ENTRIES {
             return None;
         }
+        // Reject ambiguous paths to keep canonical form unambiguous
+        if self.entries.keys().any(|p| p.chars().any(|c| c == '\n' || c == '\r' || c == '\0')) {
+            return None;
+        }
         let canonical = self.canonical();
         let root = format!("{:x}", Sha256::digest(&canonical));
         let mut encoder = GzEncoder::new(Vec::new(), Compression::default());

To apply the fix, Download .patch.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

findings: good
fix: bad

The ambiguity is real because these control characters are legal in Unix paths and the protocol is line-based. Silently dropping the path from insert is unsafe because the file remains in the ZIP while its absence from the manifest can be interpreted as deletion; encoding should escape paths or fail the archive operation.

high: Line-based manifest accepts ambiguous paths

Unix filenames may contain newlines or carriage returns. Writing such paths directly into <digest> <path>\n can create apparent additional entries and make server-side manifest parsing differ from the actual ZIP entries.

Proof or reproduction:

manifest.insert("real.py\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa injected.py".into(), digest) serializes as two apparent entry lines despite representing one archive path.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two merge blockers, both the same class of bug this PR already names: a path that is missing or spelled differently in the manifest is treated as a deletion (or as a file whose findings never update).

archives_whole_project lets a git-subdirectory walk through, and zip/manifest keys keep OS separators. Either one can drop findings on the next scan without those files having been analyzed.

Open in Web View Automation 

Sent by Cursor Automation: pr-flow

Comment thread src/utils/generic.rs Outdated
Comment on lines +83 to +85
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
target.is_none() && user_exclude.is_none()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a whole-project archive when CWD is below the git worktree root. create_zip_from_target(None, …) walks . and keys the manifest by those CWD-relative names — the same subset-looks-like-deletion case this helper exists to refuse.

get_repo_info already returns None for a nested CWD so the zip is not labeled with the parent HEAD (is_at_repo_root). The manifest gate does not use that check. A scan from backend/ (or Actions working-directory) with --project still uploads a subtree manifest. The server’s baseline is the newest scan that carries one, so the next root / shallow-clone scan subtracts {app.py, …} from {backend/app.py, other/…} and drops every finding for a file this run never looked at.

--only-uncommitted is already covered because it is a target string. Non-git trees (tarball / no .git) must keep a manifest — that is this PR. Fail closed only when git discovery succeeds and CWD is not the worktree root.

The existing test only asserts the predicate, so this path is untested.

Suggested change
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
target.is_none() && user_exclude.is_none()
}
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
if target.is_some() || user_exclude.is_some() {
return false;
}
// Packaging walks `.`. Below the worktree root that is a subtree, and a
// subtree manifest reads as every other file having been deleted.
match Repository::discover(".") {
Ok(_) => is_at_repo_root("."),
Err(_) => true,
}
}

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two merge blockers, both the same class of bug this PR already names: a path that is missing or spelled differently in the manifest is treated as a deletion (or as a file whose findings never update).

archives_whole_project lets a git-subdirectory walk through, and zip/manifest keys keep OS separators. Either one can drop findings on the next scan without those files having been analyzed.

Open in Web View Automation 

Sent by Cursor Automation: pr-flow

Comment thread src/utils/generic.rs Outdated
Comment on lines +83 to +85
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
target.is_none() && user_exclude.is_none()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a whole-project archive when CWD is below the git worktree root. create_zip_from_target(None, …) walks . and keys the manifest by those CWD-relative names — the same subset-looks-like-deletion case this helper exists to refuse.

get_repo_info already returns None for a nested CWD so the zip is not labeled with the parent HEAD (is_at_repo_root). The manifest gate does not use that check. A scan from backend/ (or Actions working-directory) with --project still uploads a subtree manifest. The server’s baseline is the newest scan that carries one, so the next root / shallow-clone scan subtracts {app.py, …} from {backend/app.py, other/…} and drops every finding for a file this run never looked at.

--only-uncommitted is already covered because it is a target string. Non-git trees (tarball / no .git) must keep a manifest — that is this PR. Fail closed only when git discovery succeeds and CWD is not the worktree root.

The existing test only asserts the predicate, so this path is untested.

Suggested change
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
target.is_none() && user_exclude.is_none()
}
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
if target.is_some() || user_exclude.is_some() {
return false;
}
// Packaging walks `.`. Below the worktree root that is a subtree, and a
// subtree manifest reads as every other file having been deleted.
match Repository::discover(".") {
Ok(_) => is_at_repo_root("."),
Err(_) => true,
}
}

Comment thread src/utils/generic.rs Outdated
Comment on lines +177 to +184
let entry_name = relative_path.to_string_lossy().into_owned();
zip.start_file(entry_name.as_str(), options)?;
let mut file = File::open(&path)?;
io::copy(&mut file, &mut zip)?;
match manifest.as_mut() {
Some(manifest) => {
let mut tee = TeeWriter::new(&mut zip);
io::copy(&mut file, &mut tee)?;
manifest.insert(entry_name, tee.finish());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to_string_lossy() is OS-native, and ZipWriter::start_file stores that string verbatim (only start_file_from_path rewrites separators). On Windows the zip entry and the manifest key are src\\app.py; on Linux they are src/app.py.

The server subtracts manifests by those keys and matches them to stored findings. A Windows laptop scan followed by Linux CI (or the reverse) looks like every file was deleted and every file was added — the exact finding-drop this PR is trying to prevent. changed_files_since already documents the other direction: git stores / on every platform, and a backslash there is a filename byte, so a blind \\/ replace is wrong. Join path components so a Unix file named foo\\bar stays one component.

The zip entry and the manifest key have to be the same string. Normalize both.

Suggested change
let entry_name = relative_path.to_string_lossy().into_owned();
zip.start_file(entry_name.as_str(), options)?;
let mut file = File::open(&path)?;
io::copy(&mut file, &mut zip)?;
match manifest.as_mut() {
Some(manifest) => {
let mut tee = TeeWriter::new(&mut zip);
io::copy(&mut file, &mut tee)?;
manifest.insert(entry_name, tee.finish());
let entry_name = relative_path
.iter()
.map(|s| s.to_string_lossy())
.collect::<Vec<_>>()
.join("/");
zip.start_file(entry_name.as_str(), options)?;
let mut file = File::open(&path)?;
match manifest.as_mut() {
Some(manifest) => {
let mut tee = TeeWriter::new(&mut zip);
io::copy(&mut file, &mut tee)?;
manifest.insert(entry_name, tee.finish());

Comment thread src/utils/generic.rs
@@ -165,6 +201,9 @@ pub fn create_zip_from_target<P: AsRef<Path>>(
let large_file_options: FileOptions<()> = options.large_file(true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high: Changed image archives are invisible to manifest diffing

The ZIP includes every extra file, but the manifest deliberately excludes them. Two uploads with identical source files but different image archives therefore have identical manifest roots, allowing the server to report no changed files and skip analysis even though scanner input changed.

Proof or reproduction:

Build archive A with source app.py plus image.tar contents A, then archive B with the same app.py plus image.tar contents B; because the extra_files loop never calls manifest.insert, archive_a.manifest.encode().root == archive_b.manifest.encode().root while the ZIP entries differ.

Comment thread src/utils/generic.rs

let mut added_files = Vec::new();
let mut excluded_files = Vec::new();
// Hashing rides along on the copy that compresses each file, so the archive

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: Disabled incremental scans still hash every archived file

Manifest construction is decided solely from target and exclusion options, so a whole-project run with --disable-incremental still computes SHA-256 for every file even though start_new_scan always discards the resulting manifest.

Proof or reproduction:

For target=None and user_exclude=None, archives_whole_project returns true and TeeWriter hashes every copied byte; later file_manifest is unconditionally set to None when disable_incremental is true.

@corgea-security corgea-security left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review risk: 4/5.

High risk: manifest-based scoping can miss changed image archives and incorrectly reuse stale findings.

Critical or high-priority changes must be addressed.

Automatic approval was not submitted: automated review found critical or high-priority findings.

@corgea-security corgea-security added the dennis-reviewed Dennis completed an automated review label Sep 15, 2026
Incremental scans pick a baseline the same way as before -- the newest
completed clean scan of trunk -- and now work out what changed since it
from the file checksums that scan stored, falling back to git diff when
it stored none.

The checksums need no git history, so a shallow clone, a detached HEAD
and a directory with no .git scan incrementally instead of analyzing
every file on every run. They also describe the archive rather than the
repository, so a dirty worktree no longer needs --ignore-dirty-worktree.

The upload names the baseline by scan id for a checksum diff, since the
scan that wrote them may have had no commit of its own.

Co-authored-by: ibrahim <ibrahim@corgea.com>
@cursor cursor Bot changed the title Upload a content manifest so incremental scans survive a shallow clone Diff the baseline's stored file checksums when git cannot reach it Sep 16, 2026
cursoragent and others added 2 commits September 16, 2026 10:26
A project scanned without git records no branch, no commit and no dirty
flag, so the trunk-and-known-clean baseline lookup could never match one
of its own earlier scans -- the case checksums exist for.

A clone with no git now makes one branchless lookup, and a run carrying
its own checksums accepts a baseline with no commit and any dirtiness:
the manifest records what that scan analyzed, so neither is needed. The
git diff keeps both requirements, since it measures from the commit.

Co-authored-by: ibrahim <ibrahim@corgea.com>
Doghouse rebuilds the manifest of scans uploaded before this CLI
computed one, from the archive they uploaded. Two implementations of
one format only stay one format if something compares them, and a
disagreement here is silent: the next scan diffs against a manifest
spelling paths differently and decides the wrong files changed.

Three fixtures close the loop. A shared vector both suites assert one
root for. An archive a real CLI build wrote, so doghouse's assumptions
about zip layout are observed rather than guessed. And the manifest
doghouse rebuilt from it, decoded back here to the tree it started as.

Co-authored-by: ibrahim <ibrahim@corgea.com>
Comment thread src/manifest.rs Outdated
#[test]
fn a_body_that_is_not_a_manifest_is_refused_rather_than_read_as_an_empty_tree() {
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(b"corgea-file-manifest/9 blake3\n").ok();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Quality - The test intentionally discards the Result from write_all by calling .ok() and not asserting anything about success or failure. If gzip writing fails for any reason, the test will continue and may fail later in a less obvious way (or even pass while not actually exercising the intended behavior). Using expect(...)/unwrap() (like nearby tests do) keeps failures explicit and improves test correctness and debuggability. View in Corgea ↗

More Details
🪄Fix Explanation: The test now explicitly handles gzip write failures instead of silently discarding them. Using "expect" makes setup failures visible and prevents the test from continuing with incomplete data.
• Replaces ".ok()" with ".expect("gzip write")", ensuring a failed write causes an immediate, diagnosable test failure.

• Avoids silently ignoring the "Result" returned by "write_all", improving reliability and making the test’s behavior consistent with its intent.

• The failure message identifies the failing operation, reducing debugging time when compression setup breaks.

• Prevents "encoder.finish()" and manifest validation from running against potentially incomplete gzip data.
Suggested change
encoder.write_all(b"corgea-file-manifest/9 blake3\n").ok();
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(b"corgea-file-manifest/9 blake3\n").expect("gzip write");

Comment thread src/utils/api.rs
Comment on lines +1272 to +1282
let url = format!("{}{}/scan/{}/file-manifest", url, API_BASE, scan_id);
debug(&format!("Sending request to URL: {}", url));
let response = http_client()
.get(url)
.send()
.map_err(|e| format!("API request failed: {}", e))?;
check_for_warnings(response.headers(), response.status());
if !response.status().is_success() {
return Err(format!("API request failed with status: {}", response.status()).into());
}
Ok(response.bytes()?.to_vec())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSRF (🔒 Security, 🔴 High) - The function accepts "url" as a caller-provided API base URL and constructs a request destination from it without enforcing an HTTPS scheme, host allowlist, or private-network restriction. The resulting URL is passed directly to "client.get", so a user who can supply the base URL can direct the client to localhost, cloud metadata services, or other internal hosts. Requests may include the shared authentication headers, making this an authenticated SSRF primitive. View in Corgea ↗

More Details
🪄Fix Explanation: The fix restricts requests to HTTPS on the approved API host, rejects unsafe DNS destinations, pins the connection to a validated address, disables redirects, and limits response size to prevent SSRF and resource exhaustion.
- Parses and validates the base URL, requiring HTTPS on port 443 with no credentials and an exact match against "ALLOWED_API_HOSTS".
- Resolves the host before connecting and rejects private, loopback, link-local, multicast, unspecified, and other prohibited network ranges.
- Pins the client to the validated address using ".resolve(host, pinned_address)", preventing DNS rebinding during the request.
- Disables redirects with "Policy::none()" and verifies the constructed request remains HTTPS and targets the approved host.
- Reads at most "MAX_RESPONSE_SIZE + 1" bytes, rejecting responses larger than 10 MiB.

💡Important Instructions: Add or verify the required imports for Duration, Read, and ToSocketAddrs, then add tests covering invalid URLs, prohibited DNS results, redirects, and oversized responses.
Suggested change
let url = format!("{}{}/scan/{}/file-manifest", url, API_BASE, scan_id);
debug(&format!("Sending request to URL: {}", url));
let response = http_client()
.get(url)
.send()
.map_err(|e| format!("API request failed: {}", e))?;
check_for_warnings(response.headers(), response.status());
if !response.status().is_success() {
return Err(format!("API request failed with status: {}", response.status()).into());
}
Ok(response.bytes()?.to_vec())
const ALLOWED_API_HOSTS: &[&str] = &["api.corgea.com"];
const MAX_RESPONSE_SIZE: u64 = 10 * 1024 * 1024;
let base_url = url::Url::parse(url)
.map_err(|e| format!("Invalid API base URL: {}", e))?;
let host = base_url.host_str()
.ok_or_else(|| "API base URL must include a host".to_string())?;
if base_url.scheme() != "https"
|| base_url.port_or_known_default() != Some(443)
|| !base_url.username().is_empty()
|| base_url.password().is_some()
|| !ALLOWED_API_HOSTS.contains(&host)
{
return Err("API base URL is not permitted".into());
}
let port = base_url.port_or_known_default()
.ok_or_else(|| "API base URL must include a valid port".to_string())?;
let addresses: Vec<_> = std::net::ToSocketAddrs::to_socket_addrs(&(host, port))
.map_err(|e| format!("Failed to resolve API host: {}", e))?
.collect();
if addresses.is_empty()
|| addresses.iter().any(|address| match address.ip() {
std::net::IpAddr::V4(ip) => {
ip.is_private()
|| ip.is_loopback()
|| ip.is_link_local()
|| ip.is_multicast()
|| ip.is_unspecified()
|| ip.is_broadcast()
|| ip.octets()[0] == 0
|| (ip.octets()[0] == 100 && (64..=127).contains(&ip.octets()[1]))
}
std::net::IpAddr::V6(ip) => {
ip.is_loopback()
|| ip.is_unicast_link_local()
|| ip.is_unique_local()
|| ip.is_multicast()
|| ip.is_unspecified()
}
})
{
return Err("API host resolved to a prohibited network address".into());
}
let pinned_address = addresses[0];
let url = url::Url::parse(&format!("{}{}/scan/{}/file-manifest", url, API_BASE, scan_id))
.map_err(|e| format!("Invalid API request URL: {}", e))?;
if url.scheme() != "https" || url.host_str() != Some(host) {
return Err("API request URL is not permitted".into());
}
debug(&format!("Sending request to URL: {}", url));
let request = http_client()
.get(url)
.build()
.map_err(|e| format!("Failed to build API request: {}", e))?;
let client = reqwest::blocking::Client::builder()
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
.redirect(reqwest::redirect::Policy::none())
.resolve(host, pinned_address)
.build()
.map_err(|e| format!("Failed to build secure API client: {}", e))?;
let mut response = client
.execute(request)
.map_err(|e| format!("API request failed: {}", e))?;
check_for_warnings(response.headers(), response.status());
if !response.status().is_success() {
return Err(format!("API request failed with status: {}", response.status()).into());
}
let mut body = Vec::new();
(&mut response)
.take(MAX_RESPONSE_SIZE + 1)
.read_to_end(&mut body)?;
if body.len() as u64 > MAX_RESPONSE_SIZE {
return Err("API response exceeded the maximum permitted size".into());
}
Ok(body)

The first baseline walk cannot ask the server to drop scans that are
not known-clean: that is how a git-less scan reports itself, and those
are the ones carrying checksums. The cost is that dirty scans now reach
the client, and enough of them bury a clean baseline past the page
budget -- which is every project in the window before any of its scans
has stored a manifest.

So when the unfiltered walk finds nothing and a git diff is still
possible, ask again for clean scans only. A checksum baseline is ruled
out by then, so nothing is left for the filter to wrongly exclude, and
the extra request only happens on the path that was going to be a full
scan anyway.

Co-authored-by: ibrahim <ibrahim@corgea.com>
Comment thread src/incremental.rs
baseline.manifest_version.as_deref().unwrap_or("unknown")
));
}
let body = api::download_scan_file_manifest(&config.get_url(), &baseline.id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSRF (🔒 Security, 🔴 High) - The API base URL returned by "config.get_url()" is passed into "api::download_scan_file_manifest" without validating its scheme, host, or destination address. That function then performs an authenticated HTTP request using the shared client, which can send the configured authentication token to an attacker-controlled internal or external endpoint when "CORGEA_URL" or the saved configuration URL is manipulated. This enables server-side request forgery and potential credential disclosure through the manifest download path. View in Corgea ↗
We could not generate a fix for this.

cursoragent and others added 2 commits September 20, 2026 12:10
Co-authored-by: ibrahim <ibrahim@corgea.com>
Co-authored-by: ibrahim <ibrahim@corgea.com>
Comment thread src/incremental.rs
checksums_usable: bool,
require_clean: bool,
) -> BaselineLookup {
let url = config.get_url();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSRF (🔒 Security, 🔴 High) - The value returned by "config.get_url()" can be controlled through the configured server URL or environment overrides, but this code does not validate or allowlist its scheme, hostname, or resolved address. That value is passed directly as "&url" to "api::query_baseline_scans", which constructs and sends an HTTP request using the configured endpoint. An attacker who can influence the CLI configuration or environment could redirect authenticated baseline-scan requests to internal services or an attacker-controlled host, causing SSRF and possible token disclosure. View in Corgea ↗
We could not generate a fix for this.

Comment thread src/incremental.rs
) -> Option<BaselineScan> {
let scan = scans
.iter()
.filter(|scan| is_usable_baseline(scan, checksums_usable))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this takes the newest scan usable by either rule, so a newer scan without a manifest hides an older one that has one, and then the checksum diff is refused and we fall back to a git diff that a shallow clone can't do

can we prefer the newest candidate with readable checksums here?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already addressed: branch_baseline searches the current page for has_readable_checksums before falling back to the newest usable scan, and a dedicated test covers this ordering.

Comment thread src/utils/api.rs
}
// Root and version travel beside the bytes rather than inside
// them: the server recomputes the root from what it
// decompressed and refuses a mismatch, so a truncated manifest

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which part of doghouse recomputes the root?

looking at read_uploaded_file_manifest it only checks that it's 64 hex chars

Comment thread src/utils/generic.rs Outdated
/// every finding for a file this run merely left out would be dropped without
/// anything having looked at it.
fn archives_whole_project(target: Option<&str>, user_exclude: Option<&str>) -> bool {
target.is_none() && user_exclude.is_none()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't account for DEFAULT_EXCLUDE_GLOBS or .gitignore, which also decide what's in the archive, if a release adds one exclude glob, every newly excluded file reads as a deletion in the next diff and its findings get dropped

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manifest membership intentionally matches the archive. Files newly omitted by default excludes or .gitignore must appear deleted so findings for content no longer scanned are retired, matching full-scan behavior documented in manifest.rs.

Review found three ways a file could go missing from a manifest without
anything having looked at it, which is a dropped finding rather than a
slower scan:

- Packaging from below the worktree root keyed entries relative to the
  subdirectory and still called the archive whole-project, so the next
  root-level scan subtracted two spellings of the same tree.
- Zip entry names and manifest keys carried OS separators, so a Windows
  scan and a Linux one described every file under a name the other did
  not have.
- A Unix filename may hold a newline, and the canonical form is one
  entry per line, so one archived path could read back as two. Refused
  whole rather than per-entry: dropping the one path is the deletion
  this is avoiding.

Also prefer a baseline with stored checksums over a newer one without.
The two are not interchangeable -- a manifest-less scan from an hour ago
was hiding one from yesterday that had a manifest, and the shallow
checkout the manifest existed for then fell back to a git diff it cannot
run.

--disable-incremental no longer hashes every file it packs for a
manifest that is discarded.

Co-authored-by: ibrahim <ibrahim@corgea.com>
Comment thread src/incremental.rs
checksums_usable: bool,
require_clean: bool,
) -> BaselineLookup {
let url = config.get_url();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSRF (🔒 Security, 🔴 High) - The API base URL is obtained from user-controlled configuration through "config.get_url()" without any host, scheme, or private-network validation. It is then passed to "api::query_baseline_scans", which performs an HTTP request using the resulting URL, allowing requests to attacker-selected or internal endpoints. Because the shared API client may attach the configured authentication token, an attacker who can influence "CORGEA_URL" or the saved URL can also cause credential disclosure to the SSRF target. View in Corgea ↗

More Details
🪄Fix Explanation: The fix parses and strictly allowlists the baseline API URL before requesting it. Only HTTPS requests to "api.corgea.com" on port 443 without user credentials are accepted, preventing requests to attacker-controlled destinations.
- Replaces direct use of "config.get_url()" with explicit parsing via "url::Url::parse".
- Requires the "https" scheme, exact host "api.corgea.com", and effective port 443.
- Rejects URLs containing usernames or passwords, preventing credential-based URL parsing ambiguities.
- Invalid or untrusted URLs log a debug message and return "BaselineLookup::LookupFailed" before any request occurs.
- Converts the validated URL back to its normalized string form before continuing.

💡Important Instructions: Verify that the HTTP client does not follow redirects to other hosts, or reapply this same destination validation to every redirect target.
Suggested change
let url = config.get_url();
let configured_url = config.get_url();
let parsed_url = match url::Url::parse(&configured_url) {
Ok(url)
if url.scheme() == "https"
&& url.host_str() == Some("api.corgea.com")
&& url.port_or_known_default() == Some(443)
&& url.username().is_empty()
&& url.password().is_none() =>
{
url
}
_ => {
crate::log::debug("Baseline scan lookup rejected an untrusted API URL");
return BaselineLookup::LookupFailed;
}
};
let url = parsed_url.to_string();

Comment thread src/manifest.rs
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
encoder.write_all(&canonical).ok()?;
let body = encoder.finish().ok()?;
Some(EncodedManifest { body, root })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: Encoded manifests can exceed the decoder's size limit

encode() limits entry count but not canonical byte size, while decode() rejects canonical data over 64 MiB. A sufficiently large valid manifest can therefore be uploaded successfully but can never be decoded by this client, forcing every subsequent scan back to Git or a full scan. Refuse oversized canonical data during encoding.

Proof or reproduction:

let mut m = Manifest::new(); for i in 0..MAX_ENTRIES { m.insert(format!("{}/{}/{}/{i}", "x".repeat(240), "y".repeat(240), "z".repeat(240)), "0".repeat(64)); } let encoded = m.encode().expect("encode succeeds"); assert!(Manifest::decode(&encoded.body, &encoded.root).is_ok()); // fails: over MAX_DECODED_BYTES

@corgea-security corgea-security left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review risk: 3/5.

No critical or high-priority defects remain in the supplied diff. One edge-case manifest-size inconsistency can unnecessarily disable checksum incrementality for very large projects.

No critical or high-priority changes were found.

Automatic approval was not submitted: automated risk 3/5 exceeds approval threshold 2.

…an-file-checksum-manifest-de65

Co-authored-by: ibrahim <ibrahim@corgea.com>
Minor, matching how the previous feature releases were numbered
(1.12.0 incremental scanning, 1.13.0 mcp install): scanning from stored
checksums is new behaviour rather than a fix to existing behaviour.
Cargo.toml is the only source -- npm and pip take the version from it at
release.

Co-authored-by: ibrahim <ibrahim@corgea.com>
The zip writer appends the slash that marks a directory only when the
name does not already end in '/' or '\\'. That second case is a separator
on Windows and an ordinary filename byte on Unix, so a directory named
'slash\' was stored as 'slash\' -- a name nothing reading the archive can
tell from a file's.

Extracting one writes an empty file where the directory belongs, and the
first entry beneath it fails with NotADirectoryError, so the scan loses
the upload rather than one path. Doghouse rebuilding the archive counts
it as a file this manifest does not have, and the roots disagree.

Found by packaging fifty trees of adversarially-spelled paths and having
doghouse rebuild each one (scripts/differential_manifest_check.py there).
Eight disagreed, all of them this. The manifest never held directories,
so no root changes: the same tree hashes to what it hashed to before.

Co-authored-by: ibrahim <ibrahim@corgea.com>
@Ibrahimrahhal
Ibrahimrahhal merged commit 46b02c1 into main Sep 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dennis-reviewed Dennis completed an automated review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants