fix(ingestion): prune stale vectors when a previously-ingested file is skipped

[detail-app]

Detail bug report: View on Detail

Closes #222

Bug

SourceCodeFileIngestionProcessor.process() returned skippedFile() carrying the file URL before any prior-state cleanup whenever a previously-ingested file subsequently exceeded the 1 MiB limit (or became binary/blank). The runner then added the skipped URL to activeFileUrls unconditionally, which shielded the stale Qdrant points and local marker from purgeDeletedFileOrphans for the rest of the run. The stale chunks — contradicting the file's on-disk bytes and reachable by search with no exclusion filter — stayed in the collection until the file shrank back below the limit or was deleted.

Root cause: the oversized skip branch at the top of process() short-circuited before readFileIngestionRecord or any prune path was reached, and the runner treated every returned URL (including skipped ones) as active. Both behaviors were present verbatim in the initial source-code ingestion pipeline (499894a) and never modified since.

Fix

Treat a skip of a previously-ingested file as a removal rather than a no-op, reusing the existing strict prune path the runner's own orphan purger already takes.

• In process(), call pruneSkipIfPreviouslyIngested(...) at both skip-before-prune sites (the validateFileAttributes oversized branch, gated on instanceof Skipped so the Failed missing-filename branch is excluded; and the readAndFingerprintFile empty branch covering binary MalformedInputException and blank fileText.isBlank()).
• The helper calls IngestedFilePruneService.pruneCollectionFileStrict(canonicalCollectionName, fileUrl, null) — the same null-record prune the orphan purger uses, which deletes by URL payload and reconstructs chunk hashes from parsed chunks on disk.
• Gate on repositoryContext.storedFileUrls().contains(fileUrl) so a never-ingested oversized file never triggers a Qdrant deleteByUrl round-trip.
• Swallow IOException at WARN so a best-effort prune failure never escalates a Skipped to a Failed outcome (which would trip the runner's failed() > 0 terminal throw over cleanup that was best-effort to begin with).
• Add a class-level CRLF_INJECTION_LOGS SpotBugs exclusion for SourceCodeFileIngestionProcessor, matching the existing GitHubRepoProcessor / LocalDocsFileIngestionProcessor pattern — the new log.warn passes the path through renderPathForLog, which escapes CR/LF before formatting, but FindSecBugs cannot follow the custom sanitizer.

Testing

• Unit tests (SourceCodeFileIngestionProcessorTest, 6 new): oversized/binary/blank transitions prune prior state; never-ingested and unrelated-URL stored sets do not trigger a prune; a prune IOException keeps the outcome Skipped. Confirmed the four positive-prune tests fail without the production fix and pass with it.
• Runner test (GitHubRepoProcessorIsolationTest, 1 new): a skipped oversized URL stays in activeFileUrls (no double-prune) while a genuinely deleted file is still purged by the orphan purger.
• No-regression sweep: 145 tests across the service.ingestion, cli, application.ingestion, and domain.ingestion packages, 0 failures.
• Full JVM lane: 1170 tests across 140 classes, 0 failures (1 pre-existing assumption skip unrelated to this change).
• Lint/format/static analysis: spotlessCheck, pmdMain, pmdTest, spotbugsMain, spotbugsTest all pass; make lint (ast-grep + frontend + JVM static analysis) and make build pass.
• Shell contracts: both GitHub ingestion driver contract scripts (test_github_repository_metadata_contract.sh, test_github_sync_failure_contract.sh) pass, as do all 16 make test-shell scripts.
• End-to-end live-Qdrant smoke (not verifiable in this environment): started Qdrant 1.18.3 and stood up a mock OpenAI-compatible embedding server returning 2560-dim vectors, built the boot JAR with the Gradle-provisioned JDK 25, created the required Qdrant collections and payload indexes, and configured the SPRING_PROFILE/state-dir env. Run N (ingesting the sub-1 MiB sentinel file) reproducibly failed at phase=qdrant-replacement because the application's replaceUrlDocuments/doUpsert post-upsert identity verification rejects the mock's vectors, so the sentinel-bearing file was never committed to the collection and the oversized→prune transition could not be demonstrated end-to-end. Reproducing it would require a real qwen/qwen3-embedding-4b model server this sandbox cannot provide. The fix's behavioral guarantee is fully covered by the unit and runner tests above, which exercise the prune at the observable code boundary without the embedding/Qdrant runtime.

---

Automatic Fixes PRs can be configured here.