fix(crawler): compute page body hash once per page instead of twice - #293
Merged
Conversation
`--dedupe-cap`'s observation and `update-page.ts`'s `page_meta.body_hash` write each independently called `computeBodyHash` on the same rendered HTML for every non-predicted internal page, since `--dedupe-cap` defaulted to enabled. That synchronous hash (body extraction + two regex passes + SHA-256) ran twice per page on the single-threaded crawl event loop, compounding CPU pressure that manifests as navigation timeouts and stalls under load. Generalize the predicted-only precompute in `#handleResult` to run for every internal page with a rendered HTML body, and forward the resulting hash through the `page` event, `Archive.setPage`, and `Database.updatePage` down to `update-page.ts`, which now prefers the precomputed value over recomputing it. The recompute path stays as a fallback for direct callers that have not precomputed one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--dedupe-cap's observation andupdate-page.ts'spage_meta.body_hashwrite each independently calledcomputeBodyHashon the same rendered HTML for every non-predicted internal page, since--dedupe-capdefaulted to enabled (Mark dedupe-cap trap pages post-hoc; default-enable --dedupe-cap #264). That synchronous hash (body extraction + two regex passes + SHA-256) ran twice per page on the single-threaded crawl event loop, compounding CPU pressure that can manifest as navigation timeouts and stalls under load.Crawler#handleResultto run for every internal page with a rendered HTML body (mirroringupdate-page.ts'swriteHtml && html.length > 0write gate), and forwards the resulting hash through thepageevent →Archive.setPage→Database.updatePage→update-page.ts, which now prefers the precomputed value over recomputing it.bodyHash ?? computeBodyHash(page.html)) stays as a fallback for direct callers that have not precomputed one (e.g. existing test call sites), so this is purely additive at every layer's public signature.Test plan
yarn lint— 0 errors (pre-existing warnings only)NX_WORKSPACE_ROOT_PATH=<worktree> yarn build— all 13 projectsyarn test— 4464 tests passing (3 new: page-event payload carries the precomputed hash;computeBodyHashcalled exactly once per internal page even with--dedupe-capenabled; external pages never trigger the computation sincesetExternalPagenever writes a body hash)yarn vitest run --config vitest.e2e.config.ts— full E2E suite (35 files / 166 tests), includingdedupe-cap.e2e.ts's real crawl → archive → query pipeline/code-review medium,/qa-engineer,/product-manager— findings addressed (explicitbodyHash: nullon the error-page emit path, unified@param bodyHashwording acrossarchive.ts/database.ts/update-page.ts, added missing coverage for the external-page skip case)