Skip to content

fix(crawler): compute page body hash once per page instead of twice - #293

Merged
YusukeHirao merged 1 commit into
devfrom
worktree-composed-gliding-stearns
Aug 13, 2026
Merged

fix(crawler): compute page body hash once per page instead of twice#293
YusukeHirao merged 1 commit into
devfrom
worktree-composed-gliding-stearns

Conversation

@YusukeHirao

Copy link
Copy Markdown
Member

Summary

  • --dedupe-cap's observation and update-page.ts's page_meta.body_hash write each independently called computeBodyHash on the same rendered HTML for every non-predicted internal page, since --dedupe-cap defaulted to enabled (Mark dedupe-cap trap pages post-hoc; default-enable --dedupe-cap #264). That synchronous hash (body extraction + two regex passes + SHA-256) ran twice per page on the single-threaded crawl event loop, compounding CPU pressure that can manifest as navigation timeouts and stalls under load.
  • Generalizes the predicted-only precompute in Crawler#handleResult to run for every internal page with a rendered HTML body (mirroring update-page.ts's writeHtml && html.length > 0 write gate), and forwards the resulting hash through the page event → Archive.setPageDatabase.updatePageupdate-page.ts, which now prefers the precomputed value over recomputing it.
  • The recompute path (bodyHash ?? computeBodyHash(page.html)) stays as a fallback for direct callers that have not precomputed one (e.g. existing test call sites), so this is purely additive at every layer's public signature.

Test plan

  • yarn lint — 0 errors (pre-existing warnings only)
  • NX_WORKSPACE_ROOT_PATH=<worktree> yarn build — all 13 projects
  • yarn test — 4464 tests passing (3 new: page-event payload carries the precomputed hash; computeBodyHash called exactly once per internal page even with --dedupe-cap enabled; external pages never trigger the computation since setExternalPage never writes a body hash)
  • yarn vitest run --config vitest.e2e.config.ts — full E2E suite (35 files / 166 tests), including dedupe-cap.e2e.ts's real crawl → archive → query pipeline
  • Reviewed via /code-review medium, /qa-engineer, /product-manager — findings addressed (explicit bodyHash: null on the error-page emit path, unified @param bodyHash wording across archive.ts/database.ts/update-page.ts, added missing coverage for the external-page skip case)

`--dedupe-cap`'s observation and `update-page.ts`'s `page_meta.body_hash`
write each independently called `computeBodyHash` on the same rendered
HTML for every non-predicted internal page, since `--dedupe-cap` defaulted
to enabled. That synchronous hash (body extraction + two regex passes +
SHA-256) ran twice per page on the single-threaded crawl event loop,
compounding CPU pressure that manifests as navigation timeouts and stalls
under load.

Generalize the predicted-only precompute in `#handleResult` to run for
every internal page with a rendered HTML body, and forward the resulting
hash through the `page` event, `Archive.setPage`, and `Database.updatePage`
down to `update-page.ts`, which now prefers the precomputed value over
recomputing it. The recompute path stays as a fallback for direct callers
that have not precomputed one.
@YusukeHirao
YusukeHirao merged commit d16cf68 into dev Aug 13, 2026
9 checks passed
@YusukeHirao
YusukeHirao deleted the worktree-composed-gliding-stearns branch August 13, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant