Skip to content

Respect Markdown fence boundaries during heading extraction - #1141

Draft
Adarsh-9182 wants to merge 1 commit into
OWASP:mainfrom
Adarsh-9182:fix/harvester-markdown-fence-boundaries
Draft

Adarsh-9182 wants to merge 1 commit into
OWASP:mainfrom
Adarsh-9182:fix/harvester-markdown-fence-boundaries

Conversation

@Adarsh-9182

Copy link
Copy Markdown

Summary

Fixes #1138.

Track the opening Markdown fence's marker and length during heading extraction.
Only a matching marker of at least that length, followed by spaces or tabs,
closes the block. Fence lines allow up to three leading spaces; four-space or
tab-indented code does not open or close a fence. Backtick fence openers reject
info strings containing backticks.

This keeps code examples from becoming harvested headings and preserves real
headings after valid closing fences.

Reproduction

For a four-backtick block containing a three-backtick line and # Fake,
unchanged upstream extracts Before and Fake while hiding After.
The fix extracts Before and After.

Regression coverage

Seven added test methods cover shorter or mismatched markers, longer closing
markers, indentation, closing-line suffixes, backticks in info strings, and
unclosed fences. All 14 heading extractor tests pass. Running these tests
against unchanged upstream previously produced 11 failing assertions/subtests.

Validation

  • make lint: passed, including the OpenAPI guardrail.
  • make test: 1,048 tests passed, 6 skipped.
  • Focused strict mypy for heading_extractor.py: passed.
  • Repository-wide make mypy: 2,063 existing errors in 193 files, identical
    to unchanged upstream after normalizing source line numbers. No added errors.
  • Independent review: no blocking findings; both files match the previously
    reviewed candidate on current upstream 00a7b8cb.
  • git diff --check: passed.

AI assistance

Codex assisted with the fix and validation. A separate review agent performed
read-only independent review.

Review status

Draft for maintainer triage: #1138 has not yet been acknowledged or assigned.

Fixes OWASP#1138. Track fence marker type and length and preserve valid opening and closing syntax during heading harvesting.
@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@atul-upadhyay-7

Copy link
Copy Markdown

@northdpole

Tested locally at e0884fc.

Before checking out this PR, I reproduced the reported heading-extraction behavior on my local main: ['Before', 'Fake']. On the PR branch, I verified the input with repr(text), using chr(96) to preserve the fence characters, and got ['Before', 'After'].

I also ran:
python -m unittest application.tests.harvester_test.heading_extractor_test -v

All 14 tests passed (OK). I only checked this reproduction and the focused heading-extractor tests, not the full suite.

Baseline and candidate reproduction Focused tests and tested revision

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harvester heading extraction misidentifies Markdown fence boundaries

2 participants