Skip to content

gh-153569: return tokenizer tokens as source spans - #156482

Open
pablogsal wants to merge 4 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-offset-state
Open

gh-153569: return tokenizer tokens as source spans#156482
pablogsal wants to merge 4 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-offset-state

gh-153569: streamline tokenizer buffer hot paths

fced7ab
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
Unresolved review
succeeded Aug 28, 2026 in 6s