Repository navigation
Conversation
matches Python, blingfire is much more accurate compared to our basic tokenizer implementation.
🦋 Changeset detectedLatest commit: a7e92a5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 41 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| const sentence = rawSentence.replace(/\s*\n+\s*/g, ' ').trim(); | ||
| if (!sentence || sentence.length < minSentenceLength) continue; | ||
|
|
||
| sentences.push([retainFormat ? rawSentence : sentence, start, end]); |
There was a problem hiding this comment.
🟡 Batched speech duplicates sentence separators
With retainFormat and sentence batching, splitSentences retains leading whitespace in each token. BufferedTokenStream adds another space between tokens, changing the text sent to TTS.
Learn more
The splitter builds consecutive slices of the original input. A sentence after a boundary can therefore start with the source's separating space. In a streamed batch, BufferedTokenStream unconditionally inserts one more space before adding this slice to a nonempty output buffer. The same join occurs on flush. The default framework adapters now enable retainFormat, and expressive TTS batches later sentences.
Example: With minSentenceLength: 1, minTokenLength: 200, and retainFormat: true, streaming First sentence. Second sentence. Third sentence. into one batch adds spaces before Second and Third on top of the retained source spaces.
Recommended fix: In BufferedTokenStream, join slices without adding a separator when their tokenizer reports retained original spacing, or have the retained-format splitter return tokens with a consistent separation contract. Test both incremental emission and flush with batching enabled.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
this is the current behavior and same as python as well. we accept
This reverts commit 0abdbd8.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
1 participant
matches Python, blingfire is much more accurate compared to our basic tokenizer implementation.