Repository navigation
fix(auto): skip blank lines in wav.scp and jsonl file lists - #3758
Open
Lesereingrape wants to merge 1 commit into
Open
Lesereingrape wants to merge 1 commit into
Lesereingrape wants to merge 1 commit into
Conversation
A blank or whitespace-only line in a file list crashes prepare_data_iterator: the id/data branch indexes lines[0] on an empty split (IndexError), and the .jsonl branch feeds an empty string to json.loads (JSONDecodeError). Skip blank lines so a trailing or separator line no longer aborts the whole batch.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prepare_data_iterator()reads every line of a file list without checking whether it is empty, so a blank or whitespace-only line inwav.scp/text.txtraisesIndexError(indexinglines[0]on an emptysplit()result), and a blank line in a.jsonllist raisesjson.decoder.JSONDecodeErrorfromjson.loads(""). Both come from the same missing guard, so skipping blank lines at the top of the loop fixes the two branches with a single two-line change.Fixes #3757
Reproduction (before):
Type of change
Validation
New
test_scp_filelist_skips_blank_linesandtest_jsonl_filelist_skips_blank_linesintests/test_pcm_input_format.py: 2 failed onmainat66d7a4c2(IndexErrorandJSONDecodeError), and the whole file passes with the fix —python -m pytest -q tests/test_pcm_input_format.py→41 passed.python -m compileall funasr examples tests→ exit 0.tests/test_pcm_input_format.pyis already in the "Validate NumPy compatibility" workflow run list andfunasr/auto/auto_model.pyis in itspathsfilter, so the new tests are exercised by that job onmain.python -m compileall funasr examples testsDocs or links checked
Runtime/deployment command tested
User impact
Anyone running batch inference through a
wav.scp,text.txtorfile.jsonllist — the documented entry point formodel.generate(input=<list>)— can keep hand-edited or concatenated lists with separator lines instead of the whole batch aborting on one empty line.Notes for reviewers