Skip to content

fix(auto): skip blank lines in wav.scp and jsonl file lists - #3758

Open
Lesereingrape wants to merge 1 commit into
modelscope:mainfrom
Lesereingrape:contrib/filelist-blank-line
Open

Lesereingrape wants to merge 1 commit into
modelscope:mainfrom
Lesereingrape:contrib/filelist-blank-line

Conversation

@Lesereingrape

Copy link
Copy Markdown

Summary

prepare_data_iterator() reads every line of a file list without checking whether it is empty, so a blank or whitespace-only line in wav.scp / text.txt raises IndexError (indexing lines[0] on an empty split() result), and a blank line in a .jsonl list raises json.decoder.JSONDecodeError from json.loads(""). Both come from the same missing guard, so skipping blank lines at the top of the loop fixes the two branches with a single two-line change.

Fixes #3757

Reproduction (before):

from funasr.auto.auto_model import prepare_data_iterator
open("wav.scp", "w", encoding="utf-8").write("utt1 /a.wav\n\nutt2 /b.wav\n")
prepare_data_iterator("wav.scp")   # IndexError: list index out of range
# after: (['utt1', 'utt2'], ['/a.wav', '/b.wav'])

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • New test_scp_filelist_skips_blank_lines and test_jsonl_filelist_skips_blank_lines in tests/test_pcm_input_format.py: 2 failed on main at 66d7a4c2 (IndexError and JSONDecodeError), and the whole file passes with the fix — python -m pytest -q tests/test_pcm_input_format.py → 41 passed.

  • python -m compileall funasr examples tests → exit 0.

  • tests/test_pcm_input_format.py is already in the "Validate NumPy compatibility" workflow run list and funasr/auto/auto_model.py is in its paths filter, so the new tests are exercised by that job on main.

  • python -m compileall funasr examples tests

  • Docs or links checked

  • Runtime/deployment command tested

User impact

Anyone running batch inference through a wav.scp, text.txt or file.jsonl list — the documented entry point for model.generate(input=<list>) — can keep hand-edited or concatenated lists with separator lines instead of the whole batch aborting on one empty line.

Notes for reviewers

  • Real entries are still parsed exactly as before; keys and data order are unchanged, and non-matching file extensions are untouched.
  • This is the practical limitation to note: a fork PR in this repository does not trigger workflow runs, so the evidence above is local, and the reproduction needs no model weights or downloads.
  • Prepared with an AI coding agent; a human maintainer at the PR author's side has not re-reviewed the diff beyond the checks listed above.

A blank or whitespace-only line in a file list crashes prepare_data_iterator:
the id/data branch indexes lines[0] on an empty split (IndexError), and the
.jsonl branch feeds an empty string to json.loads (JSONDecodeError). Skip
blank lines so a trailing or separator line no longer aborts the whole batch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: blank lines in wav.scp / jsonl file lists crash prepare_data_iterator (IndexError / JSONDecodeError)

1 participant