Looks like it uses a Japanese voicepack to generate English lyrics #17

Activity

1. Jandown commented on Mar 6, 2025

   

   Jandown

   on Mar 6, 2025

   More actions

   I have also encountered the same problem. Do you have any other better solutions?
2. lethanner commented on Mar 6, 2025

   

   lethanner

   on Mar 6, 2025

   ContributorAuthor

   More actions

   I FOUND THE REAL REASON why it need to install the Japanese package!!

   Launched step-by-step debug and found out:

   1. It's correctly determines a lyrics language code.
   2. Exception raises while all hardcoded phonemizers beginning initialization.
   3. This is only true for Japanese, as it is the only language that uses the mbrola library (among languages coded specifically in DiffRhythm). All other languages ​​in the initialization keeps using the modules embedded into eSpeak.

   Take a look at the list of languages in the DiffRhythm code:

   lang2backend = {
       "zh": phonemizer_zh, # using "cmn" code
       "ja": phonemizer_ja,
       "en": phonemizer_en, # using "en-US" code
       "fr": phonemizer_fr, # using "fr-fr" code
       "ko": phonemizer_ko,
       "de": phonemizer_de,
   }

   Now look at the return of EspeakWrapper.available_voices() (truncated):

   {
     'cmn': 'sit\\cmn',
     'ja': 'mb\\mb-jp1', # pay attention to this only mb
     'en-us': 'gmw\\en-US',
     'fr-fr': 'roa\\fr',
     'ko': 'ko',
     'de': 'gmw\\de'
   }

   And this code from phonemizer library:

   if 'mb' in voice_code: # <-- did you notice?
     # this is an mbrola voice code. Select the voice by using
     # identifier in the format 'mb/{voice_code}'
     available = {
       voice.identifier[3:]: voice.identifier
       for voice in self.available_voices('mbrola')}
   else:
     # this are espeak voices. Select the voice using it's attached
     # language code. Consider only the first voice of a given code as
     # they are sorted by relevancy
     available = {}
     for voice in self.available_voices():
       if voice.language not in available:
         available[voice.language] = voice.identifier
   # (truncated)
   if self._espeak.set_voice_by_name(voice_name.encode('utf8')) != 0:
     raise RuntimeError(  # pragma: nocover
       f'failed to load voice "{voice_code}"')

   Do you understand what I'm getting at?

   So, this model most likely DOES NOT using Japanese voicepack for English lyrics.
   And the problem of illegible lyrics lies somewhere else.
   (perhaps this topic should even be renamed)

   Will look further.
3. NZqian

   closed this as completedon Mar 11, 2025
4. lethanner commented on Mar 13, 2025

   

   lethanner

   on Mar 13, 2025

   ContributorAuthor

   More actions

   FINALLY ABOUT ILLEGIBLE LYRICS!
   (...why the issue closed so early? :))

   I compared official ASLP-lab inference code with https://github.com/newgenai79/DiffRhythm-windows and my last guess was confirmed.
   This is an encoding issue.

   In newgenai79's repository there are many encoding='utf-8' added to Python's "open" methods. Looks like Python on Linux loads files in UTF-8 by default, but not on Windows.

   After manually appending encoding='utf-8' to opens, I got much more correctly English speaking. Yes, there are still problems with some words, but in any case, the most important thing is that the words now exist at all.
   I'm ready to open a pull request.

   You can also listen proof in the archive:
   output.zip

   Demo lyrics are generated by DeepSeek.

   [00:00.00] Downloaded DiffRhythm, ready to go,
   [00:04.00] But eSpeak alone? It’s a no-show.
   [00:08.00] MBrola’s needed, a library to find,
   [00:12.00] Installed it quick, but peace? Not in my mind.
   [00:16.00] Crashed on launch, searching Japanese,
   [00:20.00] But lyrics are English, why the tease?
   [00:24.00] Installed the pack, still errors galore,
   [00:28.00] "No such file or dir," what’s it looking for?
   [00:32.00] Process Monitor, my debugging friend,
   [00:36.00] Found missing folders, brought them in the end.
   [00:40.00] Copied from Linux, eSpeak and MBrola too,
   [00:44.00] Launched DiffRhythm, but the vocals were stew.
   [00:48.00] Thought it was Japanese, loading wrong,
   [00:52.00] Debugged the code, the truth came along.
   [00:56.00] Hardcoded future, Japanese in the way,
   [01:00.00] Devs removed it, but still no pay.
   [01:04.00] File encoding, the final fight,
   [01:08.00] UTF-8 forced, and it worked just right.
   [01:12.00] Now DiffRhythm sings, clear as can be,
   [01:16.00] AI music on Windows, finally free.

   👀React with 👀1Reacted by jrked