• Editor

  • Mouth animation using Rhubarb Lip Sync

    anglais

Hi! I just released a new version of Rhubarb Lip Sync. Until now, all discussion of Rhubarb has happened in the "Lip Sync" thread over at the Unity forum. As Rhubarb has nothing to do with Unity, I'm creating a new thread for Rhubarb here.

Rhubarb Lip Sync is a free and open-source tool that automatically generates mouth animations from audio files. You can use it with a number of software packages, including Spine. Here are some links:

The latest version 1.8.0 adds support for Ogg Vorbis files. Now Rhubarb supports both .wav and .ogg files.

I'll update this thread whenever there is news regarding Rhubarb for Spine.

Please let me know what you think. I'm always happy to get feedback!

Related Discussions
...

I will follow this 🙂

5 mois plus tard

Rhubarb Lip Sync 1.9 has been released. This version now comes with basic support for languages other than English. So if you're working with dialog in French, Italian, German, Spanish, or any other non-English language, be sure to give it a try!

There are also numerous fixes and improvements. Most notably, processing WAVE files is now considerably faster.

To learn more about the new multi-language support, check out this video:

11 jours plus tard

I'm sorry for the late reaction, but OMG! We Italians are so lucky that our language is generally spoken rather clearly, the demos looked all amazing and the phonetic recognizer is definitely a game changer for us XD
Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!

I've been meaning to test this in Spine for a long time, I need to try! Thanks for the update! 😃

Daniel,

Great Work! from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?

Thanks

Amazing work indeed!

I really liked playing Thimbleweed Park - the voice acting and lip sync there were really spot an, and now I know why! 🙂

Keep up the great work!

18 jours plus tard

Hey,
love the premise of Rhubarb and absolutely loved the demo but I'm struggling to figure it out how to use it, being one of those people who can animate but has no idea how to code... :think:
Do you plan to make some tutorial for Rhubarb?

thanks

4 mois plus tard

Sorry to everybody for not answering! I somehow failed to subscribe to this thread... :wounded:

Thanks for the encouraging words! I'm really glad you like it. 🙂

Erikari a écrit

Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!

Guga is part of a team of adventure gamers doing a full Italian fan-dub of Ron Gilbert's Thimbleweed Park. The game contains about 16,000 lines of dialog, so this is quite a large project, but I believe they're almost done. Check out their thread in the Thimbleweed Park forums (with several demo videos)!

towfiqi a écrit

from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?

Sure, the faster the machine, the faster the processing. Also, if a recording is long enough, Rhubarb Lip Sync will process it with multiple threads simultaneously. Typically, processing speed is not an issue.

anapaulacaruso a écrit

Do you plan to make some tutorial for Rhubarb?

I realize that the documentation is rather meager right now. I'm planning on doing a video tutorial, but it may take some time. In the meantime, this document should get you started. Let me know if you have any questions!

By the way, I just released Rhubarb Lip Sync 1.10. The main change for Spine users is that Rhubarb now supports the beta of Spine 3.8. Previous versions of Rhubarb only worked with Spine 3.7.

Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.

Nate a écrit

Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.

That sounds great! 🙂

9 mois plus tard

@[supprimé]

hey we are going take a look at Rhubarb Lip Sync, but our work flow is a bit different, as we want to animate the spine via unity
any advice ?

tiny problem on the mac version :

For complete usage and help, type `rhubarb

help`

[Fatal] Application terminating with error: Invalid command line.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

public enum LipSync { PBM, KST, EH_AE, AA, AO_ER, UW_OW_W, F_V, L, Idle }
 public LipSync lipSync;

track0.TimeScale = 0f;

if lipSync = PBM frame = 1 if lipSync = PBM frame = 2 etc...

If you want to lip-sync your skeleton through Unity, you should probably use the command-line Rhubarb tool to generate TSV, XML, or JSON files. Then you need to write some code in Unity that parses these files and switches your skeleton's attachments accordingly. You'll have to code that yourself, but it shouldn't be hard.

7 jours plus tard

If possible, please keep support for Spine 3.7 as this is the only version of spine that works for Game Maker at the moment. Also, please don't forget about that spine tutorial on how to get it all to work properly that we talked about. Like what to export, import, and where all the files should be etc. I've been stalking your YouTube waiting for it. LOL. Rhubarb Lip Sync is awesome sauce.

Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.

DanielWolf a écrit

Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.

I'm happy to know that Spine 3.7 will stay supported. As for the tutorial, thank you for starting it, as for finishing it, Your main job and family should always come first.. I've subscribed with notifications for when you upload it and it's available.

I appreciate everything you do, Thanks again for Rhubarb!

20 jours plus tard

It took me a while, but I finished the video tutorial I promised! It gives step-by-step instructions on how to download Rhubarb Lip Sync, setup your skeleton in spine, animate your recordings, and improve the result.

Let me know what you think!

Thanks very much for creating this great in-depth tutorial video, I'm sure it will help a lot of users! Now I even learned something about the history of mouth shapes! :talk: :nerd:

2 mois plus tard

Hi Daniel. Do you have any intention of making a real-time version of Rhubarb... perhaps as a Unity asset?

2 ans plus tard

I take it that by real-time, you mean that a player speaks into a microphone and sees an animated mouth without noticeable delay. This is not something I'm planning to do in the foreseeable future. Right now, I'm focusing on improving the existing feature set for pre-recorded audio.

I just released Rhubarb Lip Sync 1.12. Until now, Rhubarb didn't support skinned skeletons. The new version now fully supports skinning!

Wow, that's definitely good news! Thanks for your great efforts 😃 :yes:

Super cool!

That's great news! 8)

7 jours plus tard

Very cool tool, had fun once I got it set up. Wouldn't work properly with 4.0+ skeletons, possibly related to the curves using doubles instead of integers.

I ended up just exporting the mouth as a 3.7 file and copying the keys over to the 4.1 skeleton.

Upon further testing it seems that a non-default timeline FPS was causing the issue, not the Spine version.

25 jours plus tard

Help, please, when you press the animation button on Rhubarb-Lip-Sync Spine 2D, the rhubarb reaches six percent and no longer responds.

un an plus tard

Help! Just got a file set up and I'm trying to do a test animation for some lipsync. I have all of the 6 basic mouth shapes in the rig, and Rhubarb seems to be detecting them correctly since it checkmarks the 6 slots for the mouthshapes, but when I got to animate the audio I get this error:

I'm using spine version 4.1.23 Pro, if that changes anything. Not sure what's going on, but any help would be appreciated.

Nevermind, I found this " "is not a valid Shape value" error fixed elsewhere. Looks like you have to have a G mouth shape as one of the defaults, not just A thru F. Was able to get past the error by adding a G shape to the rig.

7 mois plus tard

Error processing file (the file path and name).
Error performing speech recognition via PocketSphinx tools.
Error creating speech decoder
Does anyone have any luck decoding this error with Rhubarb ?

6 mois plus tard

I had a chance to use Rhubarb on stream and made a video showing the process here:

Thank you for the tool! It was very simple to use.