Quick Start - Pose Assets

Lip Sync with Pose Assets gives you more controls by providing you tools to integrate in a Professional workflow.

⚡ Quick Start: Pose Assets

This guide will help you set up your first lip sync animation using Blender's Pose Assets.

Pose Assets mode is ideal for 3D models with facial Rigs.

🎨 What You Need

A Rig with facial controllers
Pose Assets (in Asset Browser)
An audio file (e.g., .wav, .mp3, .mp4, etc.)

You need to define your own Poses.

Sadly, this is a thing Lip Sync cannot automate.

🪄 Step-by-Step Guide

Configure Language

In the Lip Sync panel, select your Language Model

Lip Sync will download your Language Model and store it in cache so you won't have to download it again.

Models are downloaded from https://alphacephei.com/vosk/models. Only small sized models are downloaded.

Add Lip Sync to your Rig

Select your Armature
Click on Add Lip Sync

Set Up Animation type

Ensure that Animation Type is set to Pose Assets

Pose Assets mode is only available if you select an Armature

Rig Settings

Basic Rig - Use when animating only standard bone transforms: Location, Scale, and Rotation (Euler/Quaternion).
Advanced Rig - Use when your poses require animating beyond standard bones, such as direct BBone manipulation or custom properties.

Choose Basic Rig if:

You're only animating standard bone transforms
You're using BBones but controlling them through standard bone controllers

Choose Advanced Rig if:

You're directly animating BBone properties (without controller bones)
You're animating custom properties

Rule of thumb: Try Basic Rig. If it doesn't work or gives weird results; try Advanced Rig.

90% of the time, Basic Rig is what you need.

Advanced Rig may handle more rigs BUT it can be way slower than Basic Rig.

Viseme Settings

This is the most tedious part. You have to manually assign your Poses to Visemes.

If your Pose Assets are not showing, it's probably because you're using an external Poses Library.

You have to load them in your blend file. You can do it automatically with the Refresh Pose Assets button

You can use the same Poses for different Visemes. It can be useful if you only have a few Poses for your character.

sil viseme is the position for "Silence".

UNK viseme is the position for unrecognized phonemes.

Animation Settings

Animation settings enable you to fine tune how Keyframes are inserting.

Settings

Description

Infos

Lip Close Duration

Time required to close your character mouth. It will be used when a silence is detected and insert sil keyframe accordingly

In between

Minimum Threshold between two keyframes.

If a keyframe should be inserted before, it will be ignored.

e.g: 417ms means that next keyframe will be at least at 417ms from the previous one.

Decreasing this value will ensure you to animate all phonemes but may result in jerky motion.

Increasing this value will give a smoother motion but sync will be less precise.

Silence

Time to have between 2 words to be considered as a "Silence".

Too short value will force you character to close their mouth often. Too high value will force your character to have longer mouth transition between two words.

Prioritize Accuracy

Animate all important visemes regardless of timing constraints.

Prevents critical mouth shapes (lip contact sounds like P, B, M, F, TH) from being skipped when they occur in rapid succession.

This feature is only available for Pose Assets Mode.

Use this if your character speaks at high rate.

While this increases accuracy, it may require manual adjustments afterwards.

Timing is very important and need to respect this constraint:

Silence should always be greater than Lip Close Duration and In between

Add Audio

Add a Video Sequencer Editor
Drag & Drop (or Add) your media file

You should only have one enabled channel. Otherwise speech recognition may product unexpected results.

Baking

Press Bake Audio and wait!

For Pose Assets, Keyframes use a Linear Interpolation

Lip Sync works with all Frame Rates (24 fps, 30 fps, 60 fps, etc..)

Blending Animations

Important: Lip Sync creates a new Action with a new Slot.

This means you need to blend the lip animation with your existing Actions and Slots, otherwise you'll only see the lip animation and lose your original character animation.

You can easily do this in the Nonlinear Animation Editor (NLA):

Add a Nonlinear Animation editor
Select your Rig
Select your character's animation (e.g an idle animation)
In NLA, Clic on "Push Down Action"

Now select the Track named "No Action" on the left, and select Lip Sync's Action and Slot on the right sidebar

Now your animation is complete! You can press Play to see it.

PreviousInstallation NextQuick Start - Shape Keys

Last updated 2 months ago