A three-step solution to lip synchronization
Lip desynchronization is usually caused by audio-video sample rate mismatch or improper model inference parameters. According to the SVLS project documentation, the following solutions are available:
- Enhancing Fluency with DAIN Interpolation: Add the command line
--use_DAIN --time_step 0.5parameter, the system will boost the video from 25fps to 50fps with a deep learning frame-filling algorithm, significantly improving motion continuity - Selecting the right enhancement mode: Choose according to the actual effect
--enhancer lip(enhances lip area only) or--enhancer face(Full Face Enhancement), both modes will improve the clarity of key areas through super resolution technology - Checking the quality of input documents: Ensure that the audio is a WAV file with a sample rate of 16kHz or higher, and that the video is recommended to be at a resolution of 1080p or higher and contains full facial features.
Tests show that when DAIN frame interpolation and lip enhancement mode are turned on at the same time, the lip synchronization accuracy can be improved by about 32%. if it is still unsatisfactory, try adjusting the--time_stepparameter (0.3-0.7 range fine-tuning).
This answer comes from the articleSVLS: SadTalker Enhanced to Generate Digital People Using Portrait VideoThe
May not be reproduced without permission:AI productivity tools " How to solve the problem that the digital human lip-sync generated by SadTalker is not synchronized with the audio?
