Intelligent action generation system with multimodal inputs
ChatAnyone realizes end-to-end mapping of audio signals to body movements, breaking the paradigm of traditional keyframe animation. Its technological breakthrough contains three levels: 1) the speech rhythm analysis module extracts acoustic features such as fundamental frequency and energy; 2) the semantic understanding module recognizes utterance stress and emotional tendency; and 3) the gesture generator transforms these features into gesture parameters that conform to social etiquette. The test data shows that the accuracy of matching the system-generated gestures with the focus of the utterance reaches 801 TP3T, which is better than the industry average of 651 TP3T.
Compared with the traditional program that needs to manually design animation curves, this system can automatically generate movements such as nodding (affirmative utterance), spreading hands (questioning tone), and so on, which are in line with human communication habits. Especially in long audio processing, the system ensures that the rhythm of the action changes through the attention mechanism to avoid mechanical repetition. Although the current version has not yet opened the real-time interaction function, its pre-processing generation mode can already meet the needs of the production of recorded content.
This answer comes from the articleChatAnyone: a tool for generating half-body digital human portrait videos from photosThe
