Core competencies for multimodal analysis
The most significant technical feature of HumanOmni is the realization of synergistic analysis of visual and auditory data. The system contains three 7B-parameter submodels: HumanOmni-Video handles the visual signal, HumanOmni-Audio handles the audio signal, and HumanOmni-Omni is responsible for multimodal fusion.
Specific operational mechanisms include:
- visual processing: Facial micro-expressions (e.g., frowning), macro-motion features (e.g., hand waving) are extracted by convolutional neural networks
- auditory processing: Analyzing speech content and intonation characteristics using the Transformer architecture
- dynamic fusion: Automatically assigns modal weights from 0 to 1 based on scene importance
The test case shows that when inputting a video of a meeting with dialog, the model can accurately correlate the audio feature of "speeding up" with the visual feature of "leaning forward" to conclude that "the speaker is agitated". The model can accurately correlate the audio feature of "faster speech" with the visual feature of "body leaning forward" to conclude that "the speaker is emotional". This cross-modal reasoning capability enables the model to perform well in analyzing complex scenarios.
This answer comes from the articleHumanOmni: a multimodal macromodel for analyzing human video emotions and actionsThe
