Speech gesture generation from the trimodal context of text, audio, and speaker identity
Explore this paper's citation graph
Summary
This paper presents an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures that are human-like and that match with speech content and rhythm.
- Type
- article
- Published
- 2020-09-04
- Cited by
- 400
- References
- 70
- Access
- Open access
- OpenAlex
- https://openalex.org/W3083173864
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:221507915
Keywords
Gesture, Computer science, Context (archaeology), Speech recognition, Identity (music)
References
- Extensions of Gaussian processes for ranking: semi-supervised and active learning
- Understanding Motion Capture for Computer Animation and Video Games
- Gesture generation by imitation: from human behavior to computer character animation
- Gesture and Thought
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Auto-Encoding Variational Bayes
- Gesture modeling and animation based on a probabilistic re-creation of speaker style
- Effects of personality and social situation on representational gesture production
- The effects of robot-performed co-verbal gesture on listener behaviour
- Gesture and speech in interaction: An overview
- Learning-Based Modeling of Multimodal Behaviors for Humanlike Robots
- Nonverbal Behaviors, Persuasion, and Credibility
- Virtual character performance from speech
- Synchronization of speech and gesture: evidence for interaction in action.
- WordNet: A Lexical Database for English
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
- Hand and Mind: What Gestures Reveal about Thought
- Distributed Representations of Words and Phrases and their Compositionality
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
- Folk Dance Evaluation Using Laban Movement Analysis
Cited by
- Moving Fast and Slow: Analysis of Representations and Post-Processing in Speech-Driven Automatic Gesture Generation
- A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents
- Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents**This work has been supported in part by ARO Grants W911NF1910069 and W911NF1910315, and Intel. Code and additional materials available at: https://gamma.umd.edu/t2g
- Learning Speech-driven 3D Conversational Gestures from Video
- A Large, Crowdsourced Evaluation of Gesture Generation Systems on Common Data: The GENEA Challenge 2020
- To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures
- Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning
- SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents
- Integrated Speech and Gesture Synthesis
- Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates
- Speech2Video: Cross-Modal Distillation for Speech to Video Generation
- Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech
- Human or Robot?: Investigating voice, appearance and gesture motion realism of conversational social agents
- GENEA Workshop 2021: The 2nd Workshop on Generation and Evaluation of Non-verbal Behaviour for Embodied Agents
- A Virtual Character Generation and Animation System for E-Commerce Live Streaming
- Probabilistic Human-like Gesture Synthesis from Speech using GRU-based WGAN
- Speech-Driven Conversational Agents using Conditional Flow-VAEs
- Crossmodal Clustered Contrastive Learning: Grounding of Spoken Language to Gesture
- The GENEA Challenge 2020: Benchmarking gesture-generation systems on common data
- Multimodal Art Pose Recognition and Interaction With Human Intelligence Enhancement
Related papers
- Real‐time hand gestures system based on leap motion
- Real-Time Gesture Recognition for Dynamic Applications
- An Intelligent Smart Home Control Using Body Gestures
- An Intelligent Smart Home Control Using Body Gestures
- Real-time continuous gesture recognition for natural human-computer interaction
- Hand Gesture Detection Using Segmentation
- Proposing Recognition Algorithms For Hand Gestures Based On Machine Learning Model
- Dynamic Hand Gesture Recognition