Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates

Keywords

Gesture, Computer science, Synchronization (alternating current), Speech recognition, Template

References

• Gesture and speech in interaction: An overviewPetra Wagner, Zofia Malisz, Stefan Kopp — 2014
• Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agentsJustine Cassell, Catherine Pélachaud, Norman I. Badler — 1994
• Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber — 1997
• Gesture controllersSergey Levine, Philipp Krähenbühl, Sebastian Thrun — 2010
• Face2Face: Real-Time Face Capture and Reenactment of RGB VideosJustus Thies, Michael Zollhöfer, Marc Stamminger — 2016
• Towards a Common Framework for Multimodal Generation: The Behavior Markup LanguageStefan Kopp, Brigitte Krenn, S. Marsella — 2006
• Audio-driven facial animation by joint end-to-end learning of pose and emotionTero Karras, Timo Aila, Samuli Laine — 2017
• Audio to Body DynamicsEli Shlizerman, Lucio Dery, Hayden Schoen — 2017
• Synthesizing Images of Humans in Unseen PosesGuha Balakrishnan, Amy Zhao, Adrian V. Dalca — 2018
• Everybody Dance NowC. Chan, Shiry Ginosar, Tinghui Zhou — 2018
• Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid RobotsYoungwoo Yoon, Woo-Ri Ko, Minsu Jang — 2018
• Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM NetworkDai Hasegawa, Naoshi Kaneko, Shinichi Shirakawa — 2018
• Analyzing Input and Output Representations for Speech-Driven Gesture GenerationTaras Kucherenko, Dai Hasegawa, Gustav Eje Henter — 2019
• Few-Shot Adversarial Learning of Realistic Neural Talking Head ModelsEgor Zakharov, Aliaksandra Shysheya, Egor Burkov — 2019
• Learning Individual Styles of Conversational GestureShiry Ginosar, Amir Bar, Gefen Kohavi — 2019
• Realistic Speech-Driven Facial Animation with GANsKonstantinos Vougioukas, Stavros Petridis, Maja Pantić — 2019
• OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsZhe Cao, Gines Hidalgo, Tomas Simon — 2018
• You said that?Joon Son Son, Amir Jamaludin, Andrew Zisserman — 2017
• A Morphable Model For The Synthesis Of 3D FacesMatthew A. Turk — 1999
• Multi-objective adversarial gesture generationYlva Ferstl, Michael Neff, Rachel McDonnell — 2019

Cited by

• Freeform Body Motion Generation from SpeechJing Xu, Wei Zhang, Yalong Bai — 2022
• Rhythmic GesticulatorTenglong Ao, Qingzhe Gao, Yuke Lou — 2022
• DisCo: Disentangled Implicit Content and Rhythm Learning for Diverse Co-Speech Gestures SynthesisHaiyang Liu, Naoya Iwamoto, Zihao Zhu — 2022
• The DeepMotion entry to the GENEA Challenge 2022Shuhong Lu, Andrew Feng — 2022
• TransGesture: Autoregressive Gesture Generation with RNN-TransducerNaoshi Kaneko, Yuna Mitsubayashi, Geng Mu — 2022
• Audio-Driven Co-Speech Gesture Video GenerationXian Liu, Qianyi Wu, Hang Zhou — 2022
• Learning Hierarchical Cross-Modal Association for Co-Speech Gesture GenerationXian Liu, Qianyi Wu, Hang Zhou — 2022
• SEEG: Semantic Energized Co-speech Gesture GenerationYuanzhi Liang, Qianyu Feng, Linchao Zhu — 2022
• Sound Generation Method with Timing-Aligned Visual Feature MappingZhifeng Xie, Luoyi Sun, Yuzhou Sun — 2022
• StyleTalk: One-shot Talking Head Generation with Controllable Speaking StylesYifeng Ma, Suzhen Wang, Zhipeng Hu — 2023
• GesGPT: Speech Gesture Synthesis With Text Parsing From ChatGPTNan Gao, Zeyu Zhao, Zhi Zeng — 2023
• A Comprehensive Review of Data‐Driven Co‐Speech Gesture GenerationSimbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja — 2023
• MPE4G : Multimodal Pretrained Encoder for Co-Speech Gesture GenerationGwantae Kim, Seonghyeok Noh, Insung Ham — 2023
• Bodyformer: Semantics-guided 3D Body Gesture Synthesis with TransformerKunkun Pang, Dafei Qin, Yingruo Fan — 2023
• A Survey on Deep Multi-modal Learning for Body Language Recognition and GenerationLi Liu, Lufei Gao, Wentao Lei — 2023
• QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture GenerationSicheng Yang, Zhiyong Wu, Minglei Li — 2023
• Generating Holistic 3D Human Motion from SpeechHongwei Yi, Hualin Liang, Yifei Liu — 2022
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture GenerationLingting Zhu, Xian Liu, Xuanyu Liu — 2023
• Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pretrained RewardsMingyang Sun, Mengchen Zhao, Yaqing Hou — 2023
• ACT2GHitoshi Teshima, Naoki Wake, Diego Thomas — 2023

Related papers

• A Sync Phenomenon in Power SystemsKwan-Shik Shim, Hae-Kon Nam — 2006
• QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage ServicesYong Cui, Zeqi Lai, Xin Wang — 2015
• QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage ServicesYong Cui, Zeqi Lai, Xin Wang — 2017
• Towards successful user interaction with systems: focusing on user-derived gestures for smart home systems.Eunjung Choi, Sunghyuk Kwon, Donghun Lee — 2014
• 전력계통에서 동조 현상沈冠埴, 南海鯤 — 2006
• Spontaneous use of gesture sequences in orangutansSebastian Tempelmann, Katja Liebal — 2012
• Optimizing background email sync on smartphonesFengyuan Xu, Yunxin Liu, Thomas Moscibroda — 2013
• Eliciting Mid-Air Gestures for Wall-Display InteractionMarkus L. Wittorf, Mikkel R. Jakobsen — 2016
• Coarse-grained cloud synchronization mechanism design may lead to severe traffic overuseZhenhua Li, Zhi-Li Zhang, Yafei Dai — 2013