Convolutional Self-Attention Networks

Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, Zhaopeng Tu · North American Chapter of the Association for Computational Linguistics · 2019

Explore this paper's citation graph

Summary

Novel convolutional self-attention networks are proposed, which offer SANs the abilities to strengthen dependencies among neighboring elements, and model the interaction between features extracted by multiple attention heads.

Type: preprint
Published: 2019-04-05
Cited by: 136
References: 37
Access: Open access
DOI: https://doi.org/10.18653/v1/n19-1407
arXiv: https://arxiv.org/abs/1904.03107
OpenAlex: https://openalex.org/W2899423466
Semantic Scholar: https://api.semanticscholar.org/CorpusID:102351629

Keywords

Computer science, Computational linguistics, Artificial intelligence, Natural language processing, Volume (thermodynamics)

References

Statistical Significance Tests for Machine Translation EvaluationPhilipp Koehn — 2004
Neural Machine Translation of Rare Words with Subword UnitsRico Sennrich, Barry Haddow, Alexandra Birch — 2015
Convolutional Neural Networks for Sentence ClassificationYoon Kim — 2014
Effective Approaches to Attention-based Neural Machine TranslationThang Luong, Hieu Pham, Christopher D. Manning — 2015
Bleu: a Method for Automatic Evaluation of Machine TranslationKishore Papineni, Salim Roukos, Todd J. Ward — 2002
Multimodal Deep LearningJiquan Ngiam, Aditya Khosla, Mingyu Kim — 2011
A Decomposable Attention Model for Natural Language InferenceAnkur P. Parikh, Oscar Täckström, Dipanjan Das — 2016
Convolutional Sequence to Sequence LearningJonas Gehring, Michael Auli, David Grangier — 2017
THUMT: An Open Source Toolkit for Neural Machine TranslationJiacheng Zhang, Yanzhuo Ding, Shiqi Shen — 2017
Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine TranslationBaosong Yang, Derek F. Wong, Tong Xiao — 2017
NTT Neural Machine Translation Systems at WAT 2017Makoto Morishita, Jun Suzuki, Masaaki Nagata — 2017
Deep Contextualized Word RepresentationsMatthew E. Peters, Mark E Neumann, Mohit Iyyer — 2018
Self-Attention with Relative Position RepresentationsPeter Shaw, Jakob Uszkoreit, Ashish Vaswani — 2018
Self-Attentional Acoustic ModelsMatthias Sperber, Jan Niehues, Graham Neubig — 2018
Group NormalizationYuxin Wu, Kaiming He — 2018
Linguistically-Informed Self-Attention for Semantic Role LabelingEmma Strubell, Patrick Verga, Daniel Andor — 2018
Phrase-level Self-Attention Networks for Universal Sentence EncodingWei Wu, Houfeng Wang, Tianyu Liu — 2018
Exploiting Deep Representations for Neural Machine TranslationZi-Yi Dou, Zhaopeng Tu, Xing Wang — 2018
The Best of Both Worlds: Combining Recent Advances in Neural Machine TranslationMia Xu Chen, Orhan Fırat, Ankur Bapna — 2018
Gaussian Transformer: A Lightweight Approach for Natural Language InferenceMaosheng Guo, Yu Zhang, Ting Liu — 2019

Cited by

Attention Augmented Convolutional NetworksIrwan Bello, Barret Zoph, Quoc V. Le — 2019
Assessing the Ability of Self-Attention Networks to Learn Word OrderBaosong Yang, Longyue Wang, Derek F. Wong — 2019
Leveraging Local and Global Patterns for Self-Attention NetworksMingzhou Xu, Derek F. Wong, Baosong Yang — 2019
Information Aggregation for Multi-Head Attention with Routing-by-AgreementJian Li, Baosong Yang, Zi-Yi Dou — 2019
Context-Aware Self-Attention NetworksBaosong Yang, Jian Li, Derek F. Wong — 2019
Hierarchical Text-Label Integrated Attention Network for Document ClassificationChangjin Gong, Kaize Shi, Zhendong Niu — 2019
Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered NeuronsJie Hao, Xing Wang, Shuming Shi — 2019
Self-Attention with Structural Position RepresentationsXing Wang, Zhaopeng Tu, Longyue Wang — 2019
Multi-Granularity Self-Attention for Neural Machine TranslationJie Hao, Xing Wang, Shuming Shi — 2019
On the use of BERT for Neural Machine TranslationStéphane Clinchant, Kweon Woo Jung, Vassilina Nikoulina — 2019
SesameBERT: Attention for AnywhereTa-Chun Su, Hsiang-Chih Cheng — 2019
Two-Headed Monster and Crossed Co-Attention NetworksYaoyiran Li, Jing Jiang — 2019
Porous Lattice-based Transformer Encoder for Chinese NERMengge Xue, Bowen Yu, Tingwen Liu — 2019
Low-Rank and Locality Constrained Self-Attention for Sequence ModelingQipeng Guo, Xipeng Qiu, Xiangyang Xue — 2019
Multi-Scale Self-Attention for Text ClassificationQipeng Guo, Xipeng Qiu, Pengfei Liu — 2019
Masked Translation ModelArne F. Nix, Yunsu Kim, Jan Rosendahl — 2019
Is Attention All What You Need? - An Empirical Investigation on Convolution-Based Active Memory and Self-AttentionThomas D. Dowdell, Hongyu Zhang — 2019
Generating Diverse Translation by Manipulating Multi-Head AttentionZewei Sun, Shujian Huang, Haoran Wei — 2019
BUT Opensat 2019 Speech Recognition SystemMartin Karafiát, Murali Karthick Baskar, Igor Szöke — 2020
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine TranslationAlessandro Raganato, Yves Scherrer, Jörg Tiedemann — 2020

Convolutional Self-Attention Networks

Summary

Keywords

References

Cited by

Related papers