Neural Machine Translation of Rare Words with Subword Units

Rico Sennrich, Barry Haddow, Alexandra Birch · Annual Meeting of the Association for Computational Linguistics · 2015

Summary

This paper introduces a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units, and empirically shows that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.3 BLEU.

Type: preprint
Published: 2015-08-31
Cited by: 8,972
References: 40
Access: Open access
DOI: https://doi.org/10.18653/v1/p16-1162
arXiv: https://arxiv.org/abs/1508.07909
OpenAlex: https://openalex.org/W1816313093
Semantic Scholar: https://api.semanticscholar.org/CorpusID:1114678

Keywords

Computer science, Natural language processing, Machine translation, Artificial intelligence, Vocabulary

References

ADADELTA: An Adaptive Learning Rate MethodMatthew D. Zeiler — 2012
Word hy-phen-a-tion by com-put-erFranklin Mark Liang — 1983
Character-Based PSMT for Closely Related LanguagesJörg Tiedemann — 2009
Modelling out-of-vocabulary words for robust speech recognitionIssam Bazzi, James Glass — 2002
Character-Based Pivot Translation for Under-Resourced Languages and DomainsJörg Tiedemann — 2012
Recurrent Continuous Translation ModelsNal Kalchbrenner, Phil Blunsom — 2013
On the difficulty of training recurrent neural networksRazvan Pascanu, Tomáš Mikolov, Yoshua Bengio — 2012
Finding Function in Form: Compositional Character Models for Open Vocabulary Word RepresentationLing Wang, Chris Dyer, Alan W. Black — 2015
Effective Approaches to Attention-based Neural Machine TranslationThang Luong, Hieu Pham, Christopher D. Manning — 2015
Character-Aware Neural Language ModelsYoon Kim, Yacine Jernite, David Sontag — 2015
Improving SMT quality with morpho-syntactic analysisSonja Nießen, Hermann Ney — 2000
Empirical Methods for Compound SplittingPhilipp Koehn, Kevin Knight — 2003
Unsupervised Morphology Rivals Supervised Morphology for Arabic MTDavid Stallard, Jacob Devlin, Michael Kayser — 2012
On Using Very Large Target Vocabulary for Neural Machine TranslationSébastien Jean, Kyunghyun Cho, Roland Memisevic — 2014
Machine Translation without Words through Substring AlignmentGraham Neubig, Taro Watanabe, Shinsuke Mori — 2012
Unsupervised Multilingual Learning for Morphological SegmentationBenjamin Snyder, Regina Barzilay — 2008
Can We Translate Letters?David Vilar, Jan-T. Peter, Hermann Ney — 2007
Unsupervised Discovery of MorphemesMathias Creutz, Krista Lagus — 2002
Addressing the Rare Word Problem in Neural Machine TranslationThang Luong, Ilya Sutskever, Quoc V. Le — 2014
Integrating an Unsupervised Transliteration Model into Statistical Machine TranslationNadir Durrani, Hassan Sajjad, Hieu Hoang — 2014

Cited by

Natural Language Understanding with Distributed RepresentationKyunghyun Cho — 2015
Character-based Neural Machine TranslationLing, Wang, Isabel Trancoso, Chris Dyer — 2015
Mutual Information and Diverse Decoding Improve Neural Machine TranslationJiwei Li, Dan Jurafsky — 2016
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention MechanismOrhan Fırat, Kyunghyun Cho, Yoshua Bengio — 2016
Variable-Length Word Encodings for Neural Translation ModelsRohan Chitnis, John DeNero — 2015
Improving Neural Machine Translation Models with Monolingual DataRico Sennrich, Barry Haddow, Alexandra Birch — 2015
Character-based Neural Machine TranslationMarta R. Costa‐jussà, José A. R. Fonollosa — 2016
A Character-level Decoder without Explicit Segmentation for Neural Machine TranslationJun‐Young Chung, Kyunghyun Cho, Yoshua Bengio — 2016
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character ModelsMinh-Thang Luong, Christopher D. Manning — 2016
Noisy Parallel Approximate Decoding for Conditional Recurrent Language ModelKyunghyun Cho — 2016
Variational Neural Machine TranslationBiao Zhang, Deyi Xiong, Jinsong Su — 2016
The AMU-UEDIN Submission to the WMT16 News Translation Task: Attention-based NMT Models as Feature Functions in Phrase-based SMTMarcin Junczys-Dowmunt, Tomasz Dwojak, Rico Sennrich — 2016
Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-EditingMarcin Junczys-Dowmunt, Roman Grundkiewicz — 2016
Syntactically Guided Neural Machine TranslationFelix Stahlberg, Eva Hasler, Aurelien Waite — 2016
The RWTH Aachen Machine Translation Systems for IWSLT 2017Parnia Bahar, Jan Rosendahl, Nick Rossenbach — 2017
Linguistic Input Features Improve Neural Machine TranslationRico Sennrich, Barry Haddow — 2016
First Result on Arabic Neural Machine TranslationAmjad Almahairi, Kyunghyun Cho, Nizar Habash — 2016
Edinburgh Neural Machine Translation Systems for WMT 16Rico Sennrich, Barry Haddow, Alexandra Birch — 2016
Can neural machine translation do simultaneous translation?Kyunghyun Cho, Masha Esipova — 2016
Word Representation Models for Morphologically Rich Languages in Neural Machine TranslationEkaterina Vylomova, Trevor Cohn, Xuanli He — 2016

Neural Machine Translation of Rare Words with Subword Units

Summary

Keywords

References

Cited by

Related papers