Training Tips for the Transformer Model

Keywords

Transformer, Computer science, Machine translation, Training set, Sentence

References

• One weird trick for parallelizing convolutional neural networksAlex Krizhevsky — 2014
• Neural Machine Translation of Rare Words with Subword UnitsRico Sennrich, Barry Haddow, Alexandra Birch — 2015
• Bleu: a Method for Automatic Evaluation of Machine TranslationKishore Papineni, Salim Roukos, Todd J. Ward — 2002
• chrF: character n-gram F-score for automatic MT evaluationMaja Popović — 2015
• The Joy of Parallelism with CzEng 1.0Ondřej Bojar, Zdeněk Žabokrtský, Ondřej Dušek — 2012
• Optimization Methods for Large-Scale Machine LearningLéon Bottou, Frank E. Curtis, Jorge Nocedal — 2016
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaNitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal — 2016
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine TranslationYonghui Wu, Mike Schuster, Zhifeng Chen — 2016
• Fully Character-Level Neural Machine Translation without Explicit SegmentationJason D. Lee, Kyunghyun Cho, Thomas Hofmann — 2016
• Train longer, generalize better: closing the generalization gap in large batch training of neural networksElad Hoffer, Itay Hubara, Daniel Soudry — 2017
• Accurate, Large Minibatch SGD: Training ImageNet in 1 HourPriya Goyal, Piotr Dollár, Ross Girshick — 2017
• Scaling SGD Batch Size to 32K for ImageNet TrainingYang You, Igor Gitman, Boris Ginsburg — 2017
• Findings of the 2017 Conference on Machine Translation (WMT17)Ondřej Bojar, Rajen Chatterjee, Christian Federmann — 2017
• Three Factors Influencing Minima in SGDStanisław Jastrzȩbski, Zachary Kenton, Devansh Arpit — 2017
• Results of the WMT17 Metrics Shared TaskOndřej Bojar, Yvette Graham, Amir Kamran — 2017
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftSergey Ioffe, Christian Szegedy — 2015
• Don't Decay the Learning Rate, Increase the Batch SizeSamuel Smith, Pieter-Jan Kindermans, Chris Ying — 2017
• Overview of the IWSLT 2017 Evaluation CampaignMauro Cettolo, Marcello Federico, Luisa Bentivogli — 2017
• Layer NormalizationJimmy Ba, Jamie Kiros, Geoffrey E. Hinton — 2016
• MizAR 60 for Mizar 50Jakubův, Jan, Chvalovský, Karel, Goertzel, Zarathustra — 2023

Cited by

• Translating Short Segments with NMT: A Case Study in English-to-HindiShantipriya Parida, Ondřej Bojar — 2018
• The University of Cambridge’s Machine Translation Systems for WMT18Felix Stahlberg, Adrià de Gispert, Bill Byrne — 2018
• An Operation Sequence Model for Explainable Neural Machine TranslationFelix Stahlberg, Danielle Saunders, Bill Byrne — 2018
• Trivial Transfer Learning for Low-Resource Neural Machine TranslationTom Kocmi, Ondřej Bojar — 2018
• Neural Machine Translation based Word Transduction Mechanisms for Low-Resource LanguagesSaurav Jha, Akhilesh Sudhakar, Anil Kumar Singh — 2018
• Testsuite on Czech–English Grammatical ContrastsSilvie Cinková, Ondřej Bojar — 2018
• CUNI Submissions in WMT18Tom Kocmi, Roman Sudarikov, Ondřej Bojar — 2018
• EvalD Reference-Less Discourse Evaluation for WMT18Ondřej Bojar, Jiří Mírovský, Kateřina Rysová — 2018
• The RWTH Aachen University Supervised Machine Translation Systems for WMT 2018Julian Schamper, Jan Rosendahl, Parnia Bahar — 2018
• CUNI Transformer Neural MT System for WMT18Martin Popel — 2018
• Findings of the 2018 Conference on Machine Translation (WMT18)Ondřej Bojar, Christian Federmann, Mark Fishel — 2018
• NTT’s Neural Machine Translation Systems for WMT 2018Makoto Morishita, Jun Suzuki, Masaaki Nagata — 2018
• DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence FusionMor Geva, Eric Malmi, Idan Szpektor — 2019
• CVIT-MT Systems for WAT-2018Jerin Philip, Vinay P. Namboodiri, C. V. Jawahar — 2019
• Competence-based Curriculum Learning for Neural Machine TranslationEmmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig — 2019
• Towards End-to-end Speech-to-text Translation with Two-pass DecodingTzu-Wei Sung, Junyou Liu, Hung-yi Lee — 2019
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be PrunedElena Voita, David Talbot, Fédor Moiseev — 2019
• Making Asynchronous Stochastic Gradient Descent Work for TransformersAlham Fikri Aji, Kenneth Heafield — 2019
• Attention over Heads: A Multi-Hop Attention for Neural Machine TranslationShohei Iida, Ryuichiro Kimura, Hongyi Cui — 2019
• Classification of human white blood cells using machine learning for stain-free imaging flow cytometryMaxim Lippeveld, Carly Knill, Emma Ladlow — 2019

Related papers

• Mantras : words of powerSwami Sivananda Radha — 1980
• The Cult of Gāyatrī: A StudyNilachal Mishra — 2021
• Living MantraMani Rao — 2018
• Is this a mantra? mantra ofMarianne Boruch — 2009
• Tantric Appearances and Non-Tantric Meanings: Four Systems of the “Maṇḍala of Mantra” in the Buddhist Cakrasaṃvara LiteratureTsunehiko Sugiki — 2021
• Evoking Steve Jobs’s mantra of “focus and simplicity”.Jack Demick — 2015
• Combining PBSMT and NMT Back-translated Data for Efficient NMTAlberto Poncelas, Maja Popović, Dimitar Shterionov — 2019
• Training Data in Statistical Machine Translation - the More, the Better?Monica Gavrila, Cristina Vertan — 2011