Finnish Language Modeling with Deep Transformer Models

Keywords

Perplexity, Language model, Transformer, Computer science, Artificial intelligence

References

• Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading BooksYukun Zhu, Ryan Kiros, Rich Zemel — 2015
• Neural Machine Translation of Rare Words with Subword UnitsRico Sennrich, Barry Haddow, Alexandra Birch — 2015
• Unsupervised models for morpheme segmentation and morphology learningMathias Creutz, Krista Lagus — 2007
• Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber — 1997
• Morfessor 2.0: Toolkit for statistical morphological segmentationPeter Smit, Sámi Virpioja, Stig-Arne Grönroos — 2014
• SGDR: Stochastic Gradient Descent with Warm RestartsIlya Loshchilov, Frank Hutter — 2016
• Investigating Bidirectional Recurrent Neural Network Language Models for Speech RecognitionXinyi Chen, Anton Ragni, Xubo Liu — 2017
• Fine-tuned Language Models for Text ClassificationJeremy Howard, Sebastian Ruder — 2018
• Deep Contextualized Word RepresentationsMatthew E. Peters, Mark E Neumann, Mohit Iyyer — 2018
• Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextDai, Zihang, Yang, Zhilin, Yang, Yiming — 2019
• Modern subword-based models for automatic speech recognitionPeter Smit — 2019
• BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin, Ming‐Wei Chang, Kenton Lee — 2019
• RoBERTa: A Robustly Optimized BERT Pretraining ApproachYinhan Liu, Myle Ott, Goyal, Naman — 2019
• A new algorithm for data compressionGagePhilip — 1994
• Advances in subword-based HMM-DNN speech recognition across languagesPeter Smit, Sámi Virpioja, Mikko Kurimo — 2021
• Attention is All you NeedAshish Vaswani, Noam Shazeer, Niki Parmar — 2017
• A new algorithm for data compressionPhilip Gage — 1994
• Adam: A Method for Stochastic OptimizationDiederik P. Kingma, Jimmy Ba — 2014
• XLNet: Generalized Autoregressive Pretraining for Language UnderstandingZhilin Yang, Dai, Zihang, Yang, Yiming — 2019
• SGDR: Stochastic Gradient Descent with RestartsI. Loshchilov, Frank Hutter — 2016

Cited by

• Speech Recognition for Conversational FinnishAnssi Moisio — 2021

Related papers

• Transformer-IC: The Solution to Information LossZhigang Song, Jiazhao Chai, Wenqian Shang — 2021
• Learning Deep Transformer Models for Machine TranslationQiang Wang, Bei Li, Tong Xiao — 2019
• Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language UnderstandingDhanasekar Sundararaman, Vivek Subramanian, Guoyin Wang — 2019
• Re-Transformer: A Self-Attention Based Model for Machine TranslationHuey-Ing Liu, Weilin Chen — 2021
• Effective Architectures for Low Resource Multilingual Named Entity TransliterationMolly A. Moran, Constantine Lignos — 2020
• Transformers for Low-Resource Languages: Is Féidir Linn!Séamus Lankford, Afli, Haithem, Andy Way — 2024