A Call for Clarity in Reporting BLEU Scores
Explore this paper's citation graph
Summary
Pointing to the success of the parsing community, it is suggested machine translation researchers settle upon the BLEU scheme, which does not allow for user-supplied reference processing, and provide a new tool, SACREBLEU, to facilitate this.
- Type
- preprint
- Published
- 2018-04-23
- Cited by
- 3,607
- References
- 22
- Access
- Open access
- OpenAlex
- https://openalex.org/W2798362442
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:13751870
Keywords
BLEU, Computer science, Machine translation, Lexical analysis, Natural language processing
References
- Re-evaluating the Role of Bleu in Machine Translation Research
- Building a Large Annotated Corpus of English: The Penn Treebank
- Neural Machine Translation of Rare Words with Subword Units
- Effective Approaches to Attention-based Neural Machine Translation
- Treebank Annotation Schemes and Parser Evaluation for German
- On Using Very Large Target Vocabulary for Neural Machine Translation
- Bleu: a Method for Automatic Evaluation of Machine Translation
- Addressing the Rare Word Problem in Neural Machine Translation
- Moses: Open Source Toolkit for Statistical Machine Translation
- Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
- A Hierarchical Phrase-Based Model for Statistical Machine Translation
- Is Machine Translation Getting Better over Time?
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- A Convolutional Encoder Model for Neural Machine Translation
- Nematus: a Toolkit for Neural Machine Translation
- A procedure for quantitatively comparing the syntactic coverage of English
- Findings of the 2017 Conference on Machine Translation (WMT17)
- A Structured Review of the Validity of BLEU
- Overview of the IWSLT 2017 Evaluation Campaign
- Attention is All you Need
Cited by
- Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation
- Zero-Shot Dual Machine Translation
- OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models
- Scaling Neural Machine Translation
- Explaining and Generalizing Back-Translation through Wake-Sleep
- Auto-Encoding Variational Neural Machine Translation
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
- Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
- Revisiting Character-Based Neural Machine Translation with Capacity and Compression
- MTNT: A Testbed for Machine Translation of Noisy Text
- Simple Fusion: Return of the Language Model
- Learning to Segment Inputs for NMT Favors Character-Level Processing
- Multilingual NMT with a Language-Independent Attention Bridge
- You May Not Need Attention
- Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
- Tilde’s Machine Translation Systems for WMT 2017
- CUNI Submissions in WMT18
- Alibaba’s Neural Machine Translation Systems for WMT18
- CUNI Transformer Neural MT System for WMT18
- An Analysis of Encoder Representations in Transformer-Based Machine Translation
Related papers
- Bleu: a Method for Automatic Evaluation of Machine Translation
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation