Fill in the BLANC: Human-free quality estimation of document summaries
Explore this paper's citation graph
Summary
Evidence is presented that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements, and the method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.
- Type
- preprint
- Published
- 2020-02-23
- Cited by
- 148
- References
- 32
- Access
- Open access
- OpenAlex
- https://openalex.org/W3008926737
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:211258800
Keywords
Computer science, Task (project management), Estimation, Measure (data warehouse), Quality (philosophy)
References
- Teaching Machines to Read and Comprehend
- Better Summarization Evaluation with Word Embeddings for ROUGE
- Automatically Assessing Machine Summary Content Without a Gold Standard
- Automatically Evaluating Content Selection in Summarization without Human Models
- LexRank: Graph-based Centrality as Salience in Text Summarization
- ROUGE: A Package for Automatic Evaluation of Summaries
- Keyphrase Based Evaluation of Automatic Text Summarization
- “Cloze Procedure”: A New Tool for Measuring Readability
- Neural Headline Generation with Sentence-wise Optimization
- A Semantic QA-Based Approach for Text Summarization Evaluation
- Source-side Prediction for Neural Headline Generation
- ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
- Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
- A Novel Repetition Normalized Adversarial Reward for Headline Generation
- Headline Generation: Learning from Decomposed Document Titles
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Question Answering as an Automatic Evaluation Metric for News Article Summarization
- HighRES: Highlight-based Reference-less Evaluation of Summarization
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Efficient and Effective Single-Document Summarizations and a Word-Embedding Measurement of Quality
Cited by
- Evaluating Machines by their Real-World Language Use
- SummEval: Re-evaluating Summarization Evaluation
- Sensitivity of BLANC to human-scored qualities of text summaries
- GO FIGURE: A Meta Evaluation of Factuality in Summarization
- Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation
- Is Human Scoring the Best Criteria for Summary Evaluation?
- Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation
- TuringAdvice: A Generative and Dynamic Evaluation of Language Use
- What’s in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization
- Finding a Balanced Degree of Automation for Summary Evaluation
- Perturbation CheckLists for Evaluating NLG Evaluation Metrics
- Does Summary Evaluation Survive Translation to Other Languages?
- ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
- Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code
- QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
- Automatic evaluation of summary on fidelity, conciseness and coherence for text summarization based on semantic link network
- SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
- Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio
- DialSummEval: Revisiting Summarization Evaluation for Dialogues
- Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation