Is Human Scoring the Best Criteria for Summary Evaluation?
Explore this paper's citation graph
Summary
The authors' observations for the BLANC family of measures suggest that the criterion of selecting the best measure not relying on correlations with human scores is universal across very different styles of summaries.
- Type
- preprint
- Published
- 2020-12-29
- Cited by
- 11
- References
- 31
- Access
- Open access
- OpenAlex
- https://openalex.org/W3116832522
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:229924264
Keywords
Measure (data warehouse), Quality (philosophy), Correlation, Computer science, Statistics
References
- Teaching Machines to Read and Comprehend
- Survey Article: Inter-Coder Agreement for Computational Linguistics
- Automatically Evaluating Content Selection in Summarization without Human Models
- ROUGE: A Package for Automatic Evaluation of Summaries
- Question Answering as an Automatic Evaluation Metric for News Article Summarization
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
- SUM-QE: a BERT-based Summary Quality Estimation Model
- Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
- Fine-Tuning Language Models from Human Preferences
- Evaluating the Factual Consistency of Abstractive Text Summarization
- BERTScore: Evaluating Text Generation with BERT
- Fill in the BLANC: Human-free quality estimation of document summaries
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
- SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
- SummEval: Re-evaluating Summarization Evaluation
- Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
- What Have We Achieved on Text Summarization?
- Sensitivity of BLANC to human-scored qualities of text summaries
- Re-evaluating Evaluation in Text Summarization
- GO FIGURE: A Meta Evaluation of Factuality in Summarization
Cited by
- A Survey on Multi-modal Summarization
- Investigating Entropy for Extractive Document Summarization
- Does Summary Evaluation Survive Translation to Other Languages?
- ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
- Neural Embeddings for Text
- ALens: An Adaptive Domain-Oriented Abstract Writing Training Tool for Novice Researchers
- Extractive Negative Opinion Summarization of Consumer Electronics Reviews
- Who Needs External References?—Text Summarization Evaluation Using Original Documents
- Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics
- WIDAR - Weighted Input Document Augmented ROUGE
- Consistency and Coherence from Points of Contextual Similarity
Related papers
- Predicting Summary Quality using Limited Human Input
- How to evaluate rankings of academic entities using test data
- Reliability-Based Feature Weighting for Automated Essay Scoring
- Simplifying the Implementation of Modern Scale Scoring Methods with an Automated R Package: Automated Moderated Nonlinear Factor Analysis (aMNLFA)
- Thinking twice about sum scores
- Assessing Dependence: Some Experimental Results
- Systematic procedure for determining criterion scores.