Computing inter-rater reliability and its variance in the presence of high agreement.
Explore this paper's citation graph
Summary
This paper explores the origin of these limitations, and introduces an alternative and more stable agreement coefficient referred to as the AC1 coefficient, and proposes new variance estimators for the multiple-rater generalized pi and AC1 statistics, whose validity does not depend upon the hypothesis of independence between raters.
- Type
- article
- Published
- 2008-05-01
- Cited by
- 1,910
- References
- 21
- OpenAlex
- https://openalex.org/W1996290308
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:13915043
Keywords
Estimator, Statistics, Variance (accounting), Inter-rater reliability, Reliability (semiconductor)
References
- Measuring nominal scale agreement among many raters.
- Integration and generalization of kappas for multiple raters.
- Large sample standard errors of kappa and weighted kappa.
- Measures of response agreement for qualitative data: Some generalizations and alternatives.
- Categorical Data Analysis
- A Note on the G Index of Agreement
- An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers.
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
- A Coefficient of Agreement for Nominal Scales
- Ramifications of a population model forκ as a coefficient of reliability
- High agreement but low kappa: II. Resolving the paradoxes.
- Beyond kappa: A review of interrater agreement measures
- Reliability of Content Analysis ; The Case of Nominal Scale Cording
- Discrete Multivariate Analysis.
- Approximate Tests of Correlation in Time‐Series
- Sampling techniques.
- Kappa revisited.
Cited by
- Insights from the bar: A model of interaction
- Validity and reliability of the Mastication Observation and Evaluation (MOE) instrument.
- Evaluation of Patient and Family Outpatient Complaints as a Strategy to Prioritize Efforts to Improve Cancer Care Delivery.
- Do concordances of social support and relationship quality predict psychological distress and well‐being of cancer patients and caregivers?
- ESTABLISHING A VALID AND RELIABLE COMPETENCY ASSESSMENT FOR GYMNASTICS COACH CERTIFICATION
- The validity of peak nasal inspiratory flow as a screening tool for nasal obstruction.
- Applying Grounded Theory to Conceptual Data Modeling: Effects of Constant Comparative Method
- Contributions to Bayesian network learning with applications to neuroscience
- In reply: intraobserver and interobserver agreement in visual inspection for xanthochromia: implications for subarachnoid hemorrhage diagnosis, computed tomography validation studies, and the walton rule: methodological mistake.
- The reliability and diagnostic accuracy of the yes/no scapular dyskinesis test when used by graduate assistant athletic trainers
- Previous maltreatment and present mental health in a high-risk adolescent population.
- Statut nutritionnel de la personne âgée vivant à domicile : prévalence, facteurs associés et conséquences
- Evaluating Text Segmentation
- A randomized trial of fellowships for early career researchers finds a high reliability in funding decisions.
- Multi-Citeria Framework for an Emergency Department in an Irish Hospital
- Reliability of the Lisfranc injury radiological classification (Myerson-modified Hardcastle classification system)
- An investigation of the biological variability of the Triple-Joint Flexion Test in adolescent male athletes using 2-D video analysis
- Generalized periodic discharges and 'triphasic waves': A blinded evaluation of inter-rater agreement and clinical significance.
- Effects of teaching games for understanding approach on students' learning outcomes / Malathi Balakrishnan
- Relative meaning frequencies for 578 homonyms in two Spanish dialects: A cross-linguistic extension of the English eDom norms
Related papers
- Interrater reliability: the kappa statistic
- Interrater reliability of EEG-video monitoring
- Evaluating interrater agreement in SPICE-based assessments
- Reliability of a Crohn's Disease Clinical Classification Scheme Based on Disease Behavior
- Interobserver agreement: Cohen's kappa coefficient does not necessarily reflect the percentage of patients with congruent classifications.
- Interrater variation in scoring radiological discrepancies.
- The kappa statistic for establishing interrater reliability in the secondary analysis of qualitative clinical data.
- Reliability assessment of a coding scheme for the physical risk factors of work-related musculoskeletal disorders.