What can AI do for me?: evaluating machine learning interpretations in cooperative play
Explore this paper's citation graph
Summary
This work designs a grounded, realistic human-computer cooperative setting using a question answering task, Quizbowl, and proposes an evaluation of interpretation on a real task with real human users, where the effectiveness of interpretation is measured by how much it improves human performance.
- Type
- article
- Published
- 2018-10-23
- Cited by
- 144
- References
- 78
- Access
- Open access
- OpenAlex
- https://openalex.org/W2896487960
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:53039904
Keywords
Interpretability, Vetting, Computer science, Interpretation (philosophy), Task (project management)
References
- Defense Advanced Research Projects Agency (DARPA) Agent Markup Language Computer Aided Knowledge Acquisition
- Smarter Than You Think: How Technology is Changing Our Minds for the Better
- Introduction to Reinforcement Learning
- New Potentials for Data-Driven Intelligent Tutoring System Development and Optimization
- Plans and Situated Actions: The Problem of Human-Machine Communication (Learning in Doing: Social,
- Building a Large Annotated Corpus of English: The Penn Treebank
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
- Besting the Quiz Master: Crowdsourcing Incremental Classification Games
- The Design of Implicit Interactions: Making Interactive Systems Less Obnoxious
- Numeracy, Ratio Bias, and Denominator Neglect in Judgments of Risk and Probability.
- XPLAIN: A System for Creating and Explaining Expert Consulting Programs
- Principles of mixed-initiative user interfaces
- Towards improving trust in context-aware systems by displaying system confidence
- Visualization of uncertainty in context aware mobile applications
- ImageNet: A large-scale hierarchical image database
- Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
- A Neural Network for Factoid Question Answering over Paragraphs
- A survey of software learnability: metrics, methodologies and guidelines
- Human-level control through deep reinforcement learning
Cited by
- On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
- Quizbowl: The Case for Incremental Question Answering
- Open data visualizations and analytics as tools for policy-making
- Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
- NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
- Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation
- Human Decision Making with Machine Assistance
- Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
- Leveraging rationales to improve human task performance
- Evaluating saliency map explanations for convolutional neural networks: a user study
- Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
- No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML
- Aligning Faithful Interpretations with their Social Attribution
- The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
- Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
- A Game-Based Approach for Helping Designers Learn Machine Learning Concepts
- How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
- Gradient-based Analysis of NLP Models is Manipulable
- Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making
Related papers
- Problems in the vetting of senior professional position in a hospital and propositions
- Barring Teachers: The New Vetting Arrangements.
- Work Matters: Management Focus - Vetting & barring - Ready to register
- Counterintelligence Vetting Techniques Compared across Multiple Domains
- Vetting and TMSA: role and requirements in the shipping industry
- POSITIVE VETTING: Protecting the Vulnerable … Positive Vetting for Social Care
- Vetting the Vetting Service: The Case for a Statutory Framework for Police Vetting in New Zealand
- Literature on Personnel Vetting Processes and Procedures: Annotated Selected Bibliography
- An Investigation of Textbook Vetting and Evaluation Process in Tanzania