Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Keywords

Replicate, Amazon rainforest, Sample (material), Population, Computer science

References

• The Wages of Pay Cuts: Evidence from a Field ExperimentDaniel L. Chen, John J. Horton — 2009
• Students as Experimental Participants: A Defense of the "Narrow Data Base" (WP-09-05)James Druckman, Cindy D. Kam — 2009
• Get another label? improving data quality and data mining using multiple, noisy labelersVictor S. Sheng, Foster Provost, Panagiotis G. Ipeirotis — 2008
• On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.Martin T. Orne — 1962
• Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language TasksRion Snow, Brendan O’Connor, Daniel Jurafsky — 2008
• Influence of Elaboration on the Framing of DecisionKazuhisa Takemura — 1994
• An information processing view of framing effects: The role of causal schemas in decision makingJerwen Jou, James Shanteau, Richard J. Harris — 1996
• Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires.Samuel D. Gosling, Simine Vazire, Sanjay Srivastava — 2004
• Framing the framing effect: the impact of context cues on solutions to the ‘Asian disease’ problemHerbert Bless, Tilmann Betsch, Axel Franzén — 1998
• Evaluating framing effectsJames Druckman — 2001
• Predicting political elections from rapid and unreflective face judgmentsCharles C. Ballew, Alexander Todorov — 2007
• Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental ResearchCindy D. Kam, Jennifer Wilking, Elizabeth J. Zechmeister — 2007
• Making Sense of Issues Through Media Frames: Understanding the Kosovo CrisisAdam J. Berinsky, Donald R. Kinder — 2006
• Looking Like a Winner: Candidate Appearance and Electoral Success in New DemocraciesChappell Lawson, Gabriel Lenz, Andy Baker — 2010
• The framing of decisions and the psychology of choice.Amos Tversky, Daniel Kahneman — 1981
• The labor economics of paid crowdsourcingJohn J. Horton, Lydia B. Chilton — 2010
• Running experiments on Amazon Mechanical TurkGabriele Paolacci, Jesse Chandler, Panagiotis G. Ipeirotis — 2010
• Who are the crowdworkers?: shifting demographics in mechanical turkJoel Ross, Lilly Irani, M. Six Silberman — 2010
• Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression TreesDonald P. Green, Holger Kern — 2012
• The Framing of Decisions: A New Look at Old ProblemsAnton Kühberger — 1995

Cited by

• Humans Are Not Machines: The Behavioral Impact of Queueing Design on Service TimeMasha Shunko, Julie Niederhoff, Yaroslav Rosokha — 2017
• I Respectfully Dissent: Linking Judicial Voting Behavior, Media Coverage, and Public Responses in the Study of U.S. Supreme Court Decisions.Michael A. Zilis — 2013
• Industry, Self-Interest, and Individual Preferences over Trade PolicyMichael Tomz — 2015
• Social Influences on Online Political Information Search and EvaluationDouglas R. Pierce, David P. Redlawsk, William W. Cohen — 2017
• Experimental analysis of the effect of standards on compliance and performanceConstantine Boussalis, Yuval Feldman, Henry E. Smith — 2018
• Numeracy and the Persuasive Effect of Policy Information and Party Cues.弘嗣 上島 — 2016
• The mediating role of child self-regulation of eating in the relationship between parental use of food as a reward and child emotional overeating.廉太郎 滝, 耕筰 山田, 耕輔 小松 — 2017
• Using food as a reward: An examination of parental reward practices.Mairie — 2018
• Knowledge of congenital cytomegalovirus (cCMV) among physical and occupational therapists in the United StatesGastón Ferrer — 2017
• The Effects of Perceived Neighborhood Immigrant Population Size on Preferences for Redistribution in New York City: A Pilot StudyAlessandra Carenzio — 2019
• Breaking Boundaries| When Parody and Reality Collide: Examining the Effects of Colbert’s Super PAC Satire on Issue Knowledge and Policy Engagement across Media FormatsHeather LaMarre — 2013
• What Theories of Political Participation Can Teach Us about the Blogosphere, and Vice Versa.W. Abraham Gong — 2013
• Prospects for Online Crowdsourcing of Social Science Research Tasks: A Case Study Using Amazon Mechanical TurkCatherine Schmitt-Sands, Richard Smith — 2014
• Harnessing complexity: Analysis methodology and ethical framework to facilitate utilization of video data in evaluationsKurt A. Wilson — 2014
• Feedback and Performance in Crowd Work: a Real Effort ExperimentTim Straub, Henner Gimpel, Florian Teschner — 2014
• More than words: Experimental evidence on the negative effects of red tape on quality and procedural justiceWesley Kaufmann, Lars Tummers — 2014
• The Polarizing Effect of Incivility in the Political Blog CommentsphereElizabeth Suhay — 2013
• The Evaluation of a Workbook Designed to Promote Self-ForgivenessC. Bell — 2014
• Examining young consumers' adoption intention of SMS-based and APP-based mobile coupon services: a perceived value perspectiveMinJung Park — 2014
• LIBERTY, EQUALITY, AND KINDNESS?: HOW THE STRATEGIC CHOICE OF VALUES AFFECT POLICY ATTITUDES AND INFORMATION CHOICESPatricia S. Hallam Joseph — 2013

Related papers

• Estimation of a Finite Population Mean and Total Using Population Ranks of Sample UnitsÖmer Öztürk — 2016
• The internal validity obsessionGregory Mitchell, Philip E. Tetlock — 2022
• Estimation of Population Parameters Using the Technique of Sub-sampling from Non-respondents in Sample Surveys: A ReviewB. B. Khare — 2014
• Making Health Research Matter: A Call to Increase Attention to External Validity.Amy G. Huebschmann, Ian M. Leavitt, Russell E. Glasgow — 2019