Pre-registered replications

In 2016, following recent developments in psychological science (the so called “replication crisis”) and gaining my academic independence, I decided to make serious changes to my research agenda to prioritize pre-registered replications and focus on the realm of judgment and decision making. I felt like I needed to revisit the research findings I once took for granted and re-establish the foundations on which I hope  to build my research. I therefore decided that all my mentoring work with students will involve pre-registered replications, to examine the classics in the field. I chose to focus my efforts on judgment and decision making, because I felt that this literature has some of the strongest most established effects in the realm of psychology, with fairly simple and transparent research designs.

 

You’re invited to:

  1. Read, use and/or contribute to our pre-registered replication projects guide.
  2. Keep track of the project on Researchgate.
  3. Watch videos about the science replication crisis.

 

This page aims to summarize the findings from my replication attempts so far:

 

Successful replications

  1. Action effect (Kahneman & Tversky, 1982): Replicated several times (> 8).
  2. Inaction effect (Zeelenberg et al., 2002): Replicated Experiment 1 several times (> 4).
  3. Omission bias (Spranca, Minsk, & Baron, 1991): Replicated two scenarios from Experiment 1.
  4. Exceptionality effect (Kahneman & Miller, 1986): Replicated two experiments (hitchhiker and car accident scenarios).
  5. Exceptionality effect (Seta et al., 2001): Replicated 3 times.
  6. Name letter effect (Nuttin, 1987): Replicated the main experiment.
  7. Endowment effect & transaction demand (Mandel, 2002): Replicated Experiment 1.
  8. Status quo bias (Samuelson & Zeckhauser, 1988): Replicated 4 scenarios from Experiment 1.
  9. Escalation of commitment (Arkes & Blumer, 1985): Replicated Experiments 1 and 4 twice (US/HK).
  10. Bias blind spot / Actor observer bias (Pronin et al., 2002): Replicated Experiments 1b and 2. Found an effect for shortcomings when none was expected.
  11. Actor-observer bias in free will attributions (Pronin et al., 2010): Replicated twice in US/HK.
  12. Bias blind spot (Pronin & Kugler, 2007): Replicated twice in US/HK, in atleast 2/3 categories.
  13. Preference for indirect harm (Royzman & Baron, 2002): Replicated Experiment 2 twice (US/HK) and Experiment 3 once (HK).
  14. Inaction inertia (Tykocinski et al., 1995): Replicated Experiment 1 twice in US/HK samples.

 

Semi-successful replications

  1. Exceptionality effect (Miller & McFarland, 1986): Replicated twice using a regret DV, but not using original compensation DV.
  2. Doing/allowing morality asymmetry (Cushman et al, 2008): Replicated Experiment 1 in US but not in a small HK sample.
  3. Folk intentionality (Malle & Knobe, 1997): Twice (US/HK) found an effect when none was expected (actor-observer asymmetry).

 

Inconclusive

  1. Force-Intention in moral judgment (Greene et al., 2009): Unsuccessful in replicating Experiment 1b in US MTurk samples and inconclusive in HK sample (medium effect size, sample underpowered to detect effect).

 

Unsuccessful replications, needs to revisit further

  1. Endowment effect and goal relevance (Irmak, Wakslak, & Trope, 2013): Unsuccessful in replicating the second experiment in paper.
  2. Actor-observer bias (Pronin et al., 2007): Unsuccessful in replicating Experiment 1 twice (US/HK)
  3. Anchoring effect by framing (Wong & Kwong, 2000): Unsuccessful in replicating twice in US and HK. Very likely culture/language bad translation issues.

 

In process

  1. Decoy effect: , two experiments: Ariely & Wallsten, 1995; Connolly, Reb, & Kausel, 2013.
  2. Past-future asymmetry (Caruso, Gilbert, & Wilson, 2008): Experiments 1 and 4
  3. Pluralistic ignorance (Miller, & McFarland, 1987): Experiment 1
  4. Outcome bias (Baron, & Hershey, 1988): Experiment 1
  5. Choosing versus rejecting (Shafir, 1993)
  6. Less is better (Hsee, 1998)
  7. Disjunction effect (Tversky & Shafir, 1992)
  8. Insensitivity to sample bias (Hamill, Wilson, & Nisbett, 1980)
  9. Relevance of irrelevant information (Schwarz, Strack, Hilton, & Naderer, 1991)
  10. Conjuction effect (Mellers, Hertwig, & Kahneman, 2001)
  11. Money illusion (Shafir, Diamond, & Tversky, 1997)
  12. Escalation of commitment (Staw, 1976)
  13. Effort heuristic (Kruger etal, 2004)
  14. Hindsight bias (Slovic & Fischhoff, 1977)
  15. Hindsight bias (Fischhoff, 1975)
  16. Retrievability (Lichtenstein etal 1978)
  17. Misuse of useless information (Bastardi & Shafir 1998)
  18. Anchoring-and-adjustment heuristic (Epley & Gilovich 2006)
  19. Regret aversion (Zeelenberg etal 1996)
  20. First instinct fallacy (Kruger, Wirtz & Miller 2005)
  21. Irrational reactions to negative outcomes (Epstein, Lipson, Holstein, & Huh 1992)