Pre-registered replications

In 2016, following recent developments in psychological science (the so-called “replication crisis”) and gaining my academic independence, I decided to make serious changes to my research agenda to prioritize pre-registered replications and focus on the realm of judgment and decision making. I felt like I needed to revisit the research findings I once took for granted and re-establish the foundations on which I hope to build my research. I, therefore, decided that all my mentoring work with students will involve pre-registered replications, to examine the classics in the field. I chose to focus my efforts on judgment and decision making because I felt that this literature has some of the strongest most consistent effects in social psychology, with fairly simple and transparent research designs.

 

You’re invited to:

  1. Read reports, browse open data and code from our mass pre-registered JDM replications project
  2. Read, use, and/or contribute to our collaborative pre-registered replication projects guide.
  3. Read, use, and/or contribute to our collaborative R/JAMOVI/JASP guide.
  4. Read, use, and/or contribute to our collaborative replications extensions guide
  5. Keep track of the project on Researchgate.
  6. Watch videos about the science replication crisis.
  7. Join the mailing list for events related to this project and open-science.
  8. If you are an Early Career Researcher (advanced PhD student, postdoc, early assistant professor) – Collaborate with us! take the lead in helping us submit the high-quality student reports to journals.

 

This page aims to summarize the findings from my replication attempts so far:

 

Successful replications

  1. Action-effect (Kahneman & Tversky, 1982): Replicated several times (> 8). [multiple projects concluded by Gilad Feldman; sample publication]
  2. Inaction effect (Zeelenberg et al., 2002): Replicated Experiment 1 several times (> 4). [multiple projects concluded by Gilad Feldman; sample preprint]
  3. Omission bias (Spranca, Minsk, & Baron, 1991): Replicated two scenarios from Experiment 1 [project concluded by Tijen Yay; preprint]
  4. Exceptionality effect (Kahneman & Miller, 1986): Replicated two experiments (hitchhiker and car accident scenarios). [project concluded by Lucas Kutscher; publication]
  5. Exceptionality effect (Seta et al., 2001): Replicated 3 times. [concluded by Gilad Feldman; preprint]
  6. Name letter effect (Nuttin, 1987): Replicated the main experiment. [project concluded by Donna Yao; preprint]
  7. Endowment effect & transaction demand (Mandel, 2002): Replicated Experiment 1 [project concluded by Donna Yao; preprint]
  8. Bias blind spot (Pronin et al., 2002): Replicated Experiments 1b and 2 [project concluded by Subramanya Prasad CHANDRASHEKAR; preprint]
  9. Actor-observer bias in free will attributions (Pronin et al., 2010): Replicated twice in US/HK [writeup led by Hallgeir Sjåstad]
  10. Bias blind spot (Pronin & Kugler, 2007): Replicated twice in US/HK, in atleast 2/3 categories [writeup led by Cory Clark]
  11. Preference for indirect harm (Royzman & Baron, 2002): Replicated Experiment 2 twice (US/HK) and Experiment 3 once (HK). [project concluded by Ignazio Ziano; preprint]
  12. Inaction inertia (Tykocinski et al., 1995): Replicated Experiment 1 twice in US/HK samples. [writeup led by Jieying Chen]

Still seeking collaborators:

  1. Escalation of commitment (Arkes & Blumer, 1985): Replicated Experiments 1 and 4 twice (US/HK). [can be comined with escalation of commitment in second semester]
  2. Status quo bias (Samuelson & Zeckhauser, 1988): Replicated 4 scenarios from Experiment 1 (2 of the 4 ran twice). [only pre–reg+data+analyses, no write-ups available, student withdrew due to personal reasons]

 

Semi-successful replications

  1. Exceptionality effect (Miller & McFarland, 1986): Replicated twice using a regret DV, but not using original compensation DV [project concluded by Lucas Kutscher; publication]
  2. Folk intentionality (Malle & Knobe, 1997): Twice (US/HK) found an effect when none was expected (actor-observer asymmetry). [writeup led by Donna Yao]

Still seeking collaborators:

  1. Doing/allowing morality asymmetry (Cushman et al, 2008): Replicated Experiment 1 in US but less so in a small underpowered HK sample.

 

Inconclusive

  1. Force-Intention in moral judgment (Greene et al., 2009): Unsuccessful in replicating Experiment 1b in US MTurk samples and inconclusive in HK sample (medium effect size, sample underpowered to detect an effect). [Joined Psychological Science accelerator to follow up on this in a mass-collaboration]

 

Unsuccessful replications; needs to revisit further

  1. Endowment effect and goal relevance (Irmak, Wakslak, & Trope, 2013): Unsuccessful in replicating the second experiment. [project concluded by Donna Yao; preprint]
  2. Actor-observer bias (Pronin et al., 2007): Unsuccessful in replicating Experiment 1 twice (US/HK) [writeup led by Cory Clark]
  3. Anchoring effect by framing (Wong & Kwong, 2000): Unsuccessful in replicating twice in US and HK. Very likely culture/language bad translation issues. [very messy methodology and findings; postponed]

 

Concluded in Autumn 2018 semester, reports/findings will be made available January 2019

  1. Conjunction effect (Mellers, Hertwig, & Kahneman, 2001): Experiment 1 [writeup led by Subramanya Prasad CHANDRASHEKAR]
  2. First instinct fallacy (Kruger, Wirtz & Miller 2005): Experiment 2 [writeup led by Paul Henne]
  3. Less is better (Hsee, 1998): Studies 1, 2, and 4 [writeup led by Andrew Vonasch]
  4. Disjunction effect (Tversky & Shafir, 1992): Experiment 1 [writeup led by Philippe van de Calseyde]
  5. Effort heuristic (Kruger etal, 2004): Combine Experiments 1-2 [writeup led by Tony Evans]

Still seeking collaborators:

  1. Pluralistic ignorance (Miller, & McFarland, 1987): Experiment 1
  2. Outcome bias (Baron, & Hershey, 1988): Experiment 1
  3. Choosing versus rejecting (Shafir, 1993): All problems in the paper
  4. Fundamental predictor error (Hsee & Weber, 1997): Experiment 1
  5. Insensitivity to sample bias (Hamill, Wilson, & Nisbett, 1980): Study 1
  6. Relevance of irrelevant information (Schwarz, Strack, Hilton, & Naderer, 1991): Experiment 1
  7. Money illusion (Shafir, Diamond, & Tversky, 1997): Problems 1-4
  8. Escalation of commitment (Staw, 1976): Study 1
  9. Hindsight bias (Slovic & Fischhoff, 1977): Experiment 1
  10. Hindsight bias (Fischhoff, 1975): Experiment 2
  11. Regret aversion (Zeelenberg etal 1996): Experiment 1
  12. Irrational reactions to negative outcomes (Epstein, Lipson, Holstein, & Huh 1992): Combining Study 1 and 2
  13. Anchoring-and-adjustment heuristic (Epley & Gilovich 2006): Study 1b [very messy methodology and findings; postponed]

 

[UPDATE Dec 20, 2018]

I presented initial findings at an HKU seminar.

2018-12-20 HKU Mass Pre-registered Replications of JDM – 2nd phase

Download the presentation

WARNING: Preliminary student findings, need to be rechecked and verified.

[End UPDATE Dec 20, 2018]

 

[UPDATE March 5, 2019]

I summarized findings in a poster for ICPS2019 in Paris

2019-03 ICPS 2019 Paris poster

Download the poster

WARNING: Preliminary student findings, need to be rechecked and verified.

[End UPDATE March 5, 2019]

 

In process (2018-9 guided thesis students & collaborations)

  1. Past-future asymmetry (Caruso, Gilbert, & Wilson, 2008): Experiments 1 and 4 [Florence]
  2. Global Self-Evaluation, Desirability and Controllability (Alicke, 1985, JPSP) [Cora]
  3. Anticipated regret and escalation of commitment (Wong & Kwong, 2007) [Rachel]
  4. Counterfactuals, causal attributions, and the hindsight bias: A conceptual integration (Roese & Olson, 1996, JESP) [Roxane]
  5. Disjunction Bias (Hsee, 2004, JPSP): Experiments 2 and 3 [Reanna]
  6. Esptein CEST model [Papara]

 

Planned

  1. Retrievability (Lichtenstein etal 1978)
  2. Unrealistic optimism (Weinstein, 1980) [suggested by Hallgeir Sjåstad]
  3. Knowing with Certainty: Appropriateness of Extreme Confidence (Fischhoff, Slovic, & Lichtenstein, 1977)
  4. Does a Surprising Outcome Reinforce or Reverse the Hindsight Bias? (Ofir & Mazursky, 1997, OBHDP, Exp3) / “I Could Never Have  Expected It to Happen”: Reversal of the Hindsight Bias (Mazursky & Ofir, 1990)
  5. Outcome feedback: Hindsight and information (Hoch & Roediger, 1989, JEP:LMC)
  6. Misuse of useless information (Bastardi & Shafir 1998)
  7. Advice taking in decision making: Egocentric discounting and reputation formation (Yaniv, & Kleinberger, 2004, OBHDP) [suggested by Shoham Choshen-Hillel]
  8. Cheerleader effect – Hierarchical Encoding Makes Individuals in a Group Seem More Attractive (Walker & Vul, 2014, Psychological Science) – 55 citations [suggested by Stefano Livi]
  9. McNeil, B. J., Pauker, S. G., & Tversky, A. (1988). On the framing of medical decisions – cited 120 times, one of the only joint evaluations of framing effects.
  10. Decoy effect (Ariely & Wallsten, 1995, OBHDP) [pre–reg mostly done, no +data+analyses/write-ups available, student withdrew due to personal reasons]
  11. Decoy effect (Connolly, Reb, & Kausel, 2013, JDM) [pre–reg mostly done, , no +data+analyses/write-ups available, student withdrew due to personal reasons]