Pre-registered replications

In 2016, following recent developments in psychological science (the so-called “replication crisis”) and gaining my academic independence, I decided to make serious changes to my research agenda to prioritize pre-registered replications and focus on the realm of judgment and decision making. The aim was to revisit research findings I once took for granted and re-establish the foundations on which I hope to build my research. I, therefore, decided that all my teaching and mentoring work with guided thesis students will involve pre-registered replications or pre-registered meta-analyses, to examine the classics in the field.

In 2017 I guided 3 masters students at Maastricht University to pre-test this realignment. It far exceeded my expectations. We completed 3 pre-registered replication, 3 pre-registered meta-analyses, and one review paper summarizing the insights gained. Once joining HKU, in 2018, I decided to scale up and mass-mobilize HKU’s undergraduate students and lead a mass pre-registered replication effort. In the first year, two semesters, of running this project, we’ve successfully completed 45 replication projects, making this one of the largest replication efforts in social-psychology. For each of the replication projects, we have full pre-registrations, data/code, and all written up in APA style submission ready student reports. In the second semester, most of the replications also include extensions with interesting insights.

I will continue running this in academic year 2019-2020 with 20+ new replications+extensions. If any of this is of interest to you – lots of ways to join in. I am looking for interested early career researchers to join us, see more info on that below.

 

 

You’re invited to:

  1. Read reports, browse open data and code from our mass pre-registered JDM replications project
  2. Read, use, and/or contribute to our collaborative pre-registered replication projects guide.
  3. Read, use, and/or contribute to our collaborative R/JAMOVI/JASP guide.
  4. Read, use, and/or contribute to our collaborative replications extensions guide
  5. Keep track of the project on Researchgate.
  6. Watch videos about the science replication crisis.
  7. Join the mailing list for events related to this project and open-science.
  8. If you are an Early Career Researcher (advanced PhD student, postdoc, early assistant professor) – Collaborate with us! (jump to section  “Collaborations on academic submissions”). Take the lead author on one of our completed projects, and help us submit the high-quality student reports to journals. Choose among the “still seeking collaborators” projects listed below, and email me.
    The team currently has collaborators from the following countries: 2 UK, 2 Hong Kong, 2 Netherlands, 1 US, 1 Canada, 1 Norway, 1 France, and 1 New Zealand. See collaborator names next to the projects listed below.

 

Media mentions of the project or related outputs:

  1. Bias Blind Spot replication

 

Project summary

WARNING: Preliminary student summarized findings, need to be rechecked and verified.

I summarized findings in a poster for ICPS2019 in Paris

2019-03 ICPS 2019 Paris poster

Download the poster

 

 

2017 Maastricht and 2017-8 HKU spring semester

Successful replications

  1. Action-effect (Kahneman & Tversky, 1982): Replicated several times (> 8). [multiple projects concluded by Gilad Feldman; sample publication]
  2. Inaction effect (Zeelenberg et al., 2002): Replicated Experiment 1 several times (> 4). [multiple projects concluded by Gilad Feldman; sample preprint]
  3. Omission bias (Spranca, Minsk, & Baron, 1991): Replicated two scenarios from Experiment 1 [project concluded by Tijen Yay; preprint]
  4. Exceptionality effect (Kahneman & Miller, 1986): Replicated two experiments (hitchhiker and car accident scenarios). [project concluded by Lucas Kutscher; publication]
  5. Exceptionality effect (Seta et al., 2001): Replicated 3 times. [concluded by Gilad Feldman; preprint]
  6. Name letter effect (Nuttin, 1987): Replicated the main experiment. [project concluded by Donna Yao; preprint]
  7. Endowment effect & transaction demand (Mandel, 2002): Replicated Experiment 1 [project concluded by Donna Yao; preprint]
  8. Bias blind spot (Pronin et al., 2002): Replicated Experiments 1b and 2 [project concluded by Subramanya Prasad CHANDRASHEKAR; preprint]
  9. Actor-observer bias in free will attributions (Pronin et al., 2010): Replicated twice in US/HK [writeup led by Hallgeir Sjåstad]
  10. Bias blind spot (Pronin & Kugler, 2007): Replicated twice in US/HK, in atleast 2/3 categories [writeup led by Cory Clark]
  11. Preference for indirect harm (Royzman & Baron, 2002): Replicated Experiment 2 twice (US/HK) and Experiment 3 once (HK). [project concluded by Ignazio Ziano; preprint]
  12. Inaction inertia (Tykocinski et al., 1995): Replicated Experiment 1 twice in US/HK samples. [writeup led by Jieying Chen]
Still seeking collaborators:
  1. Escalation of commitment (Arkes & Blumer, 1985): Replicated Experiments 1 and 4 twice (US/HK). [can be combined with escalation of commitment that ran in the second semester, and perhaps with the anticipated regret thesis project]
  2. Status quo bias (Samuelson & Zeckhauser, 1988): Replicated 4 scenarios from Experiment 1 (2 of the 4 ran twice). [only pre–reg+data+analyses, no write-ups available, student withdrew due to personal reasons]

 

Semi-successful replications

  1. Exceptionality effect (Miller & McFarland, 1986): Replicated twice using a regret DV, but not using original compensation DV [project concluded by Lucas Kutscher; publication]
  2. Folk intentionality (Malle & Knobe, 1997): Twice (US/HK) found an effect when none was expected (actor-observer asymmetry). [writeup led by Donna Yao]
  3. Doing/allowing morality asymmetry (Cushman et al, 2008): Replicated Experiment 1 in US but less so in a small underpowered HK sample. [Writeup led by Mansur Khamitov]

Inconclusive

  1. Force-Intention in moral judgment (Greene et al., 2009): Unsuccessful in replicating Experiment 1b in US MTurk samples and inconclusive in HK sample (medium effect size, sample underpowered to detect an effect). [Joined Psychological Science accelerator to follow up on this in a mass-collaboration] [Writeup led by Mansur Khamitov]

 

Unsuccessful replications; needs to revisit further

  1. Endowment effect and goal relevance (Irmak, Wakslak, & Trope, 2013): Unsuccessful in replicating the second experiment. [project concluded by Donna Yao; preprint]
  2. Actor-observer bias (Pronin et al., 2007): Unsuccessful in replicating Experiment 1 twice (US/HK) [writeup led by Cory Clark]
  3. Anchoring effect by framing (Wong & Kwong, 2000): Unsuccessful in replicating twice in US and HK. Very likely culture/language bad translation issues. [very messy methodology and findings; postponed]

 

HKU 2018-9 autumn semester

In writing

  1. Conjunction effect (Mellers, Hertwig, & Kahneman, 2001): Experiment 1 [writeup led by Subramanya Prasad CHANDRASHEKAR]
  2. First instinct fallacy (Kruger, Wirtz & Miller 2005): Experiment 2 [writeup led by Paul Henne]
  3. Less is better (Hsee, 1998): Studies 1, 2, and 4 [writeup led by Andrew Vonasch]
  4. Disjunction effect (Tversky & Shafir, 1992): Experiment 1 [writeup led by Philippe van de Calseyde]
  5. Effort heuristic (Kruger etal, 2004): Combine Experiments 1-2 [writeup led by Tony Evans]
  6. Pluralistic ignorance (Miller, & McFarland, 1987): Experiment 1 [writeup led by Paul Hanel]
  7. Money illusion (Shafir, Diamond, & Tversky, 1997): Problems 1-4 [writeup led by Ignazio Ziano]
  8. Choosing versus rejecting (Shafir, 1993): All problems in the paper [writeup led by Subramanya Prasad CHANDRASHEKAR]
Still seeking collaborators:
  1. Outcome bias (Baron, & Hershey, 1988): Experiment 1
  2. Fundamental predictor error (Hsee & Weber, 1997): Experiment 1
  3. Insensitivity to sample bias (Hamill, Wilson, & Nisbett, 1980): Study 1
  4. Relevance of irrelevant information (Schwarz, Strack, Hilton, & Naderer, 1991): Experiment 1
  5. Escalation of commitment (Staw, 1976): Study 1
  6. Hindsight bias (Slovic & Fischhoff, 1977): Experiment 1
  7. Hindsight bias (Fischhoff, 1975): Experiment 2
  8. Regret aversion (Zeelenberg etal 1996): Experiment 1
  9. Irrational reactions to negative outcomes (Epstein, Lipson, Holstein, & Huh 1992): Combining Study 1 and 2
  10. Anchoring-and-adjustment heuristic (Epley & Gilovich 2006): Study 1b [very messy methodology and findings; postponed]

 

In process (2018-9 guided thesis students & collaborations)

  1. Past-future asymmetry (Caruso, Gilbert, & Wilson, 2008): Experiments 1 and 4 [Florence]
  2. Global Self-Evaluation, Desirability and Controllability (Alicke, 1985, JPSP) [Cora]
  3. Anticipated regret and escalation of commitment (Wong & Kwong, 2007) [Rachel]
  4. Counterfactuals, causal attributions, and the hindsight bias: A conceptual integration (Roese & Olson, 1996, JESP) [Roxane]
  5. Disjunction Bias (Hsee, 2004, JPSP): Experiments 2 and 3 [Reanna]
  6. Esptein CEST model [Papara]

 

Planned

  1. Retrievability (Lichtenstein etal 1978)
  2. Unrealistic optimism (Weinstein, 1980) [suggested by Hallgeir Sjåstad]
  3. Knowing with Certainty: Appropriateness of Extreme Confidence (Fischhoff, Slovic, & Lichtenstein, 1977)
  4. Does a Surprising Outcome Reinforce or Reverse the Hindsight Bias? (Ofir & Mazursky, 1997, OBHDP, Exp3) / “I Could Never Have  Expected It to Happen”: Reversal of the Hindsight Bias (Mazursky & Ofir, 1990)
  5. Outcome feedback: Hindsight and information (Hoch & Roediger, 1989, JEP:LMC)
  6. Misuse of useless information (Bastardi & Shafir 1998)
  7. Advice taking in decision making: Egocentric discounting and reputation formation (Yaniv, & Kleinberger, 2004, OBHDP) [suggested by Shoham Choshen-Hillel]
  8. Cheerleader effect – Hierarchical Encoding Makes Individuals in a Group Seem More Attractive (Walker & Vul, 2014, Psychological Science) – 55 citations [suggested by Stefano Livi]
  9. McNeil, B. J., Pauker, S. G., & Tversky, A. (1988). On the framing of medical decisions – cited 120 times, one of the only joint evaluations of framing effects.
  10. Decoy effect (Ariely & Wallsten, 1995, OBHDP) [pre–reg mostly done, no +data+analyses/write-ups available, student withdrew due to personal reasons]
  11. Decoy effect (Connolly, Reb, & Kausel, 2013, JDM) [pre–reg mostly done, , no +data+analyses/write-ups available, student withdrew due to personal reasons]