In 2016, following recent developments in psychological science (the so-called “replication crisis”) and gaining my academic independence, I decided to make serious changes to my research agenda to prioritize pre-registered replications and focus on the realm of judgment and decision making. The aim was to revisit research findings I once took for granted and re-establish the foundations on which I hope to build my research. I, therefore, decided that all my teaching and mentoring work with guided thesis students will involve pre-registered replications or pre-registered meta-analyses, to examine the classics in the field.
In 2017 I guided 3 masters students at Maastricht University to pre-test this realignment. It far exceeded my expectations. We completed 3 pre-registered replication, 3 pre-registered meta-analyses, and one review paper summarizing the insights gained. Once joining HKU, in 2018, I decided to scale up and mass-mobilize HKU’s undergraduate students and lead a mass pre-registered replication effort. In the first year, two semesters, of running this project, we’ve successfully completed 45 replication projects, making this one of the largest replication efforts in social-psychology. For each of the replication projects, we have full pre-registrations, data/code, and all written up in APA style submission ready student reports. In the second semester, most of the replications also include extensions with interesting insights.
I will continue running this in academic year 2019-2020 with 20+ new replications+extensions. If any of this is of interest to you – lots of ways to join in. I am looking for interested early career researchers to join us, see more info on that below.
You’re invited to:
- Read reports, browse open data and code from our mass pre-registered JDM replications project
- Read, use, and/or contribute to our collaborative pre-registered replication projects guide.
- Read, use, and/or contribute to our collaborative R/JAMOVI/JASP guide.
- Read, use, and/or contribute to our collaborative replications extensions guide
- Keep track of the project on Researchgate.
- Watch videos about the science replication crisis.
- Join the mailing list for events related to this project and open-science.
- If you are an Early Career Researcher (advanced PhD student, postdoc, early assistant professor) – Collaborate with us! (jump to section “Collaborations on academic submissions”). Take the lead author on one of our completed projects, and help us submit the high-quality student reports to journals. Choose among the “still seeking collaborators” projects listed below, and email me.
The team currently has collaborators from the following countries: 2 UK, 2 Hong Kong, 2 Netherlands, 1 US, 1 Canada, 1 Norway, 1 France, and 1 New Zealand. See collaborator names next to the projects listed below.
Media mentions of the project or related outputs:
WARNING: Preliminary student summarized findings, need to be rechecked and verified.
I summarized findings in a poster for ICPS2019 in Paris
2017 Maastricht and 2017-8 HKU spring semester
- Action-effect (Kahneman & Tversky, 1982): Replicated several times (> 8). [multiple projects concluded by Gilad Feldman; sample publication]
- Inaction effect (Zeelenberg et al., 2002): Replicated Experiment 1 several times (> 4). [multiple projects concluded by Gilad Feldman; sample preprint]
- Omission bias (Spranca, Minsk, & Baron, 1991): Replicated two scenarios from Experiment 1 [project concluded by Tijen Yay; preprint]
- Exceptionality effect (Kahneman & Miller, 1986): Replicated two experiments (hitchhiker and car accident scenarios). [project concluded by Lucas Kutscher; publication]
- Exceptionality effect (Seta et al., 2001): Replicated 3 times. [concluded by Gilad Feldman; preprint]
- Name letter effect (Nuttin, 1987): Replicated the main experiment. [project concluded by Donna Yao; preprint]
- Endowment effect & transaction demand (Mandel, 2002): Replicated Experiment 1 [project concluded by Donna Yao; preprint]
- Bias blind spot (Pronin et al., 2002): Replicated Experiments 1b and 2 [project concluded by Subramanya Prasad CHANDRASHEKAR; preprint]
- Actor-observer bias in free will attributions (Pronin et al., 2010): Replicated twice in US/HK [writeup led by Hallgeir Sjåstad]
- Bias blind spot (Pronin & Kugler, 2007): Replicated twice in US/HK, in atleast 2/3 categories [writeup led by Cory Clark]
- Preference for indirect harm (Royzman & Baron, 2002): Replicated Experiment 2 twice (US/HK) and Experiment 3 once (HK). [project concluded by Ignazio Ziano; preprint]
- Inaction inertia (Tykocinski et al., 1995): Replicated Experiment 1 twice in US/HK samples. [writeup led by Jieying Chen]
Still seeking collaborators:
- Escalation of commitment (Arkes & Blumer, 1985): Replicated Experiments 1 and 4 twice (US/HK). [can be combined with escalation of commitment that ran in the second semester, and perhaps with the anticipated regret thesis project]
- Status quo bias (Samuelson & Zeckhauser, 1988): Replicated 4 scenarios from Experiment 1 (2 of the 4 ran twice). [only pre–reg+data+analyses, no write-ups available, student withdrew due to personal reasons]
- Exceptionality effect (Miller & McFarland, 1986): Replicated twice using a regret DV, but not using original compensation DV [project concluded by Lucas Kutscher; publication]
- Folk intentionality (Malle & Knobe, 1997): Twice (US/HK) found an effect when none was expected (actor-observer asymmetry). [writeup led by Donna Yao]
- Doing/allowing morality asymmetry (Cushman et al, 2008): Replicated Experiment 1 in US but less so in a small underpowered HK sample. [Writeup led by Mansur Khamitov]
- Force-Intention in moral judgment (Greene et al., 2009): Unsuccessful in replicating Experiment 1b in US MTurk samples and inconclusive in HK sample (medium effect size, sample underpowered to detect an effect). [Joined Psychological Science accelerator to follow up on this in a mass-collaboration] [Writeup led by Mansur Khamitov]
Unsuccessful replications; needs to revisit further
- Endowment effect and goal relevance (Irmak, Wakslak, & Trope, 2013): Unsuccessful in replicating the second experiment. [project concluded by Donna Yao; preprint]
- Actor-observer bias (Pronin et al., 2007): Unsuccessful in replicating Experiment 1 twice (US/HK) [writeup led by Cory Clark]
- Anchoring effect by framing (Wong & Kwong, 2000): Unsuccessful in replicating twice in US and HK. Very likely culture/language bad translation issues. [very messy methodology and findings; postponed]
HKU 2018-9 autumn semester
- Conjunction effect (Mellers, Hertwig, & Kahneman, 2001): Experiment 1 [writeup led by Subramanya Prasad CHANDRASHEKAR]
- First instinct fallacy (Kruger, Wirtz & Miller 2005): Experiment 2 [writeup led by Paul Henne]
- Less is better (Hsee, 1998): Studies 1, 2, and 4 [writeup led by Andrew Vonasch]
- Disjunction effect (Tversky & Shafir, 1992): Experiment 1 [writeup led by Philippe van de Calseyde]
- Effort heuristic (Kruger etal, 2004): Combine Experiments 1-2 [writeup led by Tony Evans]
- Pluralistic ignorance (Miller, & McFarland, 1987): Experiment 1 [writeup led by Paul Hanel]
- Money illusion (Shafir, Diamond, & Tversky, 1997): Problems 1-4 [writeup led by Ignazio Ziano]
- Choosing versus rejecting (Shafir, 1993): All problems in the paper [writeup led by Subramanya Prasad CHANDRASHEKAR]
Still seeking collaborators:
- Outcome bias (Baron, & Hershey, 1988): Experiment 1
- Fundamental predictor error (Hsee & Weber, 1997): Experiment 1
- Insensitivity to sample bias (Hamill, Wilson, & Nisbett, 1980): Study 1
- Relevance of irrelevant information (Schwarz, Strack, Hilton, & Naderer, 1991): Experiment 1
- Escalation of commitment (Staw, 1976): Study 1
- Hindsight bias (Slovic & Fischhoff, 1977): Experiment 1
- Hindsight bias (Fischhoff, 1975): Experiment 2
- Regret aversion (Zeelenberg etal 1996): Experiment 1
- Irrational reactions to negative outcomes (Epstein, Lipson, Holstein, & Huh 1992): Combining Study 1 and 2
- Anchoring-and-adjustment heuristic (Epley & Gilovich 2006): Study 1b [very messy methodology and findings; postponed]
In process (2018-9 guided thesis students & collaborations)
- Past-future asymmetry (Caruso, Gilbert, & Wilson, 2008): Experiments 1 and 4 [Florence]
- Global Self-Evaluation, Desirability and Controllability (Alicke, 1985, JPSP) [Cora]
- Anticipated regret and escalation of commitment (Wong & Kwong, 2007) [Rachel]
- Counterfactuals, causal attributions, and the hindsight bias: A conceptual integration (Roese & Olson, 1996, JESP) [Roxane]
- Disjunction Bias (Hsee, 2004, JPSP): Experiments 2 and 3 [Reanna]
- Esptein CEST model [Papara]
- Retrievability (Lichtenstein etal 1978)
- Unrealistic optimism (Weinstein, 1980) [suggested by Hallgeir Sjåstad]
Knowing with Certainty: Appropriateness of Extreme Confidence (Fischhoff, Slovic, & Lichtenstein, 1977)
- Does a Surprising Outcome Reinforce or Reverse the Hindsight Bias? (Ofir & Mazursky, 1997, OBHDP, Exp3) / “I Could Never Have Expected It to Happen”: Reversal of the Hindsight Bias (Mazursky & Ofir, 1990)
- Outcome feedback: Hindsight and information (Hoch & Roediger, 1989, JEP:LMC)
- Misuse of useless information (Bastardi & Shafir 1998)
- Advice taking in decision making: Egocentric discounting and reputation formation (Yaniv, & Kleinberger, 2004, OBHDP) [suggested by Shoham Choshen-Hillel]
- Cheerleader effect – Hierarchical Encoding Makes Individuals in a Group Seem More Attractive (Walker & Vul, 2014, Psychological Science) – 55 citations [suggested by Stefano Livi]
- McNeil, B. J., Pauker, S. G., & Tversky, A. (1988). On the framing of medical decisions – cited 120 times, one of the only joint evaluations of framing effects.
- Decoy effect (Ariely & Wallsten, 1995, OBHDP) [pre–reg mostly done, no +data+analyses/write-ups available, student withdrew due to personal reasons]
- Decoy effect (Connolly, Reb, & Kausel, 2013, JDM) [pre–reg mostly done, , no +data+analyses/write-ups available, student withdrew due to personal reasons]