In 2016, following recent developments in psychological science (the so called “replication crisis”) and gaining my academic independence, I decided to make serious changes to my research agenda to prioritize pre-registered replications and focus on the realm of judgment and decision making. I felt like I needed to revisit the research findings I once took for granted and re-establish the foundations on which I hope to build my research. I therefore decided that all my mentoring work with students will involve pre-registered replications, to examine the classics in the field. I chose to focus my efforts on judgment and decision making, because I felt that this literature has some of the strongest most established effects in the realm of psychology, with fairly simple and transparent research designs.
You’re invited to:
- Read reports, browse open data and code from our mass pre-registered JDM replications project
- Read, use and/or contribute to our pre-registered replication projects guide.
- Keep track of the project on Researchgate.
- Watch videos about the science replication crisis.
- Join the mailing list for events related to this project and open-science.
This page aims to summarize the findings from my replication attempts so far:
- Action effect (Kahneman & Tversky, 1982): Replicated several times (> 8).
- Inaction effect (Zeelenberg et al., 2002): Replicated Experiment 1 several times (> 4).
- Omission bias (Spranca, Minsk, & Baron, 1991): Replicated two scenarios from Experiment 1.
- Exceptionality effect (Kahneman & Miller, 1986): Replicated two experiments (hitchhiker and car accident scenarios).
- Exceptionality effect (Seta et al., 2001): Replicated 3 times.
- Name letter effect (Nuttin, 1987): Replicated the main experiment.
- Endowment effect & transaction demand (Mandel, 2002): Replicated Experiment 1.
- Escalation of commitment (Arkes & Blumer, 1985): Replicated Experiments 1 and 4 twice (US/HK).
- Bias blind spot / Actor observer bias (Pronin et al., 2002): Replicated Experiments 1b and 2. Found an effect for shortcomings when none was expected.
- Actor-observer bias in free will attributions (Pronin et al., 2010): Replicated twice in US/HK.
- Bias blind spot (Pronin & Kugler, 2007): Replicated twice in US/HK, in atleast 2/3 categories.
- Preference for indirect harm (Royzman & Baron, 2002): Replicated Experiment 2 twice (US/HK) and Experiment 3 once (HK).
- Inaction inertia (Tykocinski et al., 1995): Replicated Experiment 1 twice in US/HK samples.
- Status quo bias (Samuelson & Zeckhauser, 1988): Replicated 4 scenarios from Experiment 1 (2 of the 4 ran twice) [Emma]
- Exceptionality effect (Miller & McFarland, 1986): Replicated twice using a regret DV, but not using original compensation DV.
- Doing/allowing morality asymmetry (Cushman et al, 2008): Replicated Experiment 1 in US but not in a small HK sample.
- Folk intentionality (Malle & Knobe, 1997): Twice (US/HK) found an effect when none was expected (actor-observer asymmetry).
- Force-Intention in moral judgment (Greene et al., 2009): Unsuccessful in replicating Experiment 1b in US MTurk samples and inconclusive in HK sample (medium effect size, sample underpowered to detect effect).
Unsuccessful replications, needs to revisit further
- Endowment effect and goal relevance (Irmak, Wakslak, & Trope, 2013): Unsuccessful in replicating the second experiment in paper.
- Actor-observer bias (Pronin et al., 2007): Unsuccessful in replicating Experiment 1 twice (US/HK)
- Anchoring effect by framing (Wong & Kwong, 2000): Unsuccessful in replicating twice in US and HK. Very likely culture/language bad translation issues.
Concluded in Autumn 2018 semester, reports/findings will be made available January 2019
- Pluralistic ignorance (Miller, & McFarland, 1987): Experiment 1
- Outcome bias (Baron, & Hershey, 1988): Experiment 1
- Choosing versus rejecting (Shafir, 1993): All problems in the paper
- Less is better (Hsee, 1998): Studies 1, 2, and 4
- Fundamental predictor error (Hsee & Weber, 1997): Experiment 1
- Disjunction effect (Tversky & Shafir, 1992): Experiment 1
- Insensitivity to sample bias (Hamill, Wilson, & Nisbett, 1980): Study 1
- Relevance of irrelevant information (Schwarz, Strack, Hilton, & Naderer, 1991): Experiment 1
- Conjunction effect (Mellers, Hertwig, & Kahneman, 2001): Experiment 1
- Money illusion (Shafir, Diamond, & Tversky, 1997): Problems 1-4
- Escalation of commitment (Staw, 1976): Study 1
- Effort heuristic (Kruger etal, 2004): Combine Experiments 1-2
- Hindsight bias (Slovic & Fischhoff, 1977): Experiment 1
- Hindsight bias (Fischhoff, 1975): Experiment 2
- Anchoring-and-adjustment heuristic (Epley & Gilovich 2006): Study 1b
- Regret aversion (Zeelenberg etal 1996): Experiment 1
- First instinct fallacy (Kruger, Wirtz & Miller 2005): Experiment 2
- Irrational reactions to negative outcomes (Epstein, Lipson, Holstein, & Huh 1992): Combining Study 1 and 2
[UPDATE Dec 20, 2018]
I presented initial findings at an HKU seminar.
WARNING: Preliminary student findings, need to be rechecked and verified.
[End UPDATE Dec 20, 2018]
In process (2018-9 guided thesis students & collaborations)
- Decoy effect (Ariely & Wallsten, 1995, OBHDP) [Shiyuan]
- Decoy effect (Connolly, Reb, & Kausel, 2013, JDM) [Shiyuan]
- Past-future asymmetry (Caruso, Gilbert, & Wilson, 2008): Experiments 1 and 4 [Florence]
- Global Self-Evaluation, Desirability and Controllability (Alicke, 1985, JPSP) [Cora]
- Anticipated regret and escalation of commitment (Wong & Kwong, 2007) [Rachel]
- Counterfactuals, causal attributions, and the hindsight bias: A conceptual integration (Roese & Olson, 1996, JESP) [Roxane]
- Disjunction Bias (Hsee, 2004, JPSP): Experiments 2 and 3 [Reanna]
- Esptein CEST model [Papara]
- Retrievability (Lichtenstein etal 1978)
- Unrealistic optimism (Weinstein, 1980)
Knowing with Certainty: Appropriateness of Extreme Confidence (Fischhoff, Slovic, & Lichtenstein, 1977)
- Does a Surprising Outcome Reinforce or Reverse the Hindsight Bias? (Ofir & Mazursky, 1997, OBHDP, Exp3) / “I Could Never Have Expected It to Happen”: Reversal of the Hindsight Bias (Mazursky & Ofir, 1990)
- Outcome feedback: Hindsight and information (Hoch & Roediger, 1989, JEP:LMC)
- Misuse of useless information (Bastardi & Shafir 1998)
- Advice taking in decision making: Egocentric discounting and reputation formation (Yaniv, & Kleinberger, 2004, OBHDP)