In the last year or so I’ve been teaching advanced social psychology and giving workshops on open-science, discussing some of the recent developments in psychological science and my understanding of what some refer to as the “replication/reproducibility crisis”. One of the topics discussed was the importance of sharing and publishing everything, regardless of the outcomes and whether findings were “significant” or not (in the Null Hypothesis Significance Testing sense).
Following that, some asked for my advice about what to do when findings are not significant, not sure how to interpret null findings and communicate those to the academic community in a journal submission. Can’t say that I’m an expert but I’ve been following the academic discussion on the issue for a while now, and so will summarize my limited understanding below.
I would generally say that we need to be cautious when referring to p-values, especially when there are issues of power (smaller sample size than required to confidently detect an effect). Psychological science has generally been moving towards reporting effect-sizes with confidence intervals that provide more information than a dichotomous significant/non-significant and “dancing” p-values that are very sensitive to sample size.
Readings about interpreting null effects:
- Making ‘Null Effects’ Informative: Statistical Techniques and Inferential Frameworks (blog post – New Preprint: Making “Null Effects” Informative; files on the OSF)
- Improving Inferences about Null Effects with Bayes Factors and Equivalence Tests
- Absence of evidence is not evidence of absence: Testing for equivalence
- How to Statistically Show the Absence of an Effect
- TOST equivalence testing R package (TOSTER) and spreadsheet
- Using summary statistics to determine whether a non-significant result supports the absence of an effect
If you’re interested in quantifying the evidence in support of the null, then Bayesian seems like the way to go:
- Download JASP software to make it easier/faster
- see Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP (files on the OSF)
