Reviewing

The following is my reviewer philosophy. It’s work in progress

 

Starting mid-2021Strategic choice to try and move towards investing my reviewer time on reviewing for Diamond Open Access and community journals, with an emphasis on Registered Reports. 

I will most likely decline requests from journals from the big publishers, unless my team has recently published in those (not my choice, I leave it to lead authors), and the same editor is specifically requesting my help. In such cases, I will likely use this opportunity to promote (/insist on) open-science practices and replications in that journal.

 

My general approach to reviews:

  • I sign my reviews. I believe in reviewer transparency.
  • I try to be a careful reviewer and I invest time in my reviews. I will do my best to make sure I understand the manuscript before I write feedback.
  • I offer positive constructive feedback. I prefer seeing the potential in papers and data and I offer constructive ways to improve. For empirical papers – If methods and data are sound, I rarely see a reason for rejection. “Worst” case scenario: I ask for additional data collection addressing the drawbacks with a recommendation for in-principle acceptance as a Registered Report.
  • I believe in open science, transparency, and reproducibility. I will request that you report everything and share your procedures, materials, entire data, and documented code. I will also ask that you detail your decisions for exclusions, sample size (power), and selective analyses. This will also help me provide you with constructive suggestions and possibly even additional code to further improve your manuscript.
  • I am accepting of imperfect results and null findings. I see those as part of the process and as important as ‘significant’ findings. What I will be looking for is humility and that claims match the data and findings.
  • I support the Peer Reviewers’ Openness Initiative. I will only review submissions with shared data and code. If authors have a good justified reason for not sharing their data and code, that’s okay, I would just rather invest my time in those that can share theirs.
  • I liked most of what’s recommended in “Peer-Review Guidelines Promoting Replicability and Transparency in Psychological Science” (AMPPS, 2018)
  • I try and follow reviewer guidelines such as “Empowering peer reviewers with a checklist to improve transparency” (more here)

 

I have a very strong preference for and am far more (/only) likely to accept invitations for:

  • Diamond Open Access and community journals (e.g., PCI-RR).
  • Journals that follow stricter TOP guidelines.
  • Submissions that are Registered Reports. Empirical science should move towards Registered Reports.
  • Submissions posted on preprint servers. Which would mean that we can publicly share our reviews (e.g., PublishYourReviews. hypothes.is)
  • Journals/platforms that allow 1) identifiable, 2) public, 3) citable, peer reviews. Peer review should be open.

 

Resources:

I created a peer-review templates, that use in my courses and guided theses by students who work with me and review one another –

  1. Collaborative review/opinion/guide peer review template
  2. Collaborative quantitative manuscript peer review template

 

Example for what I tend to reuse in reviews:

I sign my name as I support reviewer transparency: Gilad Feldman
Thank you for inviting me to review this manuscript. [I’m happy to take part in a Registered Report and ‎especially in the wonderful PCI-RR initiative. ‎]
Editor/authors are welcome to use my review as they please, post/share it etc. I waive any copyright that I ‎might hold to these.‎

Disclosures related to this review and my background, I believe it’s important to state clearly what my ‎expertise is so that editor can take my comments in perspective and authors can decide what they’d like to ‎take away from what I write: ‎

  1. I consider myself an early career researcher, with limited experience with [XXXXXXX]. I am trying to do a bit ‎more of these projects now, and I recognize the complexity, yet it’s possible I’m overlooking some ‎factors that come with experience.‎
  2. My reviewer “code” is posted on: https://mgto.org/reviewing/ . Many of my comments will reflect ‎my support for open-science and the need for reviewers to urge authors to do well on open-‎science practices.‎
  3. My comments are NOT meant as criteria for rejection, but rather as assistance in helping authors ‎do better. I aim to help and support you, not hinder authors. I offer my review as a guide for ‎improvement, and I am very happy to work with the authors to help them in this journey. They are ‎welcome to reach out to us and ask further questions, and I’ll do my best to help ‎‎([email protected]). I leave it up to the editor to decide on which things s/he thinks is a must for ‎you to implement. I typically aim to see potential in any manuscript.
  4. Important note: Please do NOT consider any of the references to students’ or my work in this review as a request or pressure to cite me or the student’s work. I am all too aware of what might be perceived as reviewers pressure to include citations to reviewers work. This is NOT the case here. Those are provided as FYI for something I thought could be relevant from our own work.

[In most cases: I’d like to start out by saying that I think this line of research is valuable, I’d like to see these findings published.]

[…]

Requests and comments regarding all studies (these are now standard requests I make following the science reform movement and the need for more transparency and openness in psychological science, see more information here – https://osf.io/hadz3/; the editor can decide which of these are not important for the editor/journal, but I strongly recommend supporting as many of these as possible):

  1. Please include power analyses and indicate how sample size was determined.
  2. Please include a statement on whether data collection was completed prior to looking at the data.
  3. Please include supplementary materials with all procedure and stimuli used, including full scripts and translations in both native language and English.
  4. Please include demographic information for all studies (N, gender, mean age, std age). Please report any exclusions of participants, or lack of.
  5. Please include confidence intervals.
  6. Please indicate exact p values with three decimal digits.
  7. Please confirm that participants were randomly assigned into the experimental conditions.
  8. Please report N, means, and standard deviations for all conditions and all measures in all studies, and not just summary statistics. You can do a summary table.
  9. If possible, I (and the academic community) would greatly appreciate it if you decide to share your dataset and code with the academic community on Open Science Framework. I understand how some authors might be hesitant, but my experience is that both the community and the author both typically benefit, and there is much more to gain than lose. At the very least, please share this with us reviewers (http://help.osf.io/m/sharing/l/524049-view-only-links).