Check me, replicate me: Corrections & replications of my work

Please help me do better science!

Jump to: What I offer / I’ll pay you / Corrections of my work / Replications of my work / Questions about my work

 


Background

Since embarking on my open-science journey in 2017-8 I’ve been trying to be as open as I can about everything that I do, hoping that an open scientific process would help ensure that my work is solid and without any errors and mistakes. But for that we need a community, not a few peer reviewers. I, we, need your help in ensuring the validity of my work. I have been trying to promote replications and the validation of the published literature, and I would hope that others would do the same for me, by checking and replicating my work.

 

I really appreciate those who take out of their time and resources to help others check their work, revisit and assess preprints and articles, and replicate published/preprinted findings.

I would be honored to have you look into my work.

 


What I offer

  1. I try and share whatever I can from the projects I run, yet if somehow something needed is missing, please reach out to me. I would do what I can to provide you with the needed information.
  2. I would be happy to review replications of my work, and provide feedback, providing the obvious academic time pressures.
  3. Not a must, but if you wish, I would be very glad to post about this, and credit you. If at some point it will ever get to an official correction, I would like to give you the credit for the important work you did to help me and the community do better.
  4. I don’t have grants and funding, but I will commit out of my own money to pay you for the mistakes you find. Terms and conditions below.
  5. I don’t have grants and funding, but I will do what I can to try and support replications of my own work. If I happen to have a budget for data collection, I will try and support your replication’s data collection. If I can, I would love to help you in seeking out funding to replicate my work.

 

Found a mistake in my work? Great! Thank you!
and… I’ll pay you

This idea is adopted from Science Fiction’s corrections bounty, who based this on many others. I think this is something all researchers should adopt.

 

I’m willing to pay if you find minor or major objective errors in my published work.

  1. Minor errors would include reporting a numerical result incorrectly. This doesn’t include typos or grammatical errors unless they substantially change the intended meaning of what I’ve written. I’ll pay you 5 US$ if you find a minor error.
  2. Major errors would include a major reporting problem where I analyzed or reported something incorrectly, such that there is an error in my methods analysis, there is an error in my code, or that the conclusions I draw are different than the conclusions warranted from the data. I’ll pay you 50 US$ if you find a major error.

 

You can report errors using this website’s Contact Form.

 

The rules:

  • These have to be objective errors – not matters of opinion or interpretation. If you just disagree with something I wrote in an article, feel free to tell me, or write a commentary/review, but it won’t warrant payment.

  • Please send me as many details of the error as possible: which article, what page, what code, which file, etc. and please include a detailed explanation of what’s wrong.

  • Obviously, the error has to be something that isn’t already listed on this page.

  • In most cases I’ll decide myself whether it’s an error. If you insist that there’s a mistake and I don’t agree, we might ask a third-party arbiter to decide who’s correct. If you don’t end up convincing me, it’s not like you lose any money.

  • Although you can send me an anonymous or pseudonymous message, if I agree you’ve found an error, you’ll have to send me a way to pay you, which might involve revealing your identity. We can discuss the specifics when you get in touch.

  • I’ll post your name or pseudonym (or just “[anonymous]” if you wish) alongside the correction on this page.

  • If you don’t want the money yourself, I’ll donate it to charity and send you the receipt. By default I’ll give to the Malaria Consortium, currently the most cost-effective charity in the world according to Givewell.

  • I’ll keep paying out for errors until I’ve paid 2000 US$ (to be revised if I ever get any grant or funding to support this activity). I will alert issues in advance if I ever get close to this limit.

 

List of corrections of my work

So far, nothing. This just probably means we haven’t tried enough.

You can report errors using this website’s Contact Form.

 

 

List of replications of my work

I don’t know of many replications of my work, I am just an early career researcher, but I’ll list whatever little I do know. Both successful and failed.

 

Article: Feldman, G., Chao, M. M., Farh, J. L., & Bardi, A. (2015). The motivation and inhibition of breaking the rules: Personal values structures predict unethicality. Journal of Research in Personality59, 69-80.

Replication #1:

  • Article: Ring, C., Kavussanu, M., & Gürpınar, B. (2020). Basic values predict doping likelihood. Journal of sports sciences38(4), 357-365.
  • Type: Conceptual
  • Findings:
    • Replication shape consistency=.92, their 2nd replication shape consistency = .91. This seems inline with our values-ethicality shape consistency =.88.
    • Replication correlations: .29,-.21,.11,-.22. This seems inline with our values-ethicality meta correlations: .31,-.25,.20,-.26.

 

Article: Feldman, G., & Albarracín, D. (2017). Norm theory and the action-effect: The role of social norms in regret following action and inaction. Journal of Experimental Social Psychology69, 111-120.

Replication #1:

  • Article: Regret and Disappointment are Differentially Associated with Norm Compliance and Norm Deviant Failures (link to poster abstract). I was a reviewer on their submission to JESP, which for reasons I’m not sure I understand was rejected.
  • Type: Conceptual
  • Findings: Successful replication. Hope it gets published/preprinted so I can link to it and the findings.

List of questions about my work

These are issues that were raised about some of my work.

Feldman, G., Lian, H., Kosinski, M., & Stillwell, D. (2017). Frankly, we do give a damn: The relationship between profanity and honesty. Social psychological and personality science8(7), 816-826.

Questions were raised regarding two aspects of this article:

  1. The use of the so-called “lie-scale” as a measure of “honesty” in Study 1.
    1. A commentary was posted on the article, to which I wrote a response, which received another preprinted commentary, and a follow-up study.
    2. A blog post shared some of the remarkable exchange between the commenters and the editorial team.
    3. I’m puzzled by what the group of commentary authors did here, their published motivations, scientific claims and evidence raised in the commentaries. Instead of the commentaries with the vague indirect evidence and broad claims (with comments about me in the email exchanges), the more recent follow-up study was the right thing to do to begin with. This is how science should progress – by providing high-quality pre-registered open rigorous evidence. This article seems like a promising first step.
      There are a still lot of things that confuse me about this research domain, the commentaries, and the claims made, I write some of that in my response. As an early career researcher at the time, I can generally comment that the exchange with this group did not feel scientific, and I hope that future academic scientific exchanges can be more evidence based, open, positive, and constructive.
      The commentary group seems to think all issues have been resolved long ago. Maybe, I think there are still many unresolved questions in this domain, the lie-scale is self-explanatory and folk interpretations seem pretty straight-forward. That said, this is not my research domain, I am not an expert.
      I hope others will use these as a starting point for understanding this tool better. I think humility is warranted here, until this is sorted out.
      For now, I am grateful for the interest that my research sparked, and I suggest caution in interpreting these findings.
  2. The use of the dishonesty measure based on a linguistic analysis of text.
    1. In retrospect, this is weak. I based this analysis on published literature, was a bit naïve to accept it as it, and I should have spent more time re-validating this tool. I was not critical enough to recognize its possible weaknesses. I now doubt that this measure/tool really is able to detect dishonesty in text. Linguistic detection of dishonesty is yet another domain that need re-evaluation and sorting out, and is not my research domain and my expertise and ability to contribute to this domain is rather limited.

 

In addition, there were many misunderstanding about this article and our conclusions in the media. I refused to take part in this media blowup other than referring to my posted clarifications on my blog.
Bottom line, this was a first step study hoping to spur follow up research, and – again – humility is warranted here. We need far more research before we can draw any conclusions here.

 

 

My own concerns

Feldman, G., Chandrashekar, S. P., & Wong, K. F. E. (2016). The freedom to excel: Belief in free will predicts better academic performance. Personality and Individual Differences90, 377-383.

I was concerned when faced with the p-values that came out of the studies reported in this article. Some of the p-values in this article are just below 0.05, and though there were no exclusions of participants, and no peeking, these studies were not pre-registered and the p-values that felt problematic and possibly indicated issues with flexibility of data analysis. I felt like this should be shared and communicated, but was concerned about how solid this is.

In Study 1, there might be a confound of background and English proficiency, so would be best to rerun this with a more homogeneous sample and if possible to test that they have similar English skills. In Study 2, though this reports the complete sample as I received it from the university, the effects and p-values for some of what I reported seem very weak, especially given the large sample.

In retrospect, I wish I had done much better here. I should have probably run replications of this work before proceeding to a submission. This would have been better off as a preprint awaiting further data. This became especially clear when the review process at PAID turned out to be inadequate, and when the media started reporting on this paper, overinterpreting the results (assuming causality, etc.). Even if the evidence here is solid, which remains to be determined, we are talking about variance explained of ~1%, and correlational designs

At the very least I suggest strong caution and much more humility in interpreting the results in this article, and I really hope for replications of both studies. Do you know of successful or failed replications of this work? please do share and let me know.

I am not sure why, but this seems to be one of my most highly cited papers.

 

Feldman, G., Chao, M. M., Farh, J. L., & Bardi, A. (2015). The motivation and inhibition of breaking the rules: Personal values structures predict unethicality. Journal of Research in Personality59, 69-80.

I think the correlational mini meta of many samples we did in this article’s Study 1 is pretty solid, large samples, lots of measurs, and with some initial replications that give me confidence about those findings.

Yet, given the concerns regarding the profanity-honesty article Study 2 using indirect linguistic deception detection, we used a similar method here in Study 3, so, I’ll repeat what I wrote above:

“In retrospect, this is weak. I based this analysis on published literature, was a bit naïve to accept it as it, and I should have spent more time re-validating this tool. I was not critical enough to recognize its possible weaknesses. I now doubt that this measure/tool really is able to detect dishonesty in text. Linguistic detection of dishonesty is yet another domain that need re-evaluation and sorting out, and is not my research domain and my expertise and ability to contribute to this domain is rather limited.”

Study 2’s method seems very context dependent to that point of time for that status of MTurk, so I wish I used more robust methods that could remain relevant for longer.

Put together, I think the link to unethicality self-reports seems pretty solid, but we need to revisit and redo the link between values and real life unethicality using better more robust methods. I’d love to see others look further into that link, and happy to help with that.