TurkPrime is probably the best tool you can find for MTurk academic data collection. I use MTurk a lot for data collection, which makes TurkPrime one of the most essential tools available to me as a researcher. Even if you still have doubts about data quality and sample characteristics, I think there’s little disagreement that this is a suitable platform to pretest your research before going live with your target sample to tease out problems, and it is always a good idea to see if your findings replicate using this sample. Generally summarizing my experience, I find that the data quality is atleast comparable and most times exceeding that of student samples, and that findings elsewhere could generally be replicated quite well on MTurk (if done correctly). In a previous post I summarized all that I could find about the academic use of MTurk, which helps when I talk to other scholars and engage reviewers to discuss the advantages and disadvantages of this data collection platform.
Through a comment on my MTurk blog post I was made aware of a relatively new tool called TurkPrime, which refers to itself as “MTurk Turbo Charged” and is “free for academic use!”. It uses the MTurk API, so when it was asking for my MTurk account API details which grants it access to my account I became hesitant and worried of a scam, but after an email exchange with the developers at Touro College in NYC I decided to give this a try. It soon became clear that this service is extremely useful, simplifying the most tedious aspects of MTurk data collection. Attending APS conference 2015, I also had a chance to attend the developers’ symposium, and see the team in action. Since then I haven’t looked back, all my MTurk data collection is done using TurkPrime.
[Update: mid 2016, unfortunately, Turkprime started to charge for “premium” features, even for academics. This is disheartening, especially given its original promise and publication in a scientific journal that it would remain free for academics. However, the core features are still free, and are still of great help, and the charge is, still, rather low.]
What does TurkPrime do that’s so useful? If you’ve tried anything with MTurk before, you’ll appreciate each and every one of the following:
- Unique Turkers – preventing duplicate workers is a mess. There are all kinds of work arounds, but TurkPrime takes care of that for you by simply disallowing an MTurker to retake the survey again.
- Automatic validation and payment – TurkPrime checks the validation key at the end of the survey and pays the MTurker automatically if the validation code is correct. MTurkers really appreciate immediate payments.
- Stats – It gives you bounce rates, completion rates, etc.
- Exclusion/Inclusion – If you want to exclude certain MTurkers or make the survey only available to certain MTurkers, TurkPrime will take care of that for you. You can also set MTurkers groups and use those. You can also exclude/include all workers that previously participated in one of other studies you ran with TurkPrime. It’s as easy as a single click.
- Email workers – Much easier to email Turkers in a batch or all Turkers on a specific HIT.
- Easy pausing/resuming/restarting a HIT/Bonus granting – All highly automated, saving you the trouble from creating new versions of the same HIT and updating number of participants. Bonus granting is simple, and could also be done in batch.
There are two more terrific features introduced more recently:
- Microbatch (link)- Separating your HIT into micro-batches and also automatically restarting your HIT. This means you’ll get workers spread throughout the day rather than a single timeline, and that your HIT will always be ‘fresh’ on MTurk – something MTurkers seem to really respond to, making data collection quicker. In a recent move by Amazon July 2015 Amazon also started charging 20% commission for HITs with more than 9 workers ontop of the regular 20%, and if you set this Microbatch feature to 9 participants or less per micro-batch you’ll save 20% off your Amazon commission fees.
- Passing Query strings to Qualtrics (link) – A feature I requested a while back and was surprised to see implemented so quickly. TurkPrime now allows Qualtrics to automatically receive the MTurker’s WorkerID (if you need to match with future followups), assignment HIT, and HitID. This solves me previously having to combine datasets from Qualtrics and MTurk to match and followup.
There are also lots of other things in there I don’t use, like automatic bonuses, scheduling launch time (if you’re not in the US, and want to run this when it’s morning for American MTurkers), demographic targeting, etc.
All in all, the most essential tool you’ll find for MTurk, and I’ve probably seen and tried them all. This platform helped save me countless hours of having to work around bad MTurk features and interface trying to force MTurk to work for my academic data collection. Not to mention that now, with the micro-batch feature it’s also saving me (or the taxpayers) considerable money in Amazon commissions. I highly recommend this service, and I do hope they’ll keep it running and free, with more wonderful useful features. Way to go, TurkPrime team, very well done.