Sometimes, you want to make sure your participants are from a specific country. If you’re running studies on Amazon Mechanical Turk there is a problem with some workers who self-report to be the in the US and take part in HITs intended for American participants and are not located in the US. In some of my surveys, even if I make it explicitly clear that I am only interested in native-English participants, born, raised, and currently located at the US, I still get a few workers who Qualtrics shows took the survey from other countries (typically, India). When it launched, Turkprime had location validation which they called “Verify Country by IP” but some time last year they decided to start charging for this as a pro ‘feature’, even for academics. But, there’s a way to implement this on Qualtrics. Here’s from the Qualtrics team:
Qualtrics estimates location based on a respondent’s IP address. This data is called “GeoIP Location” in survey logic condition dropdown menus. You can learn more about using GeoIP Location to set conditions for your respondents here.
You can use Branch Logic in your Survey Flow to send respondents who do not meet your location requirements to a custom end of survey message.
Qualtrics uses the general country codes for GeoIP location. Here is the list of country codes that you could used for GeoIP location.
This is what it would look like on your Qualtrics survey flow:
This is the custom message I display when verification fails (in the “End of Survey” above):
Our survey in only intended for participants currently based in the United States. We use software to determine your location and the software has indicated that either you are not currently located in the US or that it was unable to determine your location. In case of the latter, this could be due to VPN, proxies, firewalls, or IP masking.
Unfortunately, this means you cannot take part in our study, please return the HIT.
We do apologize for any inconvenience caused, and hope you’ll consider taking our surveys in the future.
The research team
You’d also probably want to prevent participants from taking your survey multiple times. Turkprime also started charging for that with their “Block Duplicate IP Addresses” pro option. But Qualtrics has a feature to address this (see survey protection):
In circumstances where you are offering an incentive or conducting a sensitive vote, you may want to prevent participants from taking a survey more than once. The Prevent Ballot Box Stuffing option helps keep respondents from taking a survey multiple times by placing a cookie on their browser when they submit a response. The next time the respondent clicks on the survey link, Qualtrics will see this cookie and not permit them to take the survey.
- Open Survey Options in the Survey module.
- Select the Prevent Ballot Box Stuffing checkbox.
Both features are not 100% perfect and clever nasty participants can work around them with clearing cookies and using VPNs into the US. Still, it’s another easy measure to try and increase the quality of your data collection.
[Update August 2018]
I emailed Qualtrics to try and check with them what would be the best way to go about it, and this is what they wrote back:
You can always use the Q_Language embedded data field to capture the language the survey taker is using, however this is distinct from the default browser language. Fields like User Agent will capture most browser metainfo, however, I don’t believe browser language is included in this.
[end update August 2018]