Lots of missing data and failed manipulations are just some of the issues you may encounter while running questionnaires or experiments with coupon based or paid participants, especially students, especially undergrad students. They lack motivation & they want to finish the pain as soon as possible, so getting them to actually read the questions and think about them carefully is difficult.
Following are some of the lessons I’ve learned throughout the years on how to best approach student samples :
- Run the questionnaires on computers using software like Qualtrics or the open source Lime Surveyor.
- It allows you…
- Tight control over the flow of the questionnaire. For example, display manipulation checks after the manipulation is completed without allowing participants to go back and change their answers. Regularly, with paper, this would mean taking back the paper and giving them another one.
- Force responses (or kindly ask that they rethink leaving it blank)
- Input validation (like, an integer between 0 and 100)
- Randomization (for experiments)
- Selected views (display questions to users based on certain conditions or their responses)
- Many times of questions (integrate images, videos, etc.)
- Advanced data to play with (heat-maps, timers for page clicks and submit etc.)
- Automatic data export and coding to Excel and SPSS
- Ask the participants to explain their decisions in an open text using 2-3 sentences and check input for key length.
- Tell participants in advance that you will ask them questions about their decisions. Make sure that you indeed follow up on that promise.
- Quality of data
- Ask participants about obvious demographics for control. Use page timers and survey completion time as controls.
- If you’re giving a questionnaire not in their native language – ask for language proficiency and how well they understood the questionnaire.
- Length and time
- Don’t make the questionnaire too long. Over 20-30 minutes you’ll likely to get garbage response. If you get the participants for a one hour session, split those into two half an hours.
That’s it for now, will add more as those come up.