Using email to disseminate survey invitations provides an easy and inexpensive means of recruiting respondents. Unfortunately, response rates for most email-invite samples are quite low. Moreover, respondent self-selection almost always results in a sample which differs substantially from the target population or that can’t be compared to the target population. Options are available (such as weighting of data) to address sampling problems, but require some planning, prior testing, and changes in analytical approach:
- The concept of confidence intervals and sampling error applies only where sampling procedures truly qualify as “random.” Due to opportunity for self-selection, email invite surveys generally don’t qualify.
- No matter how respondents are recruited (random or not) every study sample should be compared to the original name list and/or target population to identify differences. If the demographic profile of respondents from an email-invite survey differ from the original list, data should be adjusted or weighted to determine whether or not differences are meaningful.
- Contact lists should contain information about potential respondents beyond just email addresses to support comparisons between respondents and the original sampling frame. Without any demographic parameters, there is no way to determine whether or not respondent data accurately reflects the broader target population.
- Email lists for stakeholder groups (current students, faculty, staff, et al.) are generally of high quality and include essentially all population members. The quality of email lists for external audiences (prospective students, parents, potential employers, even inquiries) vary widely. If available email lists include only a minority of population members or the process used to generate lists could exclude a sizable proportion, other options or respondent recruitment approaches should be used.
Beyond email invitations, there are other methods which might be used to recruit respondents for web-based surveys. Most are less susceptible to self-selection bias, though this improvement generally comes at a higher cost. A few options to consider for recruitment of respondents for web-based surveys:
- Telephone recruitment with push to web – this format relies on a telephone sample where potential respondents are screened and recruited via telephone. Respondents who qualify and agree to participate are provided an email containing a link to complete a survey online. This approach avoids self-selection bias, but the cost associated with telephone recruitment can come close to the cost of a telephone survey.
- Online panel recruitment – this format relies on a sample of potential respondents screened and recruited from an online panel. Recruitment criteria can be outlined very precisely so the sample will accurately mirror the target population. Additionally, the cost is generally reasonable. The primary limitation is that panel membership may be insufficient to generate a meaningful number of completed surveys within certain geographic markets