When less is more: improving survey invitations

This article was written by Hafsteinn Einarsson. Hafsteinn is a PhD student in the social statistics department at the University of Manchester. Before starting his PhD, Hafsteinn worked for several years as a survey project manager in Iceland. Here he shares findings from a recent article published in Survey Research Methods.

Survey participation depends on a variety of factors, from questionnaire length to respondent motivation, but only some of these can be easily leveraged by the data collector to improve survey participation. One of these factors is the design of survey invitations. And while there are recommendations regarding best practices for designing such invitations there is limited empirical evidence about the best design, especially in the context of mixed mode and web survey. In a recent paper published in Survey Research Methods, we explored whether manipulating two features, length of the invitation letter and the placement of the link to the web survey, could affect response rates in a survey of young people in Iceland.

Respondent burden

When designing surveys, particular care must be taken to minimize the effort needed by the respondents. This concept, which has received plenty of attention within the survey research literature, is called respondent burden. Avoiding it involves actions such as shortening the duration of surveys or asking questions using simple language. While well understood within the survey context, respondent burden has been explored less in the context of survey invitation design. Our experiment put this concept to the test, by sending four different versions of invitation (and reminder) letters to a sequential mixed mode web-telephone survey.

The first treatment, length, refers to the word count. We produce short letters which invited respondents to the survey using only the essential content (227 words, 1 page) and long letters, which included a much more detailed introduction (647 words, 2 pages). Theoretically, the visual calculus which goes into reading one page versus two pages of text could affect respondent burden.

As the letter invited people to participate on the web, we also tested whether link placement would affect response propensities. Invitation letters must present large amounts of information, but not all of these may be relevant for every recipient. Therefore, reading every word of the letter may be burdensome for those who are not interested in every aspect, such as how the research may be used or privacy information. Consequently, some readers may prefer letters with the link early in the letter, instead of at the end.

Finally, these treatments could interact as the long text version with the link at the end is the only one which presents the link on the second page of the letter. Aside from the length and link placement, the letters were structured in the same way.

Findings from the invitation experiment

The experiment was run in a mixed-mode survey, so different types of outcomes can be calculated. Therefore, we look at the effects of the experiment at three time points: after the invitation letter arrives, after the reminder letter and for the final results (following telephone and email reminders). Our results show that the long-end version appears to perform poorly. As data collection proceeds fewer overall responses are gathered from this version.

Participation rate over time for the four experimental groups in the web and overall survey

Regression analyses showed that both main effects (length and link placement) were significant after the reminder letter arrived, but not at other stages. Long letters and letters with links in the end negatively affected propensities to respond early but did not affect the overall results. This is important, as it shows that efficient invitation design can improve response rates early on, getting more respondents via a cheaper mode.

Practical implications of survey experiment

So, does this imply that short letters with links early on are going to improve every survey? Unfortunately, that is far from guaranteed. There is an inherent tension between the information that has to be presented to the respondents and the simplicity of the invitation letter. We showed that even while keeping the same structure it is possible to simplify survey invitations. This is especially important in the context of push to web and mixed mode methodologies.

What we can say though, is that reducing the burden of reading survey invitation letters can, in some cases, make people more likely to participate. We therefore urge survey practitioners who send mailed invitations to consider whether there is scope for reducing the burden they place on recipients. Design survey invitation letters with the recipient in mind to understand out how to present the necessary information as efficiently as possible.

You can find more about the study and the invitation letters in our paper published in Survey Research Methods here.