Why do people take longer to do surveys by mobile?

As more and more surveys are done online there is an increasing concern about the devices used. This is especially important in large and complex studies that require people to spend more time answering questions and complete more complex tasks.

Understanding if data quality is the same across devices (for example PC and mobile) is important as differences between these can lead to biased estimates of the population. One of the consistent findings in this area is that respondents answer questions faster on PC than on mobile devices. This finding is important as response times are associated with data quality and breakoffs. Understanding where this difference comes from can inform the development of surveys that give consistent answers across devices.

In a recent paper that was published in Social Science Computer Review Chris Antoun and myself used an experiment in the Netherlands to answer exactly this question.

Using an experimental design in a representative panel

We had the advantage of a great design implemented in LISS panel, a representative online panel in the Netherlands. Respondents were interviewed at two points in time using either a PC or a mobile. Two random groups received the implementation in reverse order and those without mobiles were given one. Because we have a combination of between and within experimental design in a representative sample and we also offered mobiles to those that do not have them we manage to control for a number of possible confounders (like selection into device). This makes the data and the findings unique in this area.

Why do people answer slower by mobile?

One advantage of our data and statistical approach is that we can decompose variance and really understand what is causing the differences in timing. In the paper we show that 45% of the variance in completion time was at the question level and that 15% was at the respondent level. Thus, we can say that question characteristics are more important in explaining response times than respondent-level characteristics.

Consistent with previous research we found that respondents were slower on smartphones. On average it took them 1.4 times longer when completing the same surveys on smartphone than on PCs. Looking at questions characteristics we find that multi-item pages and the use of text input leads to longer completion time on mobile compared to PC.

We also found that smartphone-PC time gap was partially explained by familiarity and experience with smartphones. For example, for those that never used a smartphone before it took 22 seconds to complete the survey on PC and 40 second to do it by smartphone (a difference of 18 seconds). If we look at those that use smartphones every day the times were 16 second and 20 seconds (a difference of 4 seconds).

How to design surveys for mixed devices?

So, what can we do with these findings? From a survey design point of view avoiding multi-item pages and text input might decrease the time difference in response times between devices. This difference might also decrease in time as more people become familiar with smartphones.

If you want to read more about this research you can have a look at our recently published paper in Social Science Computer Review. Unfortunately, this is not open access but you should be able to get free access using this link.

Want to know more about designing Web surveys? Have a look at this article discussing a new response scale for Web surveys.