Understanding the Measurement Quality in Smartphone Usage: Surveys vs. Digital Trace Data

In the evolving social science field, accurately capturing human behaviour through data collection methods is critical. In a recent study, we explored the effectiveness of traditional surveys compared to digital trace data in measuring smartphone usage. The research focused on five key activities: making phone calls, sending and receiving text messages, taking photos, engaging with social media, and browsing the internet.

The study employed an experimental design. Participants provided information about their smartphone usage via surveys and allowed tracking of their digital activities on their phones. The surveys included different question formats: a 5-point scale, a 7-point scale, and open-ended questions about duration. The digital trace data tracked the actual frequency and duration of smartphone activities.

One of the significant findings was the low correlation between survey data and digital trace data, indicating that these methods might be capturing different aspects of the same behaviours. For example, while digital trace data showed higher quality in measuring some types of activities, it struggled with accurately capturing text messaging and photo-taking behaviours. Conversely, survey data, particularly with a 7-point scale, showed better consistency in some measures.

The study utilized the MultiTrait MultiMethod (MTMM) model to separate valid data from errors and biases. This model revealed that both methods had strengths and weaknesses, with neither being perfect. Digital trace data can give granular details and reduce recall bias, but it can be affected by users’ selective tracking behaviours and device usage variations. Surveys, while susceptible to recall errors and social desirability bias, can more reliably capture certain behaviours.

These findings have significant implications for researchers and policymakers. The assumption that digital trace data is inherently superior to survey data is challenged. Instead, a hybrid approach that combines both data sources may offer the most comprehensive insights. This combined method could leverage the subjective insights from surveys and the objective granularity from digital trace data, providing a more accurate and holistic view of smartphone usage.

In conclusion, the study suggests that while digital trace data has many advantages, it should not be considered a standalone solution for measuring human behaviour. By integrating it with traditional survey methods, researchers can overcome the individual limitations of each approach and achieve better data quality and reliability. This nuanced approach to data collection is essential for advancing our understanding of digital behaviour in social science research.