New Developments in Combining Probability and Non-probability sampling

Non-probability sampling is receiving increasing attention in social research due to the escalating costs of data collection and decrease in response rates. While this has often been presented in opposition to traditional probability sampling new research is investigating the potential of combining the two approaches to collecting data. In this half day event we will be exploring how combining probability and non-probability data can help improve causal inference, population estimates and reduce costs.

Schedule

Date: 5^th of May 13:30-16:00 BST (UTC+1 time)

Location: Whitworth Building Council Chamber, University of Manchester and ONLINE

Registration Form: https://forms.office.com/r/RRbUvRUrpJ

Abstracts

Using Synergies Between Survey Statistics and Causal Inference to Improve Transportability of Clinical Trials

Michael R. Elliott, University of Michigan

Randomized trials have been the gold standard for assessing causal effects since its introduction by Fisher in the 1920s, since they eliminate both observed and unobserved confounding. When randomized trials are conducted in human populations, estimates of causal effects at the population level can still be biased if there is both effect modification and systematic differences between the trial sample and the ultimate population of inference with respect to these modifiers. Recent advances in the survey statistics literature to improve inference in nonprobability samples by using information from probability samples can provide an avenue for improving population causal inference in clinical trials when relevant probability samples of the patient population are available. We review some recent work in “transporting” causal effect estimates from trials to populations, and propose a “doubly robust” estimator of population causal effects that is consistent if either the odds of being the clinical trial versus the population can be correctly estimated, or if the effect modification of the treatment can be correctly estimated. We explore our proposed approach and compare with some standard existing methods in a simulation study, and apply it to a study of pulmonary artery catheterization in critically ill patients where we believe differences between the trial sample and the larger population might impact overall estimates of treatment effects.

Implementing and Adjusting a Non-probability Web Survey: Experiences of EVENs (Survey on the Impact of COVID19 on Ethnic Minorities in the United Kingdom)

Natalie Shlomo, Andrea Aparcio-Castro, Daniel Ellingworth, James Nazroo, Harry Taylor – University of Manchester
Nissa Finny – University of St. Andrews
Angelo Moretti – Manchester Metropolitan University

 We discuss the challenges of implementing and adjusting a large-scale non-probability web survey. For the application, we focus on the 2021 Evidence for Equality National Survey (EVENS) which was led by the Centre on Dynamics of Ethnicity (CoDE) at the University of Manchester in the United Kingdom, in partnership with Ipsos-MORI. The aim was to understand the impact of the COVID19 pandemic on ethnic and religious minority groups in the UK. Standard probability-based surveys, even with ethnic minority group boosts, do not have the sample sizes required to obtain reliable estimates for small group statistics. We therefore planned and implemented a non-probability web survey of ethnic minority groups to overcome these limitations. We formed partnerships with community organizations and used innovative recruitment strategies, including digital and social media. Daily monitoring of the data collection against desired sample sizes and R-indicator calculations allowed the team to focus attention on the recruitment of specific groups in a responsive data collection mode. We also supplemented the sample with existing members in both established non-probability and probability-based panels in the UK. We describe the measures applied to improve the quality of the collected data and the statistical adjustments to correct for selection bias based on estimating the probability of participation in the non-probability sample using a probability-based reference sample and calibration.

Supplementing Small-Sample Probability Surveys with Nonprobability Surveys to Improve Estimation and Reduce Costs

Joseph W. Sakshaug, Institute for Employment Research (IAB) & LMU-Munich, Germany

Large probability-based sample surveys can be prohibitively expensive to carry out. As such, many survey institutes have shifted away from fielding expensive probability samples in favor of less expensive, but possibly less accurate, nonprobability online samples. Instead of abandoning probability sampling (and all its useful properties) entirely, we propose a model-based approach that allows for fielding a small probability sample survey and borrowing information from a parallel (and potentially biased) nonprobability sample survey. Through simulations and real-data applications we show that the method reduces the variance and mean-squared error of regression coefficients and predictions, relative to a probability-only sample survey. The potential for cost savings is also evident.

Registration Form: https://forms.office.com/r/RRbUvRUrpJ