How Can We Believe What They Say? The Role of Missing and Validating Data in Panelists Demographic Information
Abstract:
The use of online panels (probability-based or volunteer opt-in) as a mode of data collection has become increasingly popular in market, social, psychological, and medical research (Callegaro and DiSogra 2009). The Nielsen online panel is one of the opt-in panels in the United States that is composed of respondents who voluntarily sign up (opt-in) to become members of the panel. As part of the sign up process, potential panel members are required to complete an online recruitment survey that asks for their basic demographic information. Like any other online survey, responses to some questions are optional (i.e. skip is permitted), while for others a response is required to advance to the next page. In this paper, we draw upon data obtained from the welcome survey to examine the issue of missing data (i.e., item nonresponse) in the optional set of demographic questions. Specifically, we examine the item nonresponse rate for these questions and dropout rates across all pages in the web survey questionnaire. Then, we analyze whether respondents who exhibited higher rates of nonresponse to the optional-set of survey questions differ significantly from those exhibiting lower rates – in terms of the answers that they provide to the required-set of survey questions. We construct an overall measure of data quality, defined in terms of item response rate, accuracy of responses, and completeness of responses, and identify demographic predictors of data quality in a multivariate context. Lastly, we attempt to answer one of the fundamental questions related to the practice of survey science “how can we believe what respondents say?” To answer this, we use panel members’ behavioral data, which is captured from their online and offline activity on their computers, to validate some of their responses to the demographic questions. We discuss the findings from the study and conclude with recommendations for future research.
Recommended Citation:
Rao, K., Cameron, D., & Kurzynski, D. (2011). How Can We Believe What They Say? The Role of Missing and Validating Data in Panelists Demographic Information. Paper presented at the American Association for Public Opinion Research, Phoenix.
Attached Documents:
- AAPOR 2011 Program (see page #117 for the mention)
- For a copy of this paper, please send me a comment with your email address in the box below.