Difficult Data: Comparing the Quality of Behavioral, Recall, and Proxy Data Across Survey Modes
Abstract:
The mode choice literature is rife with evidence on the impact of different survey modes on response rates, respondent cooperation, and data quality. However, insufficient attention has been paid to the quality of “difficult data” provided when respondents cannot choose the mode and thus cannot maximize their comfort with the survey. Here, “difficult data” correspond to questions that are burdensome for respondents to think about – e.g., very specific details on a behavior, on past events, or on the behavior of other persons. Which mode best helps respondents provide difficult data with fewer break-offs, straight-lined responses, and “I don’t know’s”? Do cues from a phone interviewer help respondents recall elusive details? By contrast, does the simultaneous interaction with the interviewer intimidate respondents and constrain their ability to think through the answer? Similarly, does pacing oneself through an online survey provide respondents the time and comfort needed to think carefully through the responses, or does it weaken the motivation to complete the survey altogether? This paper assesses the impact of survey mode (without choice) on the quality of “difficult data,” focusing on a survey that was administered via CATI and online. The survey collected specific information on media consumption on mobile devices in a household, as reported by one “proxy” respondent. The CATI survey was administered on a dual-frame probability sample, and the online survey was administered on an opt-in panel of mobile users. While the CATI sample is representative, the panel sample was more likely to be familiar with the information required, allowing for insightful comparisons and directional evidence on mode effect on the quality of difficult data. The paper both presents findings from these surveys and raises a call for further research on survey mode, difficult data, and respondents’ place between them.
Recommended Citation:
Dan, M. O., Rao, K., & Kurmlavage, V. (2014). Difficult Data: Comparing the Quality of Behavioral, Recall, and Proxy Data Across Survey Modes. Paper presented at the American Association for Public Opinion Research, Anaheim, CA.
Attached Documents:
- AAPOR 2014 Program (see page #83 for the mention)
- For a copy of this presentation, please send me a comment with your email address in the box below. You can also reach me at kumar dot rao at nielsen dot com.