Mode, Measurement, and Mayhem: A Closer Look at Measurement Error in a Mixed Mode Study
Abstract:
The Gallup Panel recruits members randomly using Random Digit Dial (RDD) methodology and assigns members to receive surveys either through the mail or web. Since both of these modes are self-administered, there is no interviewer to motivate respondents to answer carefully. The only communication of the researcher with the respondents is through the instructions, question wording, and design. The Gallup Panel surveys often include many long lists of questions that use the same response options. When respondents answer long batteries of questions such as these, they may be more likely to change their response patterns towards the end of the battery as they become fatigued, resulting in measurement error. When fatigued, respondents adopt response strategies (such as non-differentiation and extreme answers) that permit them to reduce their perceived response burden. In this study, we investigate measurement error in a mixed mode survey conducted by The Gallup Panel. The survey was sent to active panel members who are assigned to receive surveys, either through the mail or online. Two versions of the questionnaire (reverse item order ballot design) were used in each mode and members were randomly assigned to one of the versions. Preliminary results from the mail surveys have confirmed more extreme and straight non-differentiated responses to items when they are asked later in the response list. Our main research question is whether we observe this measurement error in both mail and web mode groups. The paper discusses how researchers can prevent or reduce these measurement errors and concludes with future research avenues.
Recommended Citation:
Kephart, K., Kaminska, O., Rao, K., & Bilgen, I. (2008). Mode, Measurement, and Mayhem: A Closer Look at Measurement Error in a Mixed Mode Study. Paper presented at the American Association of Public Opinion Research, New Orleans.
Attached Documents:
- AAPOR 2008 Program (see page #46 for the mention)
- For a copy of this presentation, please send me a comment with your email address in the box below.