An Empirical Evaluation of Probing Questions Investigating Question Comprehensibility in Web Surveys

It is essential for the quality of survey data that questions are easy to understand. We developed four variants of comprehensibility probes measuring the comprehensibility of a survey question as a whole or of individual aspects (vocabulary, sentence complexity, content). We compared the probes on four experimental questions in terms of response distributions and mean comprehensibility scores to assess their sensitivity to capture comprehensibility problems at different levels of granularity, as well as nonresponse and response times as indicators of respondents' willingness and required effort to answer the probes. We found no differences in the response distributions and mean comprehensibility scores, while item nonresponse and response times differed significantly between the probe versions.