L-R: James N. Druckman, Adam J. Berinsky
RSF trustee James N. Druckman (University of Rochester) and former RSF visiting scholar and researcher Adam J. Berinsky (Massachusetts Institute of Technology) recently published an article in Nature Human Behavior analyzing the representativeness and validity of nine opt-in non-probability online data sources with colleagues Michael N. Stagnaro (Massachusetts Institute of Technology), Antionio A. Arechar (Center for Research and Teaching in Economics), Robb Willer (Stanford University), and David G. Rand (Massachusetts Institute of Technology and Cornell University). The article aims to “offer guidance for choosing opt-in samples, depending on the purpose of the research and resource constraints.”
They found that samples from Lucid and Forthright were the most representative and that samples from Cloud Research Connect and Prolific and the Stanford CRSTAL M-Turk panel had the highest response validity.
Their study has four key takeaways:
First, there is substantial variation in the response validity and representativeness across the samples they examined, therefore, any general statements about opt-in non-probability samples would be misleading.
Second, there is a negative relationship between response validity and sample representativeness, that is, more representative samples tend to have lower response validity; a negative relationship between sample representativeness and respondents’ experience with similar studies, that is, the more representative a sample, the less likely respondents have taken similar surveys; and a positive relationship between response validity and respondents’ experience with similar studies, that is, the more valid the responses, the more likely respondents have taken similar surveys.
Third, filtering out inattentive participants improves response validity in samples that are more representative. They find that filtering by two trivial attention checks in the beginning of the study in the Forthright sample improved response validity without compromising representativeness.
Fourth, applying weights to less representative samples does not consistently close the gap between them and more representative samples when assessing how treatments impact groups differently.
Recent Books
