Building an Open Qualitative Science
Kathryn J. Edin, Princeton University
Corey D. Fields, Georgetown University
Jonathan Fisher, Stanford University
David B. Grusky, Stanford University
Jure Leskovec, Stanford University
Hazel R. Markus, Stanford University
Marybeth Mattingly, Federal Reserve Bank of Boston
Kristen Olson, University of Nebraska-Lincoln
Charles Varner, Stanford University
The qualitative research tradition appears to be on an upward trajectory. In the last five years alone, qualitative scholars have generated a raft of influential findings within such core social science areas as poverty and material deprivation, residential segregation, policing and the criminal justice system, health disparities, immigration and ethnicity, housing and eviction, public surveillance, populism and the radical right, and science and genetics (e.g., Stuart 2020; Claire 2020; Brayne 2020; Watkins-Hayes 2019; Sharkey 2018; Hochschild 2018; Western 2018; Jiménez 2017; DeLuca et al. 2016; Desmond 2016; Nelson 2016; Fields 2016; Edin and Shaefer 2015). This influential line of recent qualitative scholarship is joined by an equally influential stream of “fast science” qualitative journalism appearing in the New York Times, the Washington Post, and all manner of other media outlets. The country depends heavily on these two streams of qualitative work to build a richer basic science, to develop policy, and to understand ongoing crises and new developments in real time.
Although no one could dispute the profound impact of these scholarly and journalistic streams of qualitative work, the growing success of the form has also made it a target of criticism, much of it raising concerns about replicability, transparency, and representativeness (see Lubet 2017; Shaefer and Alvesson 2017; Lewis-Kraus 2016; Jerolmack and Khan 2014; Hammersley 2013; Lamont and White 2008; Duneier 2011; King, Keohane, and Verba 1994). In some cases, this “open science” criticism comes in an overtly hostile form, a type of criticism that’s focused on discrediting the tradition in its entirety or, alternatively, advocating on behalf of particular variants of it.
The American Voices Project (AVP), the country’s first platform for conducting qualitative interviews with a nationally representative sample, was also spawned by this growing commitment to open science but instead proceeds by developing a new qualitative form that’s intended to stand side-by-side with the already immensely successful existing variants. The AVP’s simple objective is to begin the task of building a new qualitative research form that rests on representative samples, open data, and secondary analysis and that’s intended to supplement – rather than replace – existing qualitative forms.
The purpose of this call is to roll out this AVP-based qualitative analysis by opening up the AVP dataset to qualified scholars and analysts. We welcome research on the many topics – including health, poverty, politics, protest, employment, coping, and anomie – that the AVP interviews can assist in understanding. Although most RSF issues are topically focused, this issue will be topically broad and is instead unified by a commitment to exploring the hopefully broad payoff to this new form of qualitative data collection. The balance of our call discusses the design of the AVP, the topics covered in the interview schedule, and the types of research questions that it opens up and that are supported by this call.
An overview of the AVP
The AVP received start-up funding from the Ford Foundation, the Chan-Zuckerberg Initiative, the Bill and Melinda Gates Foundation, the Russell Sage Foundation, the JPB Foundation, the James Irvine Foundation, the Annie E. Casey Foundation, a coalition of Federal Reserve Banks, and other sources. From the outset, the Russell Sage Foundation played an especially critical role in building the AVP, with two RSF conferences seeding some of its most important features.
The AVP sampling design rests on a representative draw of Census block groups (with over-sampling on high-poverty sites). Because block groups are the best available operationalizations of neighborhoods, this block-group design makes it at once possible to generalize to the U.S. population and to compare attitudes, behaviors, and experiences across different types of settings. For each block group, a representative address-based sample of households was then drawn, again with over-sampling on low-income and middle-income addresses. The AVP’s sampling design is described in more detail here.
The AVP’s qualitative protocol begins with the prompt “tell me the story of your life” and goes on to probe on the rhythm and routine of everyday life via supportive and non-judgmental questions. It addresses such topics as (a) food insecurity and household spending on food, shelter, and other basic needs, (b) the rhythm and routine of everyday life among household members, (c) employment, earnings, and job search, (d) current and past engagement with safety net programs, (e) the health of family members and their access to and experiences with healthcare, (f) mental health, drug use, anxiety, and stress, (g) parenting, family conflict and trauma, and family supports, and (h) political views, voting behavior, and reactions to ongoing protests. This protocol, which is delivered holistically as part of an engaging conversation, is then followed up with a survey assessing health, mental health, stress and anxiety, political views, social standing, discrimination, and much more. Although most respondents have consented to link their responses to administrative data (e.g., earnings reports), those linkages are not yet completed and will not be available for analyses based on this research call.
The AVP interviews cover nearly two years from the fall of 2019 through the summer of 2021. Initially delivered as a face-to-face interview, the AVP transitioned to remote interviewing in March, 2020, when face-to-face interviewing became unsafe. This conversion required modest adaptations to the protocol designed to ensure that rapport with the respondent could still be cultivated in the new remote-interview form. In a series of trials with the revised remote protocol, we found that our starting prompt (i.e., “tell me the story of your life”) remained effective when delivered over the telephone, as it signaled that the interviewer wanted to listen to the respondent’s story and was not delivering a forced-response survey.
Because the pandemic period has been so tumultuous (e.g., lockdowns, unprecedented collective action), new survey items and qualitative prompts were introduced periodically to reflect new developments, including probes designed to uncover possible crisis-induced changes in (a) health, mental health, and healthcare (including explicit probes on vaccination attitudes), (b) employment, earnings, and safety net usage (including new stimulus programs), (c) schooling and childcare arrangements, (c) attitudes about race, racism, and protest, and (d) attitudes about voting and politics. In a series of AVP crisis reports and opinion pieces, many of these items have already been analyzed, with some of the early AVP reports confirming journalistic accounts and others exposing a hidden side of the pandemic that wasn’t always well captured either in conventional journalism or in scholarship based on surveys or administrative data. It may be useful to read the AVP crisis reports when developing new proposals that are crisis-focused (but of course this call is also open to research on non-crisis topics).
The AVP interviews are typically 2-3 hours long. The interviewers, all of whom received intensive training in interviewing and qualitative methods, were a mix of advanced degree-holders (e.g., PhDs), graduate students, and undergraduates selected through a highly competitive process (see our methodological overview for further details). The full interview and survey schedule will be made available to all interested parties when a nondisclosure agreement, accessible here, is signed and submitted. When the signed NDA is submitted, access to a mock transcript will also be provided, as this will help convey the flow and rhythm of a typical interview. We will also release a description of all changes (and the timing of such changes) to the interview and survey. It will of course be useful to study these materials in advance of submitting a proposal.
We anticipate selecting 10-20 research proposals for publication in this RSF issue. If a large number of high-quality proposals is submitted, an additional tranche of proposals will be accepted without providing any commitment to publishing the resulting research in the RSF journal (and hence scholars will be free to publish in an outlet of their choosing). The main purpose of this initial public call is to build a stronger qualitative science in a host of substantive areas. This call will additionally serve as a “trial run” for future public calls for AVP research that will be open to a larger number of users and based on a larger sample of AVP transcripts. In these subsequent public calls for AVP research proposals, the data will be made available to qualified analysts who didn’t participate in this first trial run as well as to past users who did participate in this round and wish to submit a new proposal that extends their past research or is based on an altogether new topic.
After the proposals for the current round are selected, the data will be made available as deidentified transcripts, accessible only on a secure server. To ensure that confidentiality is maintained, all analyses must be completed on this server, and results cannot be made public until a disclosure review has been completed. Although the transcripts will go through a stringent deidentification process (entailing removal of names, addresses, and related information), the disclosure review is also an exceedingly important last line of defense against possible reidentification. Because the disclosure review will be comprehensive and time-consuming, it will be necessary to submit the drafts (and all tables or figures) well in advance of the RSF review process.
We anticipate that approximately 1500 transcripts will be deidentified, transcribed, and available by January, 2022 (at which time the selected proposals will be announced). The data, which will be released with weights reflecting the sampling design and nonresponse bias, will be linked to demographic information at the individual and household level (e.g., race, ethnicity, age, income, region, urban/suburban/rural neighborhoods, interview date). The demographic data can be used to limit analyses to selected subpopulations (including intersections of interest), to compare across subpopulations, or to oversample on particular subpopulations. It will also be possible to draw a representative sample from the full population of transcripts (which may be useful for researchers wishing to carry out a “conventional” qualitative analysis based on a smaller number of representative transcripts). The key descriptive statistics, including the approximate number of cases available for subpopulation analysis, are provided on our webpage dedicated to the RSF call.
Possible research questions
The AVP can support research on a wide range of topics. The following is a brief – and far from exhaustive – sampling of research areas that might be pursued.
Deprivation: The U.S. has long run a high-poverty economy even during the best of times. Although the usual quantitative measures of deprivation (e.g., poverty rates, food insecurity measures) are critically important, the AVP provides direct evidence on everyday life that reveals various types of deprivation among households that don’t always present as poor or food-insecure under conventional measures (see the AVP report on material hardship).
Poverty description: The AVP data can likewise be used to address ongoing controversies about the relative merits of different ways of describing poverty. The thresholds defined by competing measures can be compared by examining whether they capture the key qualitative differences in deprivation, disruption, and stress for different types of poverty (e.g., rural, urban), for different racial and ethnic subgroups, and for different multiples of the thresholds (referring to extreme, deep, and near poverty).
Safety net: Because the U.S. has a relatively weak system of automatic stabilizers, the new stimulus packages rolled out during the pandemic rely on a spate of income-supplementing programs that have not been fully field tested. Because some of these new programs may become permanent, it is especially important to monitor how they are or are not meeting needs, an objective that the AVP is especially well positioned to meet (see the AVP report on material hardship).
Labor market: Over the last year, employment and unemployment rates have fluctuated dramatically, especially for face-to-face workers. The AVP provides evidence on how people respond to sudden job loss, how they deal with their bills, whether they turn to friends and family for help, and what types of job search behaviors they deploy. These direct immersive reports are important supplements to the country’s existing quantitative monitoring of employment, job loss, and search behavior.
Racism and protest: The AVP provides in-depth evidence on how people are reacting to and thinking about systemic racism, ongoing protests of systemic racism, and recent events related to systemic racism. Because the AVP interviews take the form of an informal conversation, they can provide an unfiltered lens on attitudes that cannot, by contrast, be as easily ferreted out with surveys.
Class conflict: The pandemic has created a new “risk divide” between (a) face-to-face workers who bear disproportionate health and economic risks, and (b) remote workers who are better protected from those risks. In the United States, class-based conflict has long been tamped down, but the emergence of remote work during the crisis exposed inequalities in risk-bearing that may increase tensions between those on either side of the risk-bearing divide. The AVP is well positioned to explore whether new types of class conflict are surfacing (see the AVP report on noxious contracts).
Mental health: The crisis generated new types of mental health problems (e.g., “languishing”) and new types of coping as Americans dealt with job loss, domestic conflict, social isolation, and health problems and worries. The AVP protocol evaluates these worries by supplementing its battery of survey questions on mental health with targeted probes within the immersive interview itself (see the AVP reports on health).
The family: The family has faced unprecedented stress in the last year. The AVP data reveal how parents deal with stay-at-home children, whether middle-class parents have substantially more capacity than their counterparts to cope, and whether there’s much parental conflict over training responsibilities. Although surveys have been extremely useful in understanding these and other family dynamics, immersive interviewing may provide important supplementary evidence on everyday practices (see the AVP report on adolescents).
Immigration: The rich retrospective design of the AVP may also provide new evidence on how immigrants deal with discrimination, the threat of deportation, barriers to employment, noxious employment contracts, unequal access to health care, a hostile political environment, and much more. The AVP interviews have the potential to provide a new window into (a) the proximate effects of the pandemic and associated economic crisis on immigrants, and (b) the longer arc of immigrant lives during an especially fraught period in U.S. history.
Methodologies supported by the AVP
These and many other research topics can be explored with a wide range of research methodologies. It is likely that many proposals will rely on inductive analytic approaches (e.g., “grounded theory”) in which accounts and interpretations emerge from a close reading of the transcripts. We of course welcome proposals of this sort. The AVP can, however, also be analyzed with other types of methodological approaches that aren’t as conventionally associated with qualitative research, approaches that we’ll very briefly review here to emphasize our openness to a range of methodologies.
Deductive approaches: Although qualitative work is often inductive in approach, the AVP can also be used for deductive hypothesis-testing. It can be treated as a fact-checking resource to assess the hypotheses and interpretations coming out of past qualitative research, survey research, or journalism. Over the course of the pandemic, the country has relied heavily on qualitative journalism to provide real-time analysis, an invaluable function given the pressing need to understand what’s happening and to build policy around such results. As useful as qualitative journalism is, it is also important to validate the interpretations that it promotes, many of which seep permanently into our shared and implicit understandings of the world. The AVP can assist with this validation exercise by offering an opportunity to test such interpretations with representative samples of immersive interviews. It is equally important to assess hypotheses that come out of surveys, experiments, or conventional qualitative scholarship against the new type of evidence provided by the AVP. As noted above, qualitative analysis is often criticized for failing to cumulate, a charge that can be explicitly addressed by reexamining the conclusions about poverty, deprivation, work and employment, and family life coming out of past qualitative analyses.
Automated analysis: The foregoing analyses, although taking a deductive form, may nonetheless remain conventional in the sense that they rely on human interpretation of the transcripts. It is also possible, of course, to analyze the transcripts with natural language processing and related automated methods to monitor depression and mental health, social and political attitudes, trust and cynicism, the rhythm of everyday life, and all manner of related sentiments and feelings. The typical survey operates under the conceit that people have direct access to their own sentiments and attitudes, that such sentiments and attitudes can be surgically elicited by simply inquiring about them, and that people are willing to freely share such sentiments even when they can access them. The AVP may be used to assess whether sentiments and attitudes are sometimes better elicited indirectly in everyday conversations conducted in an atmosphere of trust (thereby exposing, for example, “shy voters” and socially unacceptable sentiments). Because the AVP survey includes many conventional attitude items, analysts can examine the extent to which sentiment analysis does or does not accord with those items and can explore the transcripts to understand the sources of discrepancies. If a strong research case can be made for releasing audio data (as well as transcripts), it may be possible to support research on audio-based measurement of sentiments and mental health (subject to additional confidentiality-protecting strictures).
Hidden crises: The country is facing profound challenges as prime-age employment continues to decline, highly concentrated forms of poverty spread, racism remains deeply entrenched, income inequality assumes all-time high levels, and distrust and disconnection continue to rise. These and other changes, some of which make this time period unusually volatile, create an especially pressing need for an early-warning system that allows us to detect emerging crises. If the AVP had been available in the past, the country might not have been taken by surprise by the prevalence of extreme poverty, the takeoff in disability claims, the rise of opiate addiction, or the growing disaffection within many sectors of the population. The AVP may allow us to detect new crises in the making that cannot be picked up with existing monitoring tools.
Community research: The AVP has relied on the participation of interviewees who generously shared their lives even when they were in the midst of difficult health, family, or work problems. Although our interviewees participated for a host of reasons, it was often because they believed, as the AVP team does, that doing so has the potential to help others. We are especially interested in innovative research proposals that use the AVP data to understand the problems that communities are facing and that do so by involving not just academics but also community members who can lead in interpreting the interviews, understanding the mechanisms in play, and developing policy that’s fine-tuned to addressing problems on the ground.
The foregoing list of methodological approaches and sensibilities is not intended to be exhaustive. We are open to all methodological approaches, conventional and unconventional alike, that hold promise in understanding ongoing and emerging forms of social behavior and developing evidence-based policy.
Submission instructions and timeline
To secure the interview and survey protocols and a sample interview, please submit the nondisclosure agreement here. After doing so, prospective contributors can apply by submitting a CV, an abstract of their study (up to two pages in length, single spaced), and supporting tables, figures, pictures, references, or other relevant material (up to two additional pages). These should be submitted by no later than 5 pm EST on 1/5/2022 to https://rsf.fluxx.io. (NOTE: If you wish to submit a proposal and do not yet have an account with RSF, it can take up to 48 hours to get credentials. So please start your application at least two days before the deadline.)
All submissions must be original work that has not been previously published in part or in full. Only abstracts submitted to https://rsf.fluxx.io will be considered. Each paper will receive a $1,000 honorarium when the issue is published. All questions regarding this issue should be directed to Suzanne Nichols, Director of Publications, at firstname.lastname@example.org, and not to the email addresses of the editors of the issue.
A conference will take place at the Russell Sage Foundation in New York City on December 9, 2022 (with a group dinner the night before). The selected contributors will gather for a one-day workshop to present draft papers (due a month prior to the conference on 11/11/22) and receive feedback from the other contributors and editors. Travel costs, food, and lodging for one author per paper will be covered by the foundation. Papers will be circulated before the conference. After the conference, the authors will submit their revised drafts by 2/22/2023. The papers will then be sent out to three additional scholars for formal peer review. Having received feedback from reviewers, the editors, and RSF, authors will revise their papers by 8/17/2023. The full and final issue will be published in spring 2024. Papers will be published open access on the RSF website as well as in several digital repositories, including JSTOR and UPCC/Muse.
Brayne, Sarah. 2020. Predict and Surveil: Data, Discretion, and the Future of Policing. Oxford University Press.
Clair, Matthew. 2020. Privilege and Punishment: How Race and Class Matter in Criminal Court Hardcover. Princeton University Press.
DeLuca, Stefanie, Susan Clampet-Lundquist and Kathryn Edin. 2016. Coming of Age in the Other America. Russell Sage Foundation.
Desmond, Matthew. 2016. Evicted: Poverty and Profit in the American City. Crown Publishers.
Duneier, Mitchell. 2011. “How Not to Lie with Ethnography.” Sociological Methodology 41 (1): 1-11. https://doi.org/10.1111/j.1467-9531.2011.01249.x
Edin, Kathryn J., and H. Luke Shaefer. 2015. $2.00 a Day: Living on Almost Nothing in America. Houghton Mifflin Harcourt.
Fields, Corey D. 2016. Black Elephants in the Room. University of California Press.
Hammersley, Martyn. 2013. What is Qualitative Research. Bloomsbury.
Hochschild, Arlie. 2018. Strangers in Their Own Land. The New Press.
Jerolmack, Colin, and Shamus Khan. 2014. “Talk is Cheap: Ethnography and the Attitudinal Fallacy.” Sociological Methods and Research. https://doi.org/10.1177%2F0049124114523396
Jiménez, Tomás R. 2017. The Other Side of Assimilation : How Immigrants Are Changing American Life. University of California Press.
King, Gary, Robert, O. Keohane, Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton University Press.
Lamont, Michele, and Patricia White. 2008. Workshop on Interdisciplinary Standards for Systematic Qualitative Research. National Science Foundation. https://www.nsf.gov/sbe/ses/soc/ISSQR_rpt.pdf
Lewis-Kraus, Gideon. 2016. “The Trials of Alice Goffman.” The New York Times Magazine (Jan. 12, 2016). https://www.nytimes.com/2016/01/17/magazine/the-trials-of-alice-goffman.html
Lubet, Steven. 2017. Interrogating Ethnography: Why Evidence Matters. Oxford University Press.
Nelson, Alondra. 2016. The Social Life of DNA. Penguin Random House.
Pattillo, Mary. 2013. Black Picket Fences (2nd Edition). Chicago: University of Chicago Press.
Schaefer, Stephan M. and Mats Alvesson. 2017. “Epistemic Attitudes and Source Critique in Qualitative Research.” Journal of Management Inquiry 29 (1): 33-45.
Sharkey, Patrick. 2018. Uneasy Peace. The Great Crime Decline, the Renewal of City Life, and the Next War on Violence. W.W. Norton.
Stuart, Forrest. 2020. Ballad of the Bullet: Gangs, Drill Music, and the Power of Online Infamy. Princeton University Press.
Watkins-Hayes, Celeste. 2019. Remaking a Life. University of California Press.
Western, Bruce. 2018. Homeward: Life in the Year After Prison. Russell Sage Foundation.