How Survey Non-Responses Affect U.S. Poverty Rates and Understandings of Inequality

November 20, 2015

This feature is part of an ongoing RSF blog series, Work in Progress, which highlights some of the research of our current class of Visiting Scholars.

This fall, the U.S. Census Bureau reported the official national poverty rate as 14.8%, a number virtually unchanged from the year prior. Many poverty scholars, including RSF president Sheldon Danziger, have long debated the accuracy of this official measure, pointing out that it does not take into account non-cash benefits such as food stamps and housing subsidies, and therefore fails to reflect the importance of the social safety net for low-income families.

Current RSF Visiting Scholar James Ziliak (University of Kentucky) has identified yet another factor that affects the Census Bureau’s official poverty measure, which is calculated based on responses to earnings questions on the Current Population Survey Annual Social and Economic Supplement (ASEC). In his ongoing research, Ziliak has observed a non-response rate to ASEC earnings questions of over 30%. These missing figures present a significant obstacle to our understanding of poverty and inequality—if, for example, a significant percentage of ASEC non-respondents are low-income, the national poverty rate would be much higher than the Census Bureau’s current estimate.

While the Census has in place a process for accounting for ASEC non-responses, Ziliak has also identified shortcomings with their approach. To obtain more accurate income data for non-respondents, Ziliak instead links the Current Population Survey to tax records. In a new interview with the Foundation, Ziliak discussed his work and its ramifications for the way we measure and understand poverty.

Q. Your research at the Foundation examines the rise of non-responses to earnings questions in the Current Population Survey and its effect on our understanding of poverty in the US. How does the Census Bureau currently account for non-respondents? What are the shortcomings of their approach?

Ziliak: If a survey respondent fails to provide their earnings to the Census field representative then the Census uses what is known as a "hot deck" procedure to impute an earnings value to that individual. Specifically they use twelve different categories of socioeconomic characteristics to find a potential "donor" to the individual. For illustrative purposes suppose that the match is based solely on education, gender, and age and the respondent who does not provide earnings is a 45 year old male high school graduate. Census would then randomly match this person to another 45 year old male high school graduate who actually reported their earnings.

Currently just over 20 percent of respondents to the CPS Annual Social and Economic Supplement do not provide their earnings. Another 10 percent do not any answer any question on the Supplement, raising the total nonresponse rate for earnings to just over 30 percent. This approach as the advantage of being relatively transparent and straightforward to implement, and does a pretty good job on average. However, a key shortcoming is that it assumes that the earnings are missing at random, and as discussed below, this does not seem to hold among low- and high-earnings individuals, and thus leads to bias in measures of poverty and inequality.

Q. By linking CPS data to tax records (Social Security Detailed Earnings Records), you've found evidence that non-responses to earnings questions in the CPS are not "missing at random." Can you explain what this means and how it affects our current measures of poverty?

The concept of missing at random means that once we account for confounding factors that are observed and measured (e.g. a person's age, race, gender, marital status, etc.), the fact that earnings are missing is independent of the outcome we are trying to measure such as the poverty rate or the level of inequality. In other words, there should be no bias in our government statistics. This is the assumption adopted by the Census in the hot deck procedure for the measurement of poverty and inequality, and other statistics.

Based on our analysis of linked CPS data to tax records this assumption seems to hold over much of the earnings distribution, except in the far left tail (i.e. the bottom 10-20%) and in the far right tail (i.e. the top 5%). The earnings data at these are parts of the distribution appear to be missing for reasons beyond the standard demographic characteristics measured in the CPS data, and thus in the tails of the distribution earnings are not missing at random. This means that estimates of poverty and inequality are biased.

Our estimates show that there are too few low earners and too few high earners responding to the earnings questions in the CPS. This results in an undercount of the rate of poverty and of inequality. For example, we find that the official poverty rate is too low by about 1 percentage point (about 8-9% on the baseline poverty rate of 12% during our sample period). This results in an undercount of the number of poor persons by about 3 million in a typical year. We also find that the share of earnings going to the richest top 1% of the distribution is biased downward by about 20%.

Q. How might your findings inform policy decisions related to poverty, income inequality, and the social safety net? How might population surveys be changed to reduce the number of non-responses in the future?

Because the poverty rate is presently used to allocate billions of federal dollars to the 50 states and District of Columbia across scores of programs, ensuing that our measures of poverty and inequality are as accurate as possible is important not only for that we have an accurate portrait of the financial well-being of Americans, but also that we are allocating scarce federal resources to those in greatest need. Thus, seeking out ways to improve survey response on important, but sensitive topics like earnings, is crucial.

Earnings nonresponse in household surveys such as the Panel Study of Income Dynamics and the Survey of Income and Program Participation is much lower than in the CPS. One possible reason is that the primary mission of the CPS is to collect monthly employment statistics, and not as a survey of income. However, because of the importance of CPS income to policy and research, our political leaders could begin by investing more resources in the collection of the income supplement so that Census could adopt some best practices from the other household surveys on eliciting earnings. In addition, our leaders could use moral suasion to convince residents of our country that there is a civic duty to participate in the survey. Too often the political undertones in recent years have been in the opposite direction, which in turn can have the perverse result of poor data quality leading to ineffective policy and thus more wasteful spending. High quality data is a necessary ingredient for effective evidence-based policies.

RSF

RSF: The Russell Sage Foundation Journal of the Social Sciences is a peer-reviewed, open-access journal of original empirical research articles by both established and emerging scholars.

Grants

The Russell Sage Foundation offers grants and positions in our Visiting Scholars program for research.

Newsletter

Join our mailing list for email updates.