A guest post by Katherine Swartz, Professor, Department of Health Policy & Management, Harvard T.H. Chan School of Public Health; firstname.lastname@example.org
The use of difference-in-differences analysis (“diff-in-diffs”) has grown over the past decade – in part because it seems conceptually and visually simple. On its face, it is a straightforward way to estimate the effect of a policy or intervention that affected some people and not others, so long as it’s possible to observe both groups before and after the event of interest. As such, the method seems ready-made for “natural” experiments, such as when some states instituted a policy change while other states did not. But diff-in-diffs analyses require strong assumptions. If the assumptions do not hold, researchers may believe they have estimated a causal effect of an intervention or event when the relationship is an association or is not there at all.
Application of diff-in-diffs by social scientists (especially economists and sociologists) and medical researchers has outpaced the dissemination of rigorous methods guidance. Recent developments related to diff-in-diffs include work on the assumptions needed to identify causality, model specification, statistical inference, and selection of comparison groups. Unfortunately, these methods papers are often in journals not widely read by applied researchers.
Before thinking diff-in-diffs is the appropriate method to analyze the effect of, say, a policy change affecting some groups of people or people in some states, check out the eminently user-friendly website: https://diff.healthpolicydatascience.org compiled by Bret Zeldow and Laura Hatfield, biostatisticians in the Department of Health Care Policy, Harvard Medical School. Faculty teaching methods courses or relying on empirical research for their courses should consider adding the site to their syllabi. The website discusses the fundamental basics (with dynamic illustrations of assumptions) and recent method developments for using diff-in-diffs. It also includes several interactive apps for readers to see the points being made – for example, time points needed to test parallel trends, and how correlation structures in longitudinal data affect the stability of trends over time.
The diff-in-diffs website is an evolving resource, developed by Bret Zeldow, Research Fellow in Statistics and Laura Hatfield, Associate Professor of Biostatistics – both in the Department of Health Care Policy at Harvard Medical School. Hatfield also is Co-Director of the Health Policy Data Science Lab.