Hospital Rankings Hold Up

Some basic metrics do effectively diagnose care quality

Published: Wednesday, February 19, 2020 - 13:01

Given the complexities of healthcare, do basic statistics used to rank hospitals really work well? A study co-authored by MIT economists indicates that some fundamental metrics do, in fact, provide real insight about hospital quality.

“The results suggest a substantial improvement in health if you go to a hospital where the quality scores are higher,” says Joseph Doyle, an MIT economist and co-author of a new paper detailing the study’s results.

The study was designed to work around a difficult problem in evaluating hospital quality: Some high-performing hospitals may receive an above-average number of very sick patients. Accepting those difficult cases could, on the surface, worsen the aggregate outcomes of a given hospital’s patients and make such hospitals seem less effective than they are.

However, the scholars found a way to study equivalent pools of patients, thus allowing them to judge the hospitals in level terms. Overall, the study shows, when patient sickness levels are accounted for, hospitals that score well on quality measures have 30-day readmission rates that are 15 percent lower than a set of lesser-rated hospitals, and 30-day mortality rates that are 17 percent lower.

“It wasn’t clear going in whether these quality measures do a good job of sorting hospitals out,” Doyle adds. “These results suggest that they have predictive power.”

The paper, “Evaluating Measures of Hospital Quality: Evidence from Hospital Referral Patterns,” was written by Doyle, the Erwin H. Schell Professor of Management and Applied Economics at the MIT Sloan School of Management; John Graves, an assistant professor in the Department of Health Policy at Vanderbilt University; and Jonathan Gruber, the Ford Professor of Economics at MIT. It appears in the latest issue of the Review of Economics and Statistics.

Randomized evaluations

To conduct the study, the researchers used a method that eliminates the issue of studying a skewed sample of admissions. They studied areas across the country where dispatchers’ calls are assigned randomly to different ambulance companies. Those ambulance companies tend to deliver patients to particular hospitals. Thus, otherwise similar groups of patients are admitted to different hospitals in what is essentially a random pattern; this allows outcomes to be compared among hospitals.

The patient data came primarily from Medicare claims made across the country during the period 2008–2012 and covered more than 170,000 hospital admissions for patients who had just suffered a health event requiring “nondiscretionary” hospital admission. The patients also fit some basic criteria, such as not having previously been admitted recently for the same condition.

In addition to analyzing 30-day readmission and mortality rates, the researchers looked at patient satisfaction levels. All these criteria, and more, are commonly used in hospital assessments.

The researchers also found a 37-percent difference in one-year mortality, among highly rated and lower-rated hospitals.

“I thought our results were reasonable,” says Doyle. “They’re not too big to be believed, but they suggest a substantial improvement in health if you go to a hospital where the quality scores are much higher.”

As the authors note in the paper, the subject is topical in the health policy world. Some lawmakers and experts want the hospital payment system to evolve in the direction of reimbursement for quality and outcomes, rather than treatment. As such, it is important to be able to tell if those quality measures are sturdy.

“There’s been a lot of interest in whether these quality measures are informative or not, because there is a shift away from paying for the quantity of care provided to the quality of care provided,” Doyle says. “Most of the policymakers I’ve talked to want to use these quality measures.”

Management matters

Further research will be needed to help illuminate issues surrounding hospital quality in further depth. For instance, the current study is more focused on emergency care and not on care for chronic conditions; Doyle says that analysis of chronic care is “a fascinating question” that merits further investigation.

Doyle also acknowledges the need for further study to explain why certain hospitals fare better than others on basic quality measures. He notes that some were historically quicker than others to adopt what are now almost universal practices—the allotment of blood-thinning drugs to heart patients, for instance—and suggests the rate of adoption of new practices is an important factor in this area.

“Coming from a management school, we see that a lot of the variation in outcomes stems in large part from differences in management,” Doyle says. “Do you have the right procedures in places so that it’s easy for providers to do what the guidelines suggest? Improving management could yield big improvements in patient health.”

The research was supported by the National Institutes of Health.

First published Jan. 30, 2020, in MIT News.

About The Author

Peter Dizikes

Peter Dizikes writes for MIT News and MIT Technology Review at the Massachusetts Institute of Technology (MIT). Dizikes writing covers social sciences, humanities, and business, with many stories involving the history of science and science policy. Dizikes has reported and written articles for The New York Times, The Boston Globe, The Washington Post, Salon, Slate, Seed, Nature online, and other publications. He was an editor/writer for ABCNews.com and for Time. He frequently wrote features on education and also produced articles for university publications. Special projects included writing an educator’s guide to evolution, accompanying NOVA’s acclaimed documentary, Judgment Day.