Hospital Star Ratings and Sociodemographics: A Scoring System in Need of Revision

Still in its infancy, the Hospital Compare overall hospital quality star rating program introduced by the Centers for Medicare & Medicaid Services (CMS) has generated intense industry debate. Individual health systems are microcosms of the challenges of ratings and measurement design. Sibley Memorial Hospital, a member of Johns Hopkins Medicine, is a well-run, 288-bed, community hospital located in a wealthy section of northwest District of Columbia with a five-star rating. In contrast, its academic partner, the Johns Hopkins Hospital, a 1,162-bed hospital with a century-long history of innovation situated in an impoverished Baltimore, Maryland, neighborhood, received a three-star rating.

Hospital ratings are the product of an industry in transition: As care delivery has shifted from an individual provider-driven industry to an increasingly scaled systems enterprise, policymakers implemented regulatory standards targeting quality measurement. Subsequent to the National Academy of Medicine’s 1999 report To Err is Human, policy efforts brought public reporting of quality ratings to multiple market segments, including dialysis facilities (2001), nursing homes (2003), Medicare Advantage plans (2007), and physicians (2015). The hospital industry was no exception, and in 2016—with much controversy¹—CMS launched the hospital star ratings program.

CMS Star Ratings for hospitals are based on seven measure groups: mortality, safety, readmission, patient experience, effectiveness, timeliness, and efficient use of medical imaging. Both industry and researchers have decried the challenges of star ratings, noting that hospitals with a narrower scope of services are more likely to receive higher ratings.² Measure groupings may be further flawed as shown by recent work demonstrating that larger, safety net, or academic hospitals, as well as hospitals offering transplant services, have higher readmission rates,³ which may be caused by differences in patient complexity. Other research has demonstrated that overall quality ratings inappropriately pool all hospitals together, when it may be fairer to initially categorize hospitals and then score them.⁴

It is within this maelstrom of debate that, in this month’s issue of the Journal of Hospital Medicine, Shi and colleagues explore the relationship between hospital star ratings and the socioeconomic features of the surrounding communities.⁵ Conducting their analysis by linking multiple reputable government and industry sources, Shi and colleagues found that counties with higher education attainment and a lower proportion of dual Medicare-Medicaid–eligible populations had higher hospital star ratings. Furthermore, a county’s minority population percentage negatively correlated with hospital ratings. Validating the experience of many rural hospital executives—who frequently experience financial challenges—Shi and colleagues noted that rural hospitals were less likely to receive five-star ratings.

Do these findings reflect a true disparity and lack of access to high-quality hospitals, or are they artifactual—secondary to a flawed construct of hospital quality measurement? Many lower-ranking hospitals are urban academic centers frequently providing services not offered at their five-star community counterparts, such as neurosurgery, comprehensive cancer care, and organ transplants, while simultaneously serving as safety net hospitals, research institutions, trauma centers, and national referral centers.

Sociodemographics factor significantly in self-care management for hospital aftercare. Health literacy, access to primary and behavioral healthcare, and transportation all affect star indicators. Recent work⁶ demonstrated that comprehensive investments in transitional care strategies and the social determinants of health were ineffective at reducing readmissions, which suggests that high readmission rates for hospitals in impoverished areas are not only common, but also may not accurately reflect hospital quality and local investment.

Patient experience is also complicating, with research demonstrating that patient perceptions vary significantly by education, age, primary language, ethnicity, and overall health. For example, one-third of average-ranked hospitals would have rankings vary by at least 18 percentile points when evaluated by Spanish-speaking patients. Star ratings fail to capture and communicate this granularity.⁷

More concerning is that star ratings inherently assume that hospital performance is being compared across the same tasks, regardless of patient characteristics, local resources, or the scope of services provided, the latter of which may vary between hospitals. For example, communication may differ in both complexity and time intensity: Explaining an antibiotic to the uncomplicated patient with pneumonia differs from prescribing an antibiotic to a patient who is legally blind from optic neuritis, walks with a cane because of multiple sclerosis, and has 24 other prescription medications. Similar challenges exist for differences in local neighborhood resources and for facilities with differing service scope.

Although one strategy to handle these “disparities” in star ratings might be to risk-adjust for social determinants of health, patients may be better served by first rethinking how star ratings are constructed. Clustering hospitals by scope of services provided and geographic region prior to determining star ratings would provide consumers with meaningful information by helping patients compare and make choices among either local or regional hospitals; national quality rankings are unhelpful for patients.

Arguably one of the most complex and person-dependent service enterprises, care delivery presents unique challenges for evaluation of customer experience and medical quality. Hospital star ratings are no exception: We must rethink their construction so they can be more meaningful for both patients and physicians.

The authors would like to acknowledge Daniel J Brotman, MD, for his editorial advice and input.

Dr Miller reported consulting for the Federal Trade Commission and serving as a member of the Centers for Medicare & Medicaid Services Medicare Evidence Development Coverage Advisory Committee. Drs Siddiqui and Deutschendorf have nothing to disclose.

Still in its infancy, the Hospital Compare overall hospital quality star rating program introduced by the Centers for Medicare & Medicaid Services (CMS) has generated intense industry debate. Individual health systems are microcosms of the challenges of ratings and measurement design. Sibley Memorial Hospital, a member of Johns Hopkins Medicine, is a well-run, 288-bed, community hospital located in a wealthy section of northwest District of Columbia with a five-star rating. In contrast, its academic partner, the Johns Hopkins Hospital, a 1,162-bed hospital with a century-long history of innovation situated in an impoverished Baltimore, Maryland, neighborhood, received a three-star rating.

Hospital ratings are the product of an industry in transition: As care delivery has shifted from an individual provider-driven industry to an increasingly scaled systems enterprise, policymakers implemented regulatory standards targeting quality measurement. Subsequent to the National Academy of Medicine’s 1999 report To Err is Human, policy efforts brought public reporting of quality ratings to multiple market segments, including dialysis facilities (2001), nursing homes (2003), Medicare Advantage plans (2007), and physicians (2015). The hospital industry was no exception, and in 2016—with much controversy¹—CMS launched the hospital star ratings program.

CMS Star Ratings for hospitals are based on seven measure groups: mortality, safety, readmission, patient experience, effectiveness, timeliness, and efficient use of medical imaging. Both industry and researchers have decried the challenges of star ratings, noting that hospitals with a narrower scope of services are more likely to receive higher ratings.² Measure groupings may be further flawed as shown by recent work demonstrating that larger, safety net, or academic hospitals, as well as hospitals offering transplant services, have higher readmission rates,³ which may be caused by differences in patient complexity. Other research has demonstrated that overall quality ratings inappropriately pool all hospitals together, when it may be fairer to initially categorize hospitals and then score them.⁴

It is within this maelstrom of debate that, in this month’s issue of the Journal of Hospital Medicine, Shi and colleagues explore the relationship between hospital star ratings and the socioeconomic features of the surrounding communities.⁵ Conducting their analysis by linking multiple reputable government and industry sources, Shi and colleagues found that counties with higher education attainment and a lower proportion of dual Medicare-Medicaid–eligible populations had higher hospital star ratings. Furthermore, a county’s minority population percentage negatively correlated with hospital ratings. Validating the experience of many rural hospital executives—who frequently experience financial challenges—Shi and colleagues noted that rural hospitals were less likely to receive five-star ratings.

Do these findings reflect a true disparity and lack of access to high-quality hospitals, or are they artifactual—secondary to a flawed construct of hospital quality measurement? Many lower-ranking hospitals are urban academic centers frequently providing services not offered at their five-star community counterparts, such as neurosurgery, comprehensive cancer care, and organ transplants, while simultaneously serving as safety net hospitals, research institutions, trauma centers, and national referral centers.

Sociodemographics factor significantly in self-care management for hospital aftercare. Health literacy, access to primary and behavioral healthcare, and transportation all affect star indicators. Recent work⁶ demonstrated that comprehensive investments in transitional care strategies and the social determinants of health were ineffective at reducing readmissions, which suggests that high readmission rates for hospitals in impoverished areas are not only common, but also may not accurately reflect hospital quality and local investment.

Patient experience is also complicating, with research demonstrating that patient perceptions vary significantly by education, age, primary language, ethnicity, and overall health. For example, one-third of average-ranked hospitals would have rankings vary by at least 18 percentile points when evaluated by Spanish-speaking patients. Star ratings fail to capture and communicate this granularity.⁷

More concerning is that star ratings inherently assume that hospital performance is being compared across the same tasks, regardless of patient characteristics, local resources, or the scope of services provided, the latter of which may vary between hospitals. For example, communication may differ in both complexity and time intensity: Explaining an antibiotic to the uncomplicated patient with pneumonia differs from prescribing an antibiotic to a patient who is legally blind from optic neuritis, walks with a cane because of multiple sclerosis, and has 24 other prescription medications. Similar challenges exist for differences in local neighborhood resources and for facilities with differing service scope.

Although one strategy to handle these “disparities” in star ratings might be to risk-adjust for social determinants of health, patients may be better served by first rethinking how star ratings are constructed. Clustering hospitals by scope of services provided and geographic region prior to determining star ratings would provide consumers with meaningful information by helping patients compare and make choices among either local or regional hospitals; national quality rankings are unhelpful for patients.

Arguably one of the most complex and person-dependent service enterprises, care delivery presents unique challenges for evaluation of customer experience and medical quality. Hospital star ratings are no exception: We must rethink their construction so they can be more meaningful for both patients and physicians.

The authors would like to acknowledge Daniel J Brotman, MD, for his editorial advice and input.

Dr Miller reported consulting for the Federal Trade Commission and serving as a member of the Centers for Medicare & Medicaid Services Medicare Evidence Development Coverage Advisory Committee. Drs Siddiqui and Deutschendorf have nothing to disclose.

Still in its infancy, the Hospital Compare overall hospital quality star rating program introduced by the Centers for Medicare & Medicaid Services (CMS) has generated intense industry debate. Individual health systems are microcosms of the challenges of ratings and measurement design. Sibley Memorial Hospital, a member of Johns Hopkins Medicine, is a well-run, 288-bed, community hospital located in a wealthy section of northwest District of Columbia with a five-star rating. In contrast, its academic partner, the Johns Hopkins Hospital, a 1,162-bed hospital with a century-long history of innovation situated in an impoverished Baltimore, Maryland, neighborhood, received a three-star rating.

Hospital ratings are the product of an industry in transition: As care delivery has shifted from an individual provider-driven industry to an increasingly scaled systems enterprise, policymakers implemented regulatory standards targeting quality measurement. Subsequent to the National Academy of Medicine’s 1999 report To Err is Human, policy efforts brought public reporting of quality ratings to multiple market segments, including dialysis facilities (2001), nursing homes (2003), Medicare Advantage plans (2007), and physicians (2015). The hospital industry was no exception, and in 2016—with much controversy¹—CMS launched the hospital star ratings program.

CMS Star Ratings for hospitals are based on seven measure groups: mortality, safety, readmission, patient experience, effectiveness, timeliness, and efficient use of medical imaging. Both industry and researchers have decried the challenges of star ratings, noting that hospitals with a narrower scope of services are more likely to receive higher ratings.² Measure groupings may be further flawed as shown by recent work demonstrating that larger, safety net, or academic hospitals, as well as hospitals offering transplant services, have higher readmission rates,³ which may be caused by differences in patient complexity. Other research has demonstrated that overall quality ratings inappropriately pool all hospitals together, when it may be fairer to initially categorize hospitals and then score them.⁴

It is within this maelstrom of debate that, in this month’s issue of the Journal of Hospital Medicine, Shi and colleagues explore the relationship between hospital star ratings and the socioeconomic features of the surrounding communities.⁵ Conducting their analysis by linking multiple reputable government and industry sources, Shi and colleagues found that counties with higher education attainment and a lower proportion of dual Medicare-Medicaid–eligible populations had higher hospital star ratings. Furthermore, a county’s minority population percentage negatively correlated with hospital ratings. Validating the experience of many rural hospital executives—who frequently experience financial challenges—Shi and colleagues noted that rural hospitals were less likely to receive five-star ratings.

Do these findings reflect a true disparity and lack of access to high-quality hospitals, or are they artifactual—secondary to a flawed construct of hospital quality measurement? Many lower-ranking hospitals are urban academic centers frequently providing services not offered at their five-star community counterparts, such as neurosurgery, comprehensive cancer care, and organ transplants, while simultaneously serving as safety net hospitals, research institutions, trauma centers, and national referral centers.

Sociodemographics factor significantly in self-care management for hospital aftercare. Health literacy, access to primary and behavioral healthcare, and transportation all affect star indicators. Recent work⁶ demonstrated that comprehensive investments in transitional care strategies and the social determinants of health were ineffective at reducing readmissions, which suggests that high readmission rates for hospitals in impoverished areas are not only common, but also may not accurately reflect hospital quality and local investment.

Patient experience is also complicating, with research demonstrating that patient perceptions vary significantly by education, age, primary language, ethnicity, and overall health. For example, one-third of average-ranked hospitals would have rankings vary by at least 18 percentile points when evaluated by Spanish-speaking patients. Star ratings fail to capture and communicate this granularity.⁷

More concerning is that star ratings inherently assume that hospital performance is being compared across the same tasks, regardless of patient characteristics, local resources, or the scope of services provided, the latter of which may vary between hospitals. For example, communication may differ in both complexity and time intensity: Explaining an antibiotic to the uncomplicated patient with pneumonia differs from prescribing an antibiotic to a patient who is legally blind from optic neuritis, walks with a cane because of multiple sclerosis, and has 24 other prescription medications. Similar challenges exist for differences in local neighborhood resources and for facilities with differing service scope.

Although one strategy to handle these “disparities” in star ratings might be to risk-adjust for social determinants of health, patients may be better served by first rethinking how star ratings are constructed. Clustering hospitals by scope of services provided and geographic region prior to determining star ratings would provide consumers with meaningful information by helping patients compare and make choices among either local or regional hospitals; national quality rankings are unhelpful for patients.

Arguably one of the most complex and person-dependent service enterprises, care delivery presents unique challenges for evaluation of customer experience and medical quality. Hospital star ratings are no exception: We must rethink their construction so they can be more meaningful for both patients and physicians.

The authors would like to acknowledge Daniel J Brotman, MD, for his editorial advice and input.

Dr Miller reported consulting for the Federal Trade Commission and serving as a member of the Centers for Medicare & Medicaid Services Medicare Evidence Development Coverage Advisory Committee. Drs Siddiqui and Deutschendorf have nothing to disclose.