The inherent bias in the Five-Star System
It's time to fix the skilled nursing survey process
The Five-Star System, which CMS has created to rate the performance of nursing facilities nationally, will only grow in importance as CMS plans to expand and integrate the Five Star System into elements of the reimbursement and further popularize these tools with both consumers and other players such as Medicare Advantage plans and other managed care insurers. Health systems are already beginning to use the system to differentiate between facilities as they strive to lower re-hospitalization rates or form networks for accountable care organizations. Yet the Five-Star System has a systematic and profound bias that favors smaller facilities and penalizes larger facilities.
The star system has three distinct components: survey results, outcomes measures and nursing staffing levels. The base of the Five-Star System is the survey process, where more and more widespread and serious deficiencies cited generally results in fewer stars (more stars are intended to reflect better facilities). Stars are then added and/or subtracted based on staffing and quality outcomes measures.
The quality outcomes measures reflect a subset of 10 of the 19 quality measures posted on Nursing Home Compare website of CMS. These 10 were chosen, according to CMS, because of the measures' “validity and reliability, the extent to which the measure is under the facility's control, statistical performance and importance.”
Figure 1 shows the average number of survey-process stars awarded based on deficiencies cited in annual surveys for facilities in Pennsylvania by size of facility and stars awarded based on the quality outcomes measures. The chart demonstrates that, on average, facilities achieve similar quality outcomes regardless of size (except that the very smallest facilities achieve somewhat lower quality outcomes measures than the rest). According to CMS data, there is virtually no correlation between the size of the facility and the quality outcomes measures. Yet, the smallest facilities are awarded a full star higher than the largest ones through the survey process. One full star is a dramatic and troubling difference. If the quality outcomes are meaningful, then the survey process must be inherently biased. Comparing survey results to quality outcomes measures leads to entirely different conclusions about facility quality. Providers don't expect a perfect system, but they should expect it to be unbiased, because a flawed process is not a sound foundation for a rating system.
To highlight the magnitude of the bias, Figure 2 shows the percentage of each 50-bed group that receives a 1- or 2-star rating and the percentage that receives either a 4- or 5-star rating. Overall, CMS' intent is to classify the top 10% of facilities as 5-star, the bottom 20% as 1-star, and evenly distribute the remaining facilities in the remaining categories. So, across all facilities in Pennsylvania, approximately 33% will receive 4- or 5-star ratings while 43% will be classified as 1- or 2-star. This is not the pattern observed in Figure 2, and again, note the decline in ratings for each successively larger group.
The only possible conclusion from the CMS data is that the survey system, and thus the Five Star System built on it, is profoundly and systematically biased for the benefit of smaller facilities.
So what causes this dramatic and compelling bias? It's because the star rating system uses the number of deficiencies cited in the annual survey as an absolute indicator instead of framing them as a “rate of deficiency” – a phenomenon I dub the “Absolute Effect”.
Deficiencies reflect fairly rare events that facilities try to avoid. Since most of what we do is human-based, errors and mistakes are inevitable. A 250-bed facility will have a lot more of everything than a 25-bed facility would – 10 times the residents, 10 times the staff, 10 times the charts, 10 times the medications, 10 times the physical plant, 10 times the meals, on and on. So it is inescapable that a 250-bed facility, all other things equal, will end up doing a lot more wrong things than a 25-bed facility. But, it will also do a lot more right things!
Since CMS does not convert the survey findings to some sort of rate of incident, the perspective is lost. The Absolute Effect would lead us to conclude that a rural, sparsely-populated county with only 2 occurrences of a very rare disease is doing better than a urban, densely-populated county that has 200 occurrences. But the correct analysis overlays population, and if the smaller county has 3 people and the larger county has 3 million, then clearly the rate of disease in the smaller county is 1,000 times that of the larger one – a dramatically different conclusion.
Perhaps oversimplifying, if a negative, deficiency-causing event occurs every 200 bed-years, then you might expect a 200-bed facility to see one event per year where a 20 bed facility might see that same event once in 10 years. The 200-bed facility will receive a deficiency every year where the small facility would receive none for 9 years and 1 in one year. So in one year, these facilities would be ranked the same, but in 9 of the 10 years, the smaller facility looks better. If these deficiency-causing events occurred frequently enough, all facilities would be on a level playing field.
CMS must address this indisputable bias to preserve the integrity of the Five Star System. It cannot allow the performance of larger facilities to be systematically downgraded. This data only relates to Pennsylvania facilities, but I see no reason to expect that other states show a different pattern. As the importance of this system increases, all stakeholders – providers, regulators, and especially prospective residents and their families – have to be able to rely on it to make good decisions.
Jeff A. Petty is the president and CEO of Wesley Enhanced Living.
 The database consists of the CMS current ratings as of August 9, 2012 for 655 facilities in Pennsylvania found at http://www.medicare.gov/Download/DownloaddbInterim.asp. The CMS database was adjusted to exclude the county owned and operated facilities and hospital-based/step-down units that could be identified and a handful of facilities where the bed count could not be determined – 54 facilities in total excluded. Also not reflected in Figures 1 and 2 were facilities larger than 300 beds, as the number of facilities in each 50-bed group was not large enough to be valid – although the survey results did generally get worse as size increased.