About The Ratings

About The Ratings

The Massachusetts Health Care Quality and Cost Council (HCQCC) created this website by taking ratings from other recognized organizations and by calculating some new ratings from our own Massachusetts health care database. We have tried to make these ratings easy for patients to understand, and also useful to doctors, hospitals, policy makers, and others.

How did you choose the quality and cost measures for the website?

The HCQCC has adopted these principles to guide our work on the ratings:

Principles for Selecting Quality Measures

The Council shall use the following principles to select quality measures for public reporting through its website and other media.

1. Wherever possible, measures should be drawn from nationally accepted standard measure sets.
2. The measure must reflect something broadly accepted as meaningful to providers or patients.
3. There must be empirical evidence that the measure provides stable and reliable information, and that the data sources and sample sizes are sufficient for accurate reporting at the level chosen.
4. There must be sufficient variability or insufficient performance on the measure to merit attention.
5. 1. There must be empirical evidence that the measured entity (clinician, site, group, institution) is associated with a significant amount of the variance in the measure. The measures offered for providers should, in totality, be representative of a significant proportion of their practices. OR
   2. The measure is important for patients or communities, even though a clear consensus on accountability for performance has not been determined.
6. Providers should be informed about the development and validation of the measures and given the opportunity to view their own performance, ideally for one measurement cycle, before the data are used for public reporting. Where feasible, providers should be permitted to verify data and offer corrections.

Principles for Selecting Cost Measures

1. The Council should publish a comprehensive and inclusive set of cost measures that reflect sufficient volume and relevance to be useful to an intended audience: consumers, employers, providers, insurers or policy-makers.
2. Cost measures should be accurate and reliable, and should be as timely as is feasible.
3. Cost measures should include the range of costs per procedure for an individual provider, as well as the most likely cost (median, mean or mode).
4. The Council should make efforts to display cost measures, to the extent possible, in ways that minimize harmful unintended consequences such as increased health care costs, collusion, introducing barriers to market entry, and other anti-competitive behavior.
5. The Council should display:
   cost and quality measures that are closely aligned on the same page; cost measures that do not closely align with quality measures on separate pages; and quality measures that do not closely align with cost measures on separate pages.

Here are some important things to know about our ratings.

Where are the data from?

The HCQCC gets some of its data from established health care organizations that perform data collection and analysis. While the Council makes every effort to keep this data up to date, more recent data may be available on the organizations’ websites. They include:

Centers for Medicare and Medicaid Services (CMS), for hospital quality ratings on heart attack, heart failure, pneumonia, surgical care, patient experience, readmissions and mortality.

The Leapfrog Group, for ratings of patient safety and quality for certain services (aortic valve replacement and weight loss surgery). Hospitals complete Leapfrog’s patient safety survey based on their assessment of their own practices. Leapfrog uses this survey information to assess the hospital’s patient safety practices.

The Massachusetts Division of Healthcare Finance and Policy, for a complete database of all hospital admissions in MA. Mortality measures of hip fracture, hip replacement and stroke are analyzed using Agency for Healthcare Research and Quality (AHRQ) methodology.

Massachusetts Department of Public Health, for Serious Reportable Events.

Massachusetts Department of Public Health’s Data Acquisition Center (Mass-DAC) , for angioplasty and bypass surgery death rate and volume.

Massachusetts Health Quality Partners , for Medical Group Quality Measures.

We also use our own database of healthcare services paid for by Massachusetts commercial health plans. This database includes the actual price paid by the health plan and the patient for the services provided.

How is cost calculated for hospitals?

The Health Care Quality and Cost Council calculated costs per case from our database of commercial health plan claims. Cost is based on the actual price that health plans pay hospitals. These are median dollar amounts meaning that half of the cases at this hospital cost more and half cost less. Costs are adjusted for severity of illness (how sick patients are).

Costs can vary a lot. Sometimes this happens even when patients are treated at the same hospital, by the same doctor, for the same condition. Your costs may be higher or lower depending on the specific services you receive.

To make fair comparisons among hospitals treating a variety of different patients, we adjust inpatient costs for how sick patients are, also called severity of illness. To do this, every patient claim in our database is rated for severity of illness on a scale of 1 (minor) to 4 (extreme). The claims are rated using APR-DRG (All Patient Refined-Diagnosis Related Groups) software by 3M Health Information Systems.

After each inpatient claim in the database is assigned a severity level, we calculate the average cost of caring for patients at each of the four severity levels across all hospitals in Massachusetts. Then, for each hospital, we calculate a predicted average cost for each severity level, based on the state-wide averages. We then compare the hospital’s actual cost to the predicted average cost, and adjust for the difference. View an example of this calculation, Cost Calculation. The website displays is the adjusted inpatient cost per case. This shows a hospital’s cost for treating a patient with average severity. Severity-adjusted inpatient cost allows consumers to compare costs at different hospitals, even though some hospitals treat more high-severity patients.

Cost Data Technical Information

Dates. The dates the cost data covers is noted in the source note for that data under the data on the website

Diagnostically Related Groups (DRGs). Inpatient claims are grouped using 3M’s All Patient Refined (APR-DRG) grouper software, version 24.

Cost. We show the cost of care as the “allowed amount” paid to the hospital. The allowed amount is equal to the amount paid by the health plan plus the amount due from the patient (such as a co-payment, deductible or co-insurance). Cost does not include payments for physician services. Claims with $0 payment were excluded from the analysis.

Transfer cases. Care for patients treated in one hospital and then transferred directly to another hospital is included in our data. Since the cost of care for one patient is split between two hospitals, we may underestimate the cost of care per case.

DNR (Do Not Resuscitate) cases. Some terminally ill patients may have a living will and may have “Do Not Resuscitate” orders so life-saving treatment will not be given to them if their heart stops or they stop breathing. From our database, we cannot tell which patients were “DNR” before their hospital stay. This may increase the apparent death (mortality) rate for some of the conditions.

Commercial health plan data. The HCQCC data covers about 2/3 of all privately insured Massachusetts residents. It includes all Massachusetts members of fully insured plans, plus all members in MA Group Insurance Commission (Commonwealth of Massachusetts employees, retirees and their beneficiaries) and Massachusetts residents enrolled in self-insured plans administered by Blue Cross and Blue Shield of Massachusetts.

Medicare and Medicaid. The Council’s dataset does not include information about patient enrolled in Medicare, Medicaid or other public payers. Medicare and Medicaid payments are not included in the median costs displayed on this website.

Sources for Quality Data and Cost Data differ. The cost data are from commercial health plans only, while quality data may come from all patients treated at a hospital, or just the Medicare (mostly over age 65) patients. Therefore, the cost and quality data shown do not represent the same patients.

Minimum Sample Size. We display cost data for hospitals that had 30 inpatient discharges or 30 outpatient visits for the condition or procedure. We display summary ratings using dollar signs ($$$) for conditions and procedures where at least 10 Massachusetts hospitals provided at least 30 discharges or 30 visits.

Hospital Systems. Some hospital systems provide hospital care at more than one campus. We show measures for each campus when we have them. If we don’t have information for each campus, we show information for the system. For example, we may have mortality information for each hospital campus, and cost information for the system as a whole.

How is Statistical Significance calculated?

We use significance tests to determine if a hospital's quality or cost is statistically Above Average Quality or Below Average Quality.  Each test is performed at the 0.05 significance level.  This means that there is only a 5% chance that the size of the difference could have occurred by chance.

Statistical Significance for Quality

Stroke, Hip Fracture, and Hip Replacement Mortality Rate Data 

An individual hospital's performance is measured by comparing the confidence interval (CI) for a hospital to the statewide mortality risk-adjusted rate (RAR). The hospital's performance is described by whether its risk-adjusted mortality rate statistically is not significantly different, statistically is significantly higher, or statistically is significantly lower than Massachusetts's risk-adjusted rate (using the 95% confidence interval).

If a hospital's CI intersects the statewide RAR, the hospital's risk-adjusted mortality rate statistically is neither significantly higher or lower than the statewide rate. For these hospitals, no differentiation can be made among hospitals' performance because each hospital's risk-adjusted mortality rate can fall anywhere within its confidence interval.

If a hospital's CI falls entirely below the statewide RAR, the hospital's risk-adjusted mortality statistically is significantly lower (better) than the statewide rate.

If a hospital's CI falls entirely above the statewide RAR, the hospital's risk-adjusted mortality statistically is significantly higher (worse) than the statewide rate.

Heart Attack, Heart Failure, and Pneumonia Process of Care Data 

The statewide measure is defined as the sum of all numerators across hospitals divided by the sum of all denominators across all hospitals.

For each hospital, upper and lower Confidence Intervals (CI) are computed:

CI = p + - (1.96*sqrt(p*(1-p)/n))

where p is the hospital's percent criteria as a proportion between 0 and 1, and n is the denominator.

If the statewide proportion falls in between lower and upper, then the hospital is not significantly different from the statewide average; if the statewide proportion is above "upper", then the hospital is significantly lower than average; and if the statewide proportion is lower than "lower" then the hospital is significantly higher than average.

Angioplasty (PCI) and Bypass Surgery (CABG)

We use statistical significance calculated by Mass-DAC www.massdac.org.

Medical Group Quality Measures

For each measure, the medical group’s rate is calculated based on a “numerator” that represents the number of people who actually received a recommended healthcare service divided by a “denominator” that represents the number of people who should have received that healthcare service. Ninety-five percent confidence intervals are calculated for each medical group’s rate. A Normal Approximation method of the binomial confidence interval is used and when the Normal Distribution is not a good estimator of the Binomial, the Exact Binomial Confidence Interval is calculated.

The benchmark used for comparison is the MHQP Massachusetts Statewide Rate, which is defined as the sum of all numerators across medical groups divided by the sum of all denominators across all medical groups. If the statewide rate falls in between lower and upper bounds of the confidence interval of the medical group’s rate, then the medical group is not statistically different from the statewide average; if the statewide rate is above upper bound of the confidence interval, then the medical group is statistically lower than statewide average; and if the statewide rate is lower than lower bound of the confidence interval then the medical group is statistically higher than statewide average.

 

Calculating Statistical Significance of Cost Measures

We use a “bootstrap” approach to determine statistical significance. For each condition or procedure, we construct “bootstrapped” 95% confidence intervals to evaluate the difference between each hospital’s median cost and the statewide median cost, using the following steps:

1. We construct a bootstrap sampleby randomly selecting with replacement claims or discharges costs from each hospital. The number of claims or discharges selected for each hospital is the hospital’s original sample size. 
2. Acting as if the bootstrapped sample were the actual data, we compute the median cost for each hospital and for the entire state.
3. We then calculate the difference between the bootstrapped median cost for the hospital and for the state. We repeat these three steps 1,000 times, resulting in 1,000 bootstrapped differences of medians for each hospital. 
4. Finally, for each hospital, we determine the 2.5 and 97.5 percentiles of the 1,000 differences. If both percentiles are above zero,then the measure meets the criteria for an “Above Median State Cost” rating.If both percentiles are below zero, then the measure meets the criteria for a “Below Median State Cost” label. All other measures are shown as “Not Different from Median State Cost.” 

For more information about this statistical method see Bootstrapping (statistics).


Variance between reported cost and dollar sign ratings

In some cases, the cost reported on the website appears to be inconsistent with the dollar sign rating. For example, the hospital specific cost for a procedure is the same as the statewide value, yet the dollar sign rating is “above” or “below” the median state cost.

The Council’s methodology for calculating ratings and reporting values is as follows:

• Cost measures are calculated on a hospital specific and statewide basis.
• These median values are used in tests of statistical significance and are run through a bootstrap methodology.
• The dollar sign rankings are based on statistical significance: above, not different, and below state medians.
• The dollar values reported on the website for both hospital specific and statewide costs have been rounded. If a cost is $100 or under, it is rounded up or down to the nearest $10. If a cost is between $101 and $1000, it is rounded up or down to the nearest $25. If a cost is between $1,001 and $10,000, it is rounded up or down to the nearest $100. And if a cost is over $10,000, it is rounded up or down to the nearest $500. For example, if the statewide value is $120 and a hospital’s particular cost is $130, both will be rounded to $125.
• The statistical significance test could return a result indicating that that the hospital is “no different than the median state cost” or the test could show that the hospital is “above the median state cost.” The statistical test result is affected by the number of observations (the sample size) and the range of variation within the sample.