About The Ratings
The Massachusetts Health Care Quality and Cost Council (HCQCC) created this website
by taking ratings from other recognized organizations and by calculating some new
ratings from our own Massachusetts health care database. We have tried to make these
ratings easy for patients to understand, and also useful to doctors, hospitals,
policy makers, and others.
How did you choose the quality and cost measures for the website?
The HCQCC has adopted these principles to guide our work on the ratings:
Principles for Selecting Quality Measures
The Council shall use the following principles to select quality measures for public
reporting through its website and other media.
- Wherever possible, measures should be drawn from nationally accepted standard measure
- The measure must reflect something broadly accepted as meaningful to providers or
- There must be empirical evidence that the measure provides stable and reliable information,
and that the data sources and sample sizes are sufficient for accurate reporting
at the level chosen.
- There must be sufficient variability or insufficient performance on the measure
to merit attention.
- There must be empirical evidence that the measured entity (clinician, site, group,
institution) is associated with a significant amount of the variance in the measure.
The measures offered for providers should, in totality, be representative of a significant
proportion of their practices. OR
- The measure is important for patients or communities, even though a clear consensus
on accountability for performance has not been determined.
- Providers should be informed about the development and validation of the measures
and given the opportunity to view their own performance, ideally for one measurement
cycle, before the data are used for public reporting. Where feasible, providers
should be permitted to verify data and offer corrections.
Principles for Selecting Cost Measures
- The Council should publish a comprehensive and inclusive set of cost measures that
reflect sufficient volume and relevance to be useful to an intended audience: consumers,
employers, providers, insurers or policy-makers.
- Cost measures should be accurate and reliable, and should be as timely as is feasible.
- Cost measures should include the range of costs per procedure for an individual
provider, as well as the most likely cost (median, mean or mode).
- The Council should make efforts to display cost measures, to the extent possible,
in ways that minimize harmful unintended consequences such as increased health care
costs, collusion, introducing barriers to market entry, and other anti-competitive
- The Council should display:
cost and quality measures that are closely aligned on the same page; cost measures
that do not closely align with quality measures on separate pages; and quality measures
that do not closely align with cost measures on separate pages.
Here are some important things to know about our ratings.
Where are the data from?
The HCQCC gets some of its data from established health care organizations that
perform data collection and analysis. While the Council makes every effort to keep
this data up to date, more recent data may be available on the organizations’ websites.
Centers for Medicare and Medicaid Services (CMS), for
hospital quality ratings on heart attack, heart failure, pneumonia, surgical care,
patient experience, readmissions and mortality.
The Leapfrog Group, for
ratings of patient safety and quality for certain services (aortic valve replacement
and weight loss surgery). Hospitals complete Leapfrog’s patient safety survey based
on their assessment of their own practices. Leapfrog uses this survey information
to assess the hospital’s patient safety practices.
The Massachusetts Division of Healthcare Finance and Policy,
for a complete database of all hospital admissions in MA. Mortality measures of hip fracture,
hip replacement and stroke are analyzed using Agency for Healthcare Research and Quality (AHRQ)
Massachusetts Department of Public
Health, for Serious Reportable Events.
Massachusetts Department of Public Health’s
Data Acquisition Center (Mass-DAC) , for angioplasty and bypass surgery
death rate and volume.
Massachusetts Health Quality Partners
, for Medical Group Quality Measures.
We also use our own database of healthcare services paid for by Massachusetts commercial
health plans. This database includes the actual price paid by the health plan and
the patient for the services provided.
How is cost calculated for hospitals?
The Health Care Quality and Cost Council calculated costs per case from our database
of commercial health plan claims. Cost is based on the actual price that health
plans pay hospitals. These are median dollar amounts meaning that half of the cases
at this hospital cost more and half cost less. Costs are adjusted for severity of
illness (how sick patients are).
Costs can vary a lot. Sometimes this happens even when patients are treated at the
same hospital, by the same doctor, for the same condition. Your costs may be higher
or lower depending on the specific services you receive.
To make fair comparisons among hospitals treating a variety of different patients,
we adjust inpatient costs for how sick patients are, also called severity of illness.
To do this, every patient claim in our database is rated for severity of illness
on a scale of 1 (minor) to 4 (extreme). The claims are rated using APR-DRG (All
Patient Refined-Diagnosis Related Groups) software by 3M Health Information Systems.
After each inpatient claim in the database is assigned a severity level, we calculate
the average cost of caring for patients at each of the four severity levels across
all hospitals in Massachusetts. Then, for each hospital, we calculate a predicted
average cost for each severity level, based on the state-wide averages. We then
compare the hospital’s actual cost to the predicted average cost, and adjust for
the difference. View an example of this calculation, Cost Calculation. The
website displays is the adjusted inpatient cost per case. This shows a hospital’s
cost for treating a patient with average severity. Severity-adjusted inpatient cost
allows consumers to compare costs at different hospitals, even though some hospitals
treat more high-severity patients.
Cost Data Technical Information
Dates. The dates the cost data covers is noted in the source note
for that data under the data on the website
Diagnostically Related Groups (DRGs). Inpatient claims are grouped
using 3M’s All Patient Refined (APR-DRG) grouper software, version 24.
Cost. We show the cost of care as the “allowed amount” paid to the
hospital. The allowed amount is equal to the amount paid by the health plan plus
the amount due from the patient (such as a co-payment, deductible or co-insurance).
Cost does not include payments for physician services. Claims with $0 payment were
excluded from the analysis.
Transfer cases. Care for patients treated in one hospital and then
transferred directly to another hospital is included in our data. Since the cost
of care for one patient is split between two hospitals, we may underestimate the
cost of care per case.
DNR (Do Not Resuscitate) cases. Some terminally ill patients may have
a living will and may have “Do Not Resuscitate” orders so life-saving treatment
will not be given to them if their heart stops or they stop breathing. From our
database, we cannot tell which patients were “DNR” before their hospital stay. This
may increase the apparent death (mortality) rate for some of the conditions.
Commercial health plan data. The HCQCC data covers about 2/3 of all
privately insured Massachusetts residents. It includes all Massachusetts members
of fully insured plans, plus all members in MA Group Insurance Commission (Commonwealth
of Massachusetts employees, retirees and their beneficiaries) and Massachusetts
residents enrolled in self-insured plans administered by Blue Cross and Blue Shield
Medicare and Medicaid. The Council’s dataset does not include information
about patient enrolled in Medicare, Medicaid or other public payers. Medicare and
Medicaid payments are not included in the median costs displayed on this website.
Sources for Quality Data and Cost Data differ. The cost data are from
commercial health plans only, while quality data may come from all patients treated
at a hospital, or just the Medicare (mostly over age 65) patients. Therefore, the
cost and quality data shown do not represent the same patients.
Minimum Sample Size. We display cost data for hospitals that had 30
inpatient discharges or 30 outpatient visits for the condition or procedure. We
display summary ratings using dollar signs ($$$) for conditions and procedures where
at least 10 Massachusetts hospitals provided at least 30 discharges or 30 visits.
Hospital Systems. Some hospital systems provide hospital care at more
than one campus. We show measures for each campus when we have them. If we don’t
have information for each campus, we show information for the system. For example,
we may have mortality information for each hospital campus, and cost information
for the system as a whole.
How is Statistical Significance calculated?
We use significance tests to determine if a hospital's quality or cost is statistically
Above Average Quality or Below Average Quality. Each test
is performed at the 0.05 significance level. This means that there is only
a 5% chance that the size of the difference could have occurred by chance.
Statistical Significance for Quality
Stroke, Hip Fracture, and Hip Replacement Mortality Rate Data
An individual hospital's performance is measured by comparing the confidence interval
(CI) for a hospital to the statewide mortality risk-adjusted rate (RAR). The hospital's
performance is described by whether its risk-adjusted mortality rate statistically
is not significantly different, statistically is significantly higher, or statistically
is significantly lower than Massachusetts's risk-adjusted rate (using the 95% confidence
If a hospital's CI intersects the statewide RAR, the hospital's risk-adjusted mortality
rate statistically is neither significantly higher or lower than the statewide rate.
For these hospitals, no differentiation can be made among hospitals' performance
because each hospital's risk-adjusted mortality rate can fall anywhere within its
If a hospital's CI falls entirely below the statewide RAR, the hospital's risk-adjusted
mortality statistically is significantly lower (better) than the statewide rate.
If a hospital's CI falls entirely above the statewide RAR, the hospital's risk-adjusted
mortality statistically is significantly higher (worse) than the statewide rate.
Heart Attack, Heart Failure, and Pneumonia Process of Care Data
The statewide measure is defined as the sum of all numerators across hospitals divided
by the sum of all denominators across all hospitals.
For each hospital, upper and lower Confidence Intervals (CI) are computed:
CI = p + - (1.96*sqrt(p*(1-p)/n))
where p is the hospital's percent criteria as a proportion between 0 and 1, and
n is the denominator.
If the statewide proportion falls in between lower and upper, then the hospital
is not significantly different from the statewide average; if the statewide proportion
is above "upper", then the hospital is significantly lower than average; and if
the statewide proportion is lower than "lower" then the hospital is significantly
higher than average.
Angioplasty (PCI) and Bypass Surgery (CABG)
We use statistical significance calculated by Mass-DAC www.massdac.org.
Medical Group Quality Measures
For each measure, the medical group’s rate is calculated based on a “numerator”
that represents the number of people who actually received a recommended
healthcare service divided by a “denominator” that represents the number of
people who should have received that healthcare service. Ninety-five percent
confidence intervals are calculated for each medical group’s rate. A Normal
Approximation method of the binomial confidence interval is used and when the Normal
Distribution is not a good estimator of the Binomial, the Exact Binomial Confidence Interval is calculated.
The benchmark used for comparison is the MHQP Massachusetts Statewide Rate, which
is defined as the sum of all numerators across medical groups divided by the sum
of all denominators across all medical groups. If the statewide rate falls in
between lower and upper bounds of the confidence interval of the medical group’s
rate, then the medical group is not statistically different from the statewide
average; if the statewide rate is above upper bound of the confidence interval,
then the medical group is statistically lower than statewide average; and if the
statewide rate is lower than lower bound of the confidence interval then the medical
group is statistically higher than statewide average.
Calculating Statistical Significance of Cost Measures
We use a “bootstrap” approach to determine statistical significance. For each condition or procedure, we construct “bootstrapped” 95% confidence intervals to evaluate the difference between each hospital’s median cost and the statewide median cost, using the following steps:
- We construct a bootstrap sampleby randomly selecting with
replacement claims or discharges costs from each hospital. The number of claims or discharges selected
for each hospital is the hospital’s original sample size.
- Acting as if the bootstrapped sample were the actual data, we compute the median cost for each hospital and for the entire state.
- We then calculate the difference between the bootstrapped median cost for the hospital and for the state.
We repeat these three steps 1,000 times, resulting in 1,000 bootstrapped differences of medians for each hospital.
- Finally, for each hospital, we determine the 2.5 and 97.5 percentiles of the 1,000 differences. If both percentiles are above zero,then the measure meets the criteria for an “Above Median State Cost” rating.If both percentiles are below zero, then the measure meets the criteria for a “Below Median State Cost” label. All other measures are shown as “Not Different from Median State Cost.”
For more information about this statistical method see Bootstrapping (statistics)
Variance between reported cost and dollar sign ratings
In some cases, the cost reported on the website appears to be inconsistent with the dollar sign rating. For example, the hospital specific cost for a procedure is the same as the statewide value, yet the dollar sign rating is “above” or “below” the median state cost.
The Council’s methodology for calculating ratings and reporting values is as follows:
- Cost measures are calculated on a hospital specific and statewide basis.
- These median values are used in tests of statistical significance and are run through a bootstrap methodology.
- The dollar sign rankings are based on statistical significance: above, not different, and below state medians.
- The dollar values reported on the website for both hospital specific and statewide costs have been rounded. If a cost is $100 or under, it is rounded up or down to the nearest $10. If a cost is between $101 and $1000, it is rounded up or down to the nearest $25. If a cost is between $1,001 and $10,000, it is rounded up or down to the nearest $100. And if a cost is over $10,000, it is rounded up or down to the nearest $500. For example, if the statewide value is $120 and a hospital’s particular cost is $130, both will be rounded to $125.
- The statistical significance test could return a result indicating that that the hospital is “no different than the median state cost” or the test could show that the hospital is “above the median state cost.” The statistical test result is affected by the number of observations (the sample size) and the range of variation within the sample.