Performance Measurement in Continuing Care The Hospital Report Balanced

In recent years health care has moved to be in step with a global movement towards greater transparency and accountability in both the public and private sectors. There is a growing level of expectation and participation of individuals in making decisions concerning their healthcare.

Scorecard for Complex Continuing Care

In some instances the expectation is for direct participation in decision making – as a consumer – for example in deciding about treatment options with their doctor. In relation to other less direct and immediate issues such as how hospitals and healthcare systems are operated, people as citizens expect that governments and public servants demonstrate evidence of prudent management of public resources. In the realm of long-term care (LTC) the mix of public and private payment, and the vulnerable nature of those cared for, evokes expectations from residents and their family in both domains – consumer and citizen. A strong and growing demand for information and openness to support decision-making and accountability is being felt in the LTC sector.

The Experience of Ontario Hospitals: Hospital Report In 1997 the Ontario Hospital Association (OHA) made a bold commitment to hospital accountability through public reporting of performance measures. Importantly for the credibility of such reports, the OHA contracted an independent research group at the University of Toronto to develop the performance measures and create the reports, since one of OHA's major functions is as a member-supported advocacy organization. In 1998 the first Ontario Hospital Report, focused on inpatient acute care, was released to the public. That report, which reported results at peer group and regional levels of aggregation, was followed the next year by the first major hospital-specific public reporting of acute care hospital performance indicators in Canada – Hospital Report '99.

In 2000, the Government of Ontario joined the Hospital Report as a funding partner and the scope of the report was expanded to include other inpatient hospital sectors – emergency department care, complex continuing care, mental health, and rehabilitation – along with special reports focused on cross-sectoral issues – women's health, population health, and nursing care.

The Balanced Scorecard

From the beginning of its development Hospital Report has had a two pronged objective – to provide comparative hospital performance information to hospitals and the public to provide accountability and to strengthen hospitals' performance improvement efforts. In order to adequately capture the complexity of healthcare the University of Toronto researchers selected the balanced scorecard (BSC) framework that had been developed in the USA by Harvard professors Kaplan and Norton, and popularized in the business world in the 1990's.

The BSC approach grew out of a realization that to understand and effectively manage businesses towards strategic ends required more than a simple focus on the financial bottom-line. Kaplan and Norton recognized that achieving strategic ends required positive impacts on a series of intermediate steps that contribute to the bottom line, including knowing and meeting customer needs, ensuring product quality and production efficiency, and sustaining growth and leadership through organizational learning.

Thus they developed an indicator monitoring framework that balanced four perspectives: financial, customer, core business, and learning and growth. Professors Baker and Pink, of University of Toronto's Department of Health Administration (now, Health Policy, Management and Evaluation) proposed a "translation" of the BSC model to fit the Canadian hospital context, recognizing that strategic and effective management of hospitals also requires a multiple perspective monitoring system (Table 1). This BSC framework was the basis for the Hospital Report. From the beginning the target audience for Hospital Report has been hospital managers and informed citizens. The goal has been – and remains – to facilitate effective hospital management while opening the "black box" to public scrutiny.

[Table 1 available in the print version of Stride]

Hospital Report for Complex Continuing Care

The development of Hospital Report for Complex Continuing Care (HRCCC) began in 2000. There are approximately 6,000 CCC beds distributed among some 140 hospitals in Ontario. The number of CCC beds at individual hospitals ranges from less than 10 to well over 400 – with 10% of the hospital sites accounting for approximately 50% of all patient days in any given fiscal year. These hospital-based CCC beds are designated by the Ontario Ministry of Health and Long-Term Care (MOHLTC) to provide care to persons with medically complex long-term or intensive shorter-term post-acute care needs. The CCC beds represent about 10% of all facility-based LTC beds in the province.

In the spirit of the intention of the Hospital Report – to enhance transparency and effectiveness in hospital management – the Hospital Report research collaborative is committed to 4 main principles:

• Consultation of end users in the development of the reports. To be an effective tool in the hands of hospital administrators; the indicators and reporting need to reflect their strategic information needs. To this end, Advisory Panels, consisting of 32 clinicians, managers and other end users were formed to provide vital input into the development of the indicators for the HRCCC BSC quadrants. The hospital staff panel members were selected for a balanced representation of expertise, and regional and hospital-type representation from over 80 individuals nominated by the hospitals.
• Indicators are selected based on their importance, relevance, and feasibility. Importance means that the issue is of common concern to a broad crosssection of hospitals that provide CCC. Relevance means that the indicator can be related with reasonable confidence to processes amenable to management by hospital staff. Feasibility refers to the capacity to reliably and validly measure the indicator in a reasonably timely way.
• Existing data sources are to be used wherever possible. This principle flows naturally from the preceding one and from the recognition that to be sustainable, an indicator monitoring system should not place excessive additional burden on hospital staff.
• The methodology for the development and calculation of the Hospital Report indicators are made available in the public domain. This principle is key to maintaining confidence in and the utility of the performance measures and supports ongoing dialogue between the researchers and users regarding the indicators and their refinement. The methods pertaining to the Hospital Report are described in the Reports, available from the researchers, and are posted on the Hospital Report website (www.hospitalreport.ca).

Indicators and Methods for the HRCCC BSC Quadrants

The indicators in each of the BSC quadrants of HRCCC are listed in Table 2. In the case of the Clinical quadrant, a source of data was available because information about all patients in CCC beds since 1996 has been submitted by hospitals to the Canadian Institute for Health Information (CIHI) quarterly, and hospitals are required to submit RAI-Minimum Data Set (RAI-MDS) assessments to CIHI quarterly for all patients with a stay of at least 14 days. Data for the 1999/2000 fiscal year was available for use in HRCCC 2001.

The role of the Clinical quadrant Advisory Panel will be described as an example of how the Advisory Panels were involved in development of the HRCCC. The Panel, consisting of nurses, physicians, therapy professionals and others were asked, in a survey, to independently score the importance and relevance of 32 potential clinical indicators derived from the research literature, that could be calculated based on the RAI-MDS data. The results of the survey showed that several indicators were strongly endorsed and others were strongly rejected. There remained a "middle band" of 15 indicators with equivocal support. The Panel was assembled for a one-day meeting to discuss those indicators, resulting in a semi-final list of indicators for further consideration by the research team. In the end, the 9 indicators shown in Table 2 were selected for the first year of reporting based on the Panel input and the desire to have a mix of process and outcome indicators. In order that the indicators would reflect care received in the CCC unit – not at the referral source – only RAI-MDS assessments completed at least 80 days after admission were included in indicator calculations. Due to the timing of RAI-MDS assessments and length of stay patterns in CCC, this meant that only just over 1/3 of CCC patients were represented in the indicators. However, those patients accounted for nearly 80% of all CCC patient days in the 1999/2000 fiscal year.

[Table 2 available in the print version of Stride]

In the case of the System Integration and Change (SIC) quadrant, there was no available data set to capture the needed information to operationalize indicators suggested through our Advisory Panel and by other means. Therefore, the research team developed a hospital survey questionnaire, with input from the SIC Panel, that was completed by staff at participating hospitals during the summer of 2001. Indicators for the Financial quadrant were developed in consultation with a panel comprised largely of hospital financial officers. The data for these indicators was available in the Ontario Hospital Reporting System statistical and financial data reported annually to the MOHLTC by hospitals. Patient Satisfaction measures were obtained by means of a standardized continuing care survey that had been used for several years in numerous Canadian LTC facilities. Since few CCC patients would be able to respond to a self-completed survey, patients were interviewed by trained, paid interviewers. Simultaneously, a similar survey was mailed to the patient's most significant visitor (usually a family member) to obtain their perspective.

Participation and Selected Results

One hundred and nine (of 140) hospitals with CCC beds participated in at least some elements of HRCCC. The participating hospitals delivered over 95% of all CCC patient days of care in the province. Participation – particularly among the smallest CCC units - was lower in the Patient Satisfaction quadrant, likely due to the cost involved in the interviews. In all, 1,370 patients and 1,520 family members responded to the survey, for overall response rates of 69% for patients (of those with cognitive ability to respond; 49% of all patients) and 55% for families among the participating hospitals.

A few key messages were apparent in considering together the results of the various quadrants and indicators.

• Hospitals have an opportunity and need to strengthen their patient-centred focus in CCC. This theme was suggested by results from the SIC, Clinical and Satisfaction quadrants. For example, both patients and families were significantly dissatisfied with their level of participation in decision-making regarding the patient's care and daily choices. A related indicator in the SIC quadrant showed that only just over 1/3 of hospitals ensured that patients and/or families were given printed information to help inform them of the full scope of their rights and opportunities to participate in their care. Finally, an indicator in the clinical quadrant showed that over 1/2 of the patients in CCC for at least 80 days participated in little or no recreational activity, possibly suggesting a need to delve further into serving the recreation needs of patients with more activity limitations.
• There was considerable variability among hospitals on several of the clinical outcome and process indicators – for example frequency of antipsychotic use - both across the province as a whole and among peer groups of hospital CCC programs categorized by size (total patient days in a year). This suggests an opportunity to investigate and address that portion of the variability that is due to unnecessary practice variation.
• Other cross-quadrant findings indicate that hospitals are using tools such as best practice guidelines or protocols for a few limited clinical issues and that there is a need to extend this practice to other important clinical issues. For example, over 25% of patients (with stays more than 80 days) experienced significant activitylimiting pain daily, yet only 1 in 3 hospitals reported routine use of a practice guideline or protocol for pain management.

Next Steps

Participating hospitals have committed to public, hospital-specific reporting for HRCCC in the Fall of 2003. A major effort in currently underway to review, refine, and risk adjust as necessary the HRCCC indicators so that inter-hospital comparisons can be made on a “level playing field”. Other related projects include:

• Data audits including a chart audit to evaluate which elements of the RAI-MDS are amenable to audit by chart review, and an audit of materials related to hospital responses on the SIC quadrant survey; and
• Research to establish the relationships between indicators in different BSC quadrants such as (a) the relationship between financial efficiency indicators and clinical indicators, and (b) clinical and demographic correlates of CCC patient satisfaction.

Due to global societal forces, the role and necessity of performance measurement in healthcare will only grow over the next decade and beyond. In facility-based long-term care the mixed nature of payment sources – public and private – creates an especially strong demand for accountability systems that address both the consumer and the citizen orientations. The balanced scorecard framework provides an ideal model for presenting a view of performance that takes into account a variety of perspectives – and is especially useful to healthcare administrators for tracking progress towards strategic goals.