April/May 2000

Outcomes Management Systems

By Deborah L. Wilkerson, MA

Find a vendor or do it yourself?

Should you use a data system or a data vendor to assist in meeting outcomes management standards? If the answer is yes, then what are the characteristics of an acceptable and appropriate system?

CARF...The Rehabilitation Accreditaton Commission, as a long-standing policy, does not endorse or specify any particular outcome measure, tool, or system. Moreover, CARF does not require the use of a vendor service. When one is used, that use alone is not sufficient to meet CARF standards. CARF does require measurement of certain domains at certain specified time frames, at individual and aggregate levels, attention to the validity and the reliability of the data system, use of comparison data, and communication with stakeholders.

Although there are many successful home-grown data systems, subscribing or participating with a pooled data system can be a cost-effective method for acquiring the data needed, obtaining comparative data, aggregating data, and generating key elements of a management report.

The reason for implementing a supporting information system is to maximize the quality of services and ultimately the benefits to the persons served. The intent of CARF standards is to ensure that information is collected and utilized in a way that is useful to the organization, contributes to management decision-making, supports quality improvement within the organization, and generates valid and reliable information for dissemination to the organization’s consumers and other stakeholders.

INDICATORS, MEASURES, AND DATA SETS

CARF IOM standards for medical rehabilitation require accredited programs to gather information at the level of the person served and at the level of the organization. This information includes demographics of the persons served, as well as information the organization will need in order to analyze outcome data for meaningful groups of persons. The standards also specify that three dimensions of the World Health Organization’s International Classification of Impairments, Disabilities, and Handicaps (ICIDH2; WHO 1997) be addressed. The 2000-2001 manual refers to the newer language of the ICIDH2, Impairments, Activities, and Participation, to reflect the organ, person, and societal levels of function. In addition, though CARF standards do not require data collection here, the ICIDH2 framework includes contextual or environmental factors—issues either external or internal to the individual that affect the potential for change in one or more of the other three domains. For example, an external environmental factor might be the architectural accessibility of the person’s home and an internal factor might be the person’s own motivation or values.

The trilogy of CARF domains should also be addressed somewhere in the organization’s outcomes management system, though many organizations will need to employ more than one approach to address all three. It may well be that no one vendor or data set can address all the data elements the organization needs. The trilogy is made up of effectiveness or results (outcomes) for persons served, efficiency or the relationship between the outcomes and the resources used, and satisfaction or the stakeholders’ opinions about the experience of care. Organizations will want to ensure that the domains, indicators, and data elements needed to adequately answer its quality improvement questions are included in the system used.

Questions to ask:

1. What descriptive data elements are included (eg, patient characteristics or referral sources)?

2. What information about processes of care is included (eg, treatments, therapists, or programs)?

3. What outcome measures and data elements are included?

4. What domains of outcome measurement (eg, impairment, activity [disability], participation [handicap]) are included in the tools and data elements? Are they relevant both to standards and to the organization’s population served?

5. Do the measures and data elements help us address effectiveness, efficiency, and satisfaction? If not, are there other systems that will or does the organization need to augment the system under consideration? (CARF requires measurement of all three domains.)

6. Are standardized measures included in the data set? Will we be able to report uniform information or can our consumers compare our organization to the same measures for other organizations?

7. What validity and reliability assessments have been conducted on the measures? What are the findings?

8. Are there accommodations for user-specified fields (organization-developed data elements)?

DATA COLLECTION AND SUBMISSION

CARF standards require that information be collected at the level of the person served and at the level of the organization. Moreover, the standards ask organizations to address the accuracy, completeness, validity, and reliability of the data, as well as to make comparisons of their own performance relative to past trends or to external data from the literature or pooled data. Questions to ask:

1. Is there a pooled database to which users submit data?

2. What is the data collection format (forms, software)? Is it cost-effective and feasible for your kind of organization?

3. How are data transmitted to the pooled database (paper, fax, electronically, over the Internet)?

4. What is the expected frequency of data transmission?

5. What are the specifications for inclusion of cases in the pooled database?

6. How much time is necessary to complete data elements (measures and other data items)?

7. Are there certain specific individuals identified who must complete data for submission in order to conform to the data system specifications (person served, family, or clinicians, and which ones)? If so, can your organization accommodate these requirements?

8. Are the protocols for data collection and submission consistent with our organization’s structure, process, and resources?

ANALYSIS

CARF standards require that outcome goals be a part of developing predicted outcomes at the level of the individual person served. They also require that person-level data be aggregated into meaningful groups for analysis. Key to using the outcome and other performance information is analyzing data in ways meaningful to the organization and its mission.

Some of the most frequently cited standards on CARF surveys relate to the use of outcome and other performance data in management decision-making. Using data in a practical way for quality improvement and planning means that the analysis must fit into the organization’s own template for decision-making.

Questions to ask:

1. Does any data analysis provided by the service offer information at the individual level of analysis (ie, for clinical decision- making, prediction of outcomes for person served)?

2. If so, in what format is individual-level information provided? Is this format going to be useful to the team in our organization? Is it understandable to consumers if it is appropriate to share this information with them, their families, and/or payors?

3. Does any of the data analysis provided offer information at the impairment group, program, or organization levels of analysis? In other words, on what variables is the data aggregated for presentation and comparison? Are these variables the ones that are important to our organization?

4. Is the analysis conducted on groups of relatively homogeneous persons? Are those groupings meaningful for the population we serve?

5. Is the analysis based on sound scientific practice? Are appropriate statistics used? Are any models for prediction or comparative statistics done appropriately?

6. Will the analyses available meet the organization’s need for information for program management and decision-making?

REPORTING Using data for decision-making requires the sharing of information with people throughout the organization. Questions to ask:

1. What reports are provided from the system?

2. Is the report content relevant to your organization and useful in decision-making?

3. Are the population breakdowns in the report appropriate and consistent with the case mix of your program?

4. What is the turnaround time between data submission and receiving a report?

5. Are special reports available? How flexible is the content? What is the cost? Process? Timing?

6. Can organizations produce their own or tailor-made reports using the system?

7. Can organizations receive their own data compiled in a database format for manipulation and analysis? In other words, is a raw data file created for the organization’s own use, for example, with a statistical analysis package, or to link with other data files maintained by the organization?

8. Is the confidentiality of individuals and provider organizations sufficiently protected? Will the organization need to transform identification codes to comply with the vendor’s security contract? If so, does the organization have the capacity to make these transformations?

9. Will pooled data be used for research or publicly available comparative information (ie, in published aggregate reports)? If so, are you comfortable with the uses anticipated and the mechanisms for seeking approval and ensuring confidentiality? If not, are you comfortable with limitations on the uses of the database to which you are contributing?

COMPARATIVE AND PREDICTIVE INFORMATION

Finally, CARF standards ask organizations to compare their performance to past trends within the organization and, if information is available, to the performance of external groups of organizations. While the competitiveness in the rehabilitation industry has tended to discourage the public availability of data for benchmarking, there are many organizations that are interested in sharing data on a reasonable basis, either through formal pooled systems or through voluntary collaboration. Questions to ask:

1. Who participates in the comparison base for any pooled database?

2. What information do subscribers receive to describe the comparison base (eg, types, sizes, and locations of providers)?

3. What data elements are included in the comparison summaries?

LOOKING FOR EXPERT ADVICE? Experts here are available to answer all your questions! Please contact us for more information about this feature, or to become an expert.

MEDIA CENTER Interactive Media  Archives  · December 2008  · November 2008  · October 2008  · August/September 2008  · Fall 2008 Product Directory  · July 2008  · June 2008  · May 2008  · April 2008  · March 2008  · All Archives  Newsletter  · Rehab Today Resources Classifieds Calendar Consumer Resources Media Kit Advertiser Index EAB Reprints Submit an Article