International Journal for Quality in Health Care 15:79-085 (2003)
© 2003 International Society for Quality in Health Care
Methodology Matters |
Critical appraisal of clinical performance measures in Germany
1Public Health Program, School of Medicine, University of Duesseldorf
2Institute for Medical Information Processing, University of Tuebingen
3German Agency for Quality in Medicine, Cologne, Germany
Background. One of the tasks of the Federal Co-ordinating Committee, as part of the self-governing health care system in Germany, is to develop clinical performance measures (CPMs). As these measures generally exert a strong impact on health care delivery, their methodological quality should meet the highest standards.
Objective. To develop, for use in the German health care system, methodological quality requirements for CPMs and performance assessment programs.
Materials and Methods. We assembled and reviewed internationally available materials on CPMs. From these we prepared lists of requirements for appraising the methodological quality of CPMs and CPM programs.
Products. Lists of criteria for use in appraising the appropriateness of the development, content, format and application of CPMs and their value to the health care system for performance measurement; a checklist for assessing the methodological quality of CPMs/CPM programs.
Conclusion. The methods applied and the products from this project offer a useful framework for prospective CPM/CPM program developers and set a high threshold for the development of methodologically rigorous clinical performance measurement.
Keywords: clinical performance measure, Germany, quality indicator, quality management
An objective assessment of the quality of medical care requires a commonly accepted concept of the term ‘quality’, for example in the form of evidence-based nominal performance values laid down in clinical guidelines. Actual medical care delivery can then be compared with these nominal performance values. To decide whether nominal and actual performance values match or differ, it is common practice to formulate clinical performance measures (CPMs). (For definitions of this and other special terms used in this paper, see the Appendix.) Any discrepancies between the two values may serve as a good starting point for setting up quality improvement activities.
It appears it is exactly this kind of quality assessment that the German legislature intended to implement in Germany’s health care system when passing the Health Care Reform Act of 2000. Article 137e of the ‘Fifth Social Code Book’ provides that the Federal Co-ordinating Committee—a joint committee of both the Physicians and Sickness Funds Committee (ambulatory care) and the Hospitals Committee (in-patient care) representing all medical care providers and purchasers at the federal level—should determine, on the basis of evidence-based clinical guidelines, ‘criteria’ for assessing the appropriate and efficient management of at least 10 diseases per year. Diseases should be selected according to indications that diagnostic or therapeutic approaches to these diseases are underused, overused or misused and that eliminating these deficiencies could have a strong impact on morbidity and mortality in the population [1].
In simple terms, one might say that the outstanding importance of CPMs is due to the widely acknowledged fact that ‘what gets measured gets done’ [2]. The effects of these measures on the quality of care, however, need not necessarily be positive. On the one hand, official application of particular CPMs in routine care might focus on improving only those aspects that get measured and thus cause a lack of resources that would be required for equally important aspects of care elsewhere. On the other hand, methodologically unsound CPMs can erroneously indicate poor quality of care. Utilizing unsound CPMs could eventually lead to deterioration of the quality of medical care if resources were being used up or health care providers excluded who would otherwise be necessary to maintain high quality of care. For this reason, only rigorous CPMs should be employed to assess the quality of medical care.
So far in Germany no criteria have been available to critically appraise the methodological quality of CPMs. Basing the design of our project on accounts of the experiences with clinical guideline development where the publication of quality criteria did actually lead to better guidelines [3,4], we took it as our objective to promote the development of rigorous high-quality CPMs. We compiled sets of criteria for CPM selection, and formulated methodological quality requirements for individual CPMs and CPM programs. Finally, we developed a checklist as a tool to critically appraise the methodological quality of CPMs.
Materials and methods
We conducted a search for internationally available lists of CPMs and screened them to identify those that proposed quality requirements for CPMs. Our search strategy was to identify English- and German-speaking organizations generally known to be involved in quality assessment in health care using the keywords: quality indicator, CPM, quality requirement, quality specification, methodological quality, methods, criteria, assessment. The international sources we consulted that proved relevant for this task are listed in Table 1. We then assembled these materials and drew on them to formulate firstly quality requirements for CPM programs (Table 2) and for individual CPMs (Table 3), secondly quality specifications for CPMs and CPM programs (Table 4) (for definitions see Appendix), and thirdly the checklist for the critical appraisal of CPMs presented in Tables 5–7
. The process of compilation comprised mainly the sorting out of all methodological quality criteria for CPMs mentioned, striking out duplicates, and classifying the criteria by principal category. To formulate the major categories of the checklist we drew especially on the established German Checklist for Appraising the Quality of Clinical Guidelines [4].
|
|
|
|
|
|
|
Requirements for clinical performance measures
Before developing CPMs it is important to select carefully those aspects of health care that are worthy of researchers’ effort in undertaking the laborious CPM development process. Several criteria for selecting aspects of care suitable for quality assessment have been proposed in the literature [5–8]. These include procedures or processes with the following characteristics.
- Those with a high frequency of occurrence.
- Those with high risk for patients.
- Problem-prone procedures.
- Those suspected of overuse, underuse or misuse.
- Those with proven inter-provider variability.
- Processes of care that have recently undergone major changes.
- Procedures with a strong financial impact.
- Processes considered to be measurable or adaptable.
- Service procedures provided by caregivers favorably disposed to system changes.
- Those that have the potential to improve health care delivery and outcomes.
- Those that have high relevance to consumer interests.
- Those that might be used to improve decision-making processes (for or against specific organizations providing medical care).
By drawing on the sources mentioned above (see Table 1) and by using this list of characteristics to assess a selected number of aspects of care we were able to assemble a list of quality requirements for use in the development of programs to assess quality of care by means of CPMs (Table 2).
Quality requirements for individual CPMs are listed in Table 3 and quality specifications for CPMs/CPM programs in Table 4.
Checklist for appraising the quality of clinical performance measures
The questions in the checklists listed in Tables 5–7
were derived from the quality requirements and specifications for choosing and specifying the aspects of medical care to be measured listed in Tables 2–4. In addition, we included those questions from the German Checklist for Appraising the Quality of Clinical Guidelines [4] that we found useful for formulating CPMs and that had been helpful in evaluating the methodological quality of CPMs in surgical gynecology [9]. The checklist contains questions concerning development, content, format and application of CPMs. A total of 39 questions promote comprehensive documentation and appraisal of the quality of CPMs (see Tables 5–7).
Conclusion
Given all the criteria established above, both quality requirements presented for CPMs and CPM programs and the checklist derived to document and assess their methodological quality establish a maximum requirement for those developing CPMs and it is unlikely that all these quality requirements can be fulfilled at once. Existing CPMs/CPM programs do not appear to have reached such a high-quality level yet [10]. As a framework for prospective CPM/CPM program developers, though, the quality requirements described above offer the possibility of developing methodologically more rigorous CPMs/CPM programs and a more objective appraisal of the quality of medical care.
Appendix
Definitions
We define here a number of terms relating to quality assessment of medical care as they are used in this paper. Most definitions were derived from the following sources [7,11,12,13,14].
Aspects of health care are structures, processes or outcomes of medical care that are considered suitable for assessment in the context of quality management in health care.
Clinical performance measures (CPMs) are algorithms or rules of procedure for measuring aspects of health care service provision (e.g. structures, processes or outcomes) for the purpose of assessing their quality and the need for quality improvement.
Clinical performance measure programs are programs that use sets of CPMs to assess the quality of medical care delivered by different health care providers. The results of the assessments are used for comparative performance reporting. In Germany such programs are run jointly by the Federal associations of health care providers, the German Hospital Association, and the Federal Associations of Sickness Funds. Health care providers showing poor results may have their licence to practice revoked.
Performance thresholds are specific levels on a scale of values that mark the points at which the performance of a health care activity as measured by a quality indicator is assigned a rating such as ‘Good’, ‘Satisfactory’, ‘Poor’, etc.
(Quality) indicators—an alternate term for CPMs—are applied to aspects of care provision to evaluate the quality of structures, processes, or outcomes of medical care.
Quality requirements for CPM are needs or expectations that are stated, generally implied or obligatory (e.g. as part of the procedural rules of the Federal Co-ordinating Committee) and must be taken under consideration and addressed when selecting performance measures.
Quality specifications are data elements pertaining to the resources and materials used, contributors engaged, groups targeted and results applications produced in the process of performance measures development. These elements and their mode of application to the measure development process should be clearly documented to permit full appraisal of the quality of the measures.
The authors would like to thank the expert group on quality indicators at the Agency for Quality in Medicine, Cologne, for their critical comments (Prof. Brech, Dr Brenner, Dr Gramsch, Prof. Kolkmann, Dr Krumpaszky, Prof. Lorenz, Dr Oesingmann, Dr Rheinberger, Dr v. Stillfried, Herr Stobrawa).
Address reprint requests to Professor Dr Max Geraedts, MPH, Public Health Program, School of Medicine, University of Duesseldorf, PO Box 10 10 07, D-40001 Duesseldorf, Germany. E-mail: geraedts{at}uni-duesseldorf.de 
Accepted for publication October 1, 2002.
References
- Social Code Book V (SGB V). Statutory Health Insurance, as last amended by law of December 22, 1999 [BGBl I (Federal Law Gazette I, p. 2626)] (in German): http://www.bmgesundheit.de Accessed 30 January 2002.
- Eddy DM. Performance measurement: problems and solutions. Health Affairs 1998; 17: 7–25.[Medline]
- German Medical Association, Federal Association of Statutuory Health Insurance Physicians. Criteria for appraising clinical guidelines in medical care (in German). Dtsch Aerztebl 1997; 94: A2154–2155.
- Ollenschlaeger G, Helou A, Kostovic-Cilic L et al. Checklist on the methodological quality of guidelines—a contribution to promote the quality of guidelines (in German). ZaeFQ 1998; 92: 191–194.
- Crombie IK, Davies HTO, Abraham SCS. Audit Handbook—Improving Health Care Through Clinical Audit. Chichester, NY: John Wiley and Sons, 1993.
- Foundation for Accountability (FACCT). The facts about FACCT. Accountability Action 1996; 1: 5–8.
- Joint Commission on Accreditation of Healthcare Organizations (JCAHO). Primer on Indicator Development and Application. Measuring Quality in Health Care. Oakbrook Terrace, IL: JCAHO, 1990.
- Committee of experts for the ‘concerted action in the health care system’ (SVR). Expert Opinion 2000/2001—Meeting of Demands and Efficiency (in German): http://www.svr-gesundheit.de Accessed 30 January 2002.
- Reiter A, Geraedts M, Stillger R, Misselwitz B. Evaluation and revision of quality indicators in surgical gynecology (in German). Gebh Frauenheilkunde 2002; 62: 635–643.
- Katz JN, Sangha O. Assessment of the quality of care. Arthritis Care Res 1997; 10/6: 359–369.[Medline]
- Quality Assurance Working Group of the German Association for Medical Informatics, Biometry, and Epidemiology. Terms and concepts in quality management (in German). Informatik Biometrie Epidemiol 1996; 4: 200–230.
- Joint Commission on Accreditation of Healthcare Organizations (JCAHO). Guide to Quality Assurance. Oakbrook Terrace, IL: JCAHO, 1988.
- Joint Commission on Accreditation of Healthcare Organizations (JCAHO). National Library of Healthcare Indicators—Health Plan and Network Edition. Oakbrook Terrace, IL: JCAHO, 1997.
- German Standards Institute. Quality management systems—Fundamentals and vocabulary (ISO 9000:2000). Berlin: Beuth, 2001.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||