|
Advertisement |
|
|
||
 |
|
|||||
|
|||||
|
 |
||||||
|
||||||
|
||||||
|
Journal of Clinical Oncology, Vol 22, No 10 (May 15), 2004: pp. 2000-2007 © 2004 American Society of Clinical Oncology. DOI: 10.1200/JCO.2004.06.157
International Assessment of the Quality of Clinical Practice Guidelines in Oncology Using the Appraisal of Guidelines and Research and Evaluation InstrumentFrom the Fédération Nationale des Centres de Lutte Contre le Cancer, Paris; Centre Léon Bérard, Lyon, France; Centre for Quality of Care Research, University Medical Centre Nijmegen, the Netherlands; Program in Evidence-Based Cancer Care and Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada; and Department of Community Health Sciences, St George's Hospital Medical School, London, United Kingdom. Address reprint requests to Béatrice Fervers, MD, Centre Léon Bérard, 69373 Lyon Cedex 8, France; e-mail: fervers{at}lyon.fnclcc.fr
PURPOSE: To describe the quality of oncology guidelines developed in different countries. METHODS: The Appraisal of Guidelines and Research and Evaluation (AGREE) Instrument was used to assess the quality of 100 guidelines (including 32 oncology guidelines) from 13 countries. The criteria of the instrument are grouped into six quality domains: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence. RESULTS: Oncology guidelines had significantly higher scores on rigor of development than nononcology guidelines (42.2% v 29.4%; P = .02). In particular, systematic methods to search for evidence were more often used (P = .01); the methods for formulating the recommendations were more clearly described (P = .02); and health benefits, risks, and side effects were more often considered in formulating the recommendations (P = .03). Although the standardized scores for the other domains were not significantly different, the oncology guidelines had significantly higher scores for items measuring inclusion of all relevant professional groups (P = .05), consideration of patient views (P = .04), and presentation of different options (P = .05). Only three organizations producing oncology guidelines had standardized scores more than 60% for more than three domains. CONCLUSION: The quality of clinical practice guidelines (CPGs) is modest in general, but for certain domains, oncology guidelines seem to be of better quality than others. The experience of the organization may explain higher scores for some items. Research projects and training aimed at improving the quality of guidelines should be developed. The AGREE instrument could provide a basis for defining steps in a shared development approach to produce high-quality CPGs.
Clinical practice guidelines (CPGs) are increasingly used throughout the world to improve the quality of patient care in all areas of medicine.1,2 The driving forces behind their development include the evidence-based medicine paradigm, and concerns about the quality of care and increasing healthcare costs.3 This has led to an increase of guideline activities in the 1990s.4 Numerous guidelines development programs have been set up in most countries to develop CPGs in various specialties, including oncology. In MEDLINE, the number of publications indexed under the publication type "practice guideline" and the medical subject heading term "neoplasms" increased from 10 in 1991 to 108 in 2003. Several studies have suggested, however, that the quality of many published guidelines is modest and that there is heterogeneity among different guideline programs.5-8 The proliferation of guidelines and concerns about their quality have led to a growing need to develop internationally recognized criteria for the quality of guideline development and reporting. The Appraisal of Guidelines and Research and Evaluation (AGREE) project was established in 1998 to respond to this need. Researchers from 10 European countries, Canada, New Zealand, and the United States (including representatives from two oncology guideline programs) worked to promote collaborative research on the quality appraisal of guidelines and the methodology of guideline development. The main product of the collaboration was the AGREE Instrument, a validated generic instrument for the appraisal of guidelines applicable to any disease area.9 In the validation phase of the instrument, 100 guidelines (including 32 oncology guidelines) from 13 countries were evaluated.10 In this article we present the quality assessment of the oncology guidelines compared with that of nononcology guidelines and present possible explanations for the differences observed. We also discuss the opportunities to create international networks as a means to improve the efficiency and effectiveness of guideline development.
Selection of Guidelines Guidelines were defined as systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific circumstances.11 Documents that did not contain recommendations for clinical practice (eg, systematic reviews, service documents) were excluded. Collaborators in different countries were asked to select from seven to 10 guidelines, published between 1992 and 1999, preferably on asthma, breast cancer, and diabetes. In all, 100 guidelines developed by 74 different agencies and organizations from 13 countries were selected. The sample included 24 guidelines on breast cancer and eight guidelines on other types of cancer (Table 1). Two guidelines were produced by organizations in the United States: one from the American Society of Clinical Oncology and the other from the National Comprehensive Cancer Network.12
Selection of Appraisers In each country four independent appraisers were recruited per guideline. Where possible, each appraiser assessed two guidelines. The appraisers included medical practitioners, clinical experts, clinical researchers, and methodologists. Members of the guideline development group, members of the secretariat that produced the guidelines, and external referees were not eligible to be appraisers.
Instrument Development The final instrument has 23 items grouped into the above-listed six domains and one overall judgment on whether the guideline ought to be recommended for use. To help users understand the items, the instrument contains a users' guide with explanatory notes. Each item is scored on a 4-point Likert scale (4 = strongly agree, 3 = agree, 2 = disagree, or 1 = strongly disagree) and the overall judgment is scored with a 4-point categoric scale (not recommended, recommended with provisos or alterations, strongly recommended, or unsure). The internal consistency of the final instrument was acceptable (Cronbach's alpha ranged from .64 to .88). The intraclass correlations averaged across four appraisers ranged from .57 to .91. The appraisers said that they found the instrument useful for assessing guidelines (95%) and that they found the users' guide helpful (98%). More details about the development and validation of the instrument are described in the main article of the AGREE project.10
Analyses The minimum value for the standardized domain score is therefore 0%, and the maximum is 100%. According to the instructions for use of the AGREE Instrument, the six domain scores were considered independently and a single quality score was not calculated. The mean item and standardized domain scores of the oncology guidelines were compared with those of the other guidelines using one-way analysis of variance. The level of significance was set at P < .05, which means that one of 20 tests could be expected to be significant by chance alone. In addition, we compared the scores for oncology guidelines developed by organizations specializing in oncology with those developed by organizations not specifically specializing in oncology. All analyses were performed using SPSS 9.0 (SPSS Inc, Chicago, IL).
Quality of Oncology Guidelines Compared With Guidelines in Other Disease Areas The analyses of variance revealed a significant difference between oncology and nononcology guidelines for the domain rigor and development (Table 2). Compared with the nononcology guidelines, the oncology guidelines had significantly higher scores for item 8 (search methods for evidence; 2.34 v 1.78; P = .011), item 10 (methods for formulating recommendations; 2.29 v 1.84; P = .016), and item 11 (health benefits, side effects, and risks considered; 2.63 v 2.26; P = .026; Table 3). Although the standardized scores for the other domains were not significantly different, the oncology guidelines had significantly higher scores for item 4 (all relevant professional groups included; 2.60 v 2.21; P = .047), item 5 (patient views sought; 1.76 v 1.52; P = .038), and item 16 (different options clearly presented; 2.90 v 2.63; P = .048). However, the mean score across all guidelines was low for item 5, indicating that even the oncology guidelines performed poorly for this item. The scores for item 7 (pilot testing), item 9 (selection criteria described), and the three items (19 to 21) in the domain applicability were also low for all guidelines. The nononcology guidelines scored higher than the average (2.5) for six items, whereas the oncology guidelines scored higher than the average for these items and also for item 4 (all relevant professional groups included) and item 11 (benefits, side effects, and risks considered). For the oncology guidelines the average percentage of appraisers (strongly) recommending the guideline was 72.7 (95% CI, 61.1 to 84.2), which did not differ significantly from that for the other guidelines (67.4; 95% CI, 58.7 to 76.1).
Quality of Oncology Guidelines According to Type of Organization None of the guidelines had high scores for all domains. Guidelines from three organizations scored more than 60% for four domains: the Royal College of Radiologists & Clinical Oncology Information Network (commissioned by the National Health Service Executive), the French Agency for Accreditation and Evaluation in Healthcare, and the French Federation of Comprehensive Cancer Centers (FNCLCC). The guideline from the American Society of Clinical Oncology scored more than 60% for the domain rigor of development and more than 50% for the domains scope and purpose, clarity and presentation, and editorial independence. Almost half (15 of 32) of the oncology guidelines were developed by organizations specialized in oncology. Compared with guidelines from organizations that develop CPGs across several disease areas, guidelines from oncology-specific organizations had slightly higher scores for four domains (Table 4), but these differences were not statistically significant. The scores were similar for the domain scope and purpose. The guidelines developed by organizations that covered several disease areas had a significantly higher score for the domain applicability (31.8 v 19.1; P = .045). However, the mean domain scores were low for all organizations and the only organizations with scores of more than 50% for applicability were the Royal College of Radiologists and Clinical Oncology Information Network and Health Technology Assessment Office Osteba and Basque Health Department. Additional analysis at the item level revealed no other statistically significant differences, but for all three items in this domain the mean item score for the oncology guidelines was lower than that for the others.
Here we report the results of the first international comparison of the quality of oncology guidelines. The quality was assessed using the AGREE instrument, which has been validated.10 The instrument has been endorsed by the WHO and is becoming an accepted standard in guidelines development.18 The results presented here were obtained with the first version of the instrument, but it is unlikely that the few changes made invalidate the analysis. We used the same statistical approach as that used in the validation study, on the basis of the assumption that each domain and each individual item contribute equally to the quality of a guideline.10 It should be noted that the instrument evaluates the methodologic quality of a guideline but this is not independent of how well the guideline development process is reported.19 However, explicit reporting does not guarantee optimal recommendations. It is possible that a well-reported guideline contains flawed recommendations, and conversely, a guideline that is not systematically developed may provide sound advice that is consistent with evidence. This can be compared with the activities of the Consolidated Standards of Reporting Trials group and their statement on reporting for clinical trials.20 The results show that the quality of the guidelines assessed is modest, in general, but varies among the different organizations. However, for oncology guidelines and other guidelines the range of scores was wide, indicating heterogeneous quality. The quality scores for the oncology guidelines were higher than those obtained for guidelines in other disease areas for several aspects (eg, multidisciplinary development, patient involvement, use of systematic methods and explicit criteria for selecting evidence, and consideration of benefits and harms in formulating the recommendations). The guidelines analyzed here were not a random sample and the number per organization was small; therefore, they might not be representative of all existing guidelines. Some countries or organizations may have selected their best guidelines, whereas others may have included a range of guidelines with varying quality (based on face validity) to test the discriminative value of the instrument. In addition, most of the guidelines included in this analysis are likely to be outdated now; thus, the quality of the updated versions could have been improved since the study was conducted. Overall, the items for which high scores and those for which low scores were obtained were similar for oncology guidelines and other guidelines. This suggests that the high-scoring items correspond to those aspects of guideline development that are more generally accepted and respected by the various programs. Research projects, training, and other initiatives should target those aspects that correspond to the low-scoring items in an attempt to improve the methodology and quality of CPG. For instance, several organizations now are introducing patient participation in guideline development. Some, such as the Scottish Intercollegiate Guideline Network and the National Institute for Clinical Excellence (United Kingdom) have formal units to address questions such as how to improve effective involvement of the patient's perspective and their expectations, and how to develop specific methods for this.21,22 The poor scores for the domain applicability emphasizes the need to improve the conceptualization of the implementation of guidelines during the development process. CPGs in oncology have been shown to lead to significant changes in practice when dissemination and implementation are conceptualized from the beginning of the guideline development process.23,24 The low scores for guidelines from organizations specializing in oncology may be explained in settings in which links with regional cancer networks exist, given that the networks adapt CPGs into protocols for local implementation and use. The higher scores obtained by the oncology guidelines for item 4 (all relevant professional groups included), item 9 (search methods for evidence), and item 10 (methods used for formulating recommendations), compared with nononcology guidelines, might reflect a specificity of oncology, which has the tradition of a multidisciplinary approach and is heavily reliant on clinical trials as part of routine practice.25,26 The clinical management of cancer patients often requires a transparent, multidisciplinary approach because the treatment modalities (such as surgery, radiotherapy, and chemotherapy) cannot be provided by the same specialist.26-29 There is some evidence to suggest that the absence of multidisciplinary care may affect survival.30 Furthermore, the measurement of cancer treatment outcomes is complex, with important tradeoffs between benefit and harm that require closer evaluation of outcomes and treatment choices.31,32 Compared with many other treatments, cancer treatments tend to have more side effects, some of which are short term (and resolve when treatment is stopped) and long term (and may continue after treatment is stopped). The uncertainty of the outcome for an individual patient, particularly in terms of length of survival, means that other outcomes such as quality of life need to be considered. This might explain the statistically significant differences in favor of oncology guidelines for item 11 (health benefits, side effects, and harms considered) and item 16 (different options clearly presented). Patients, clinicians, researchers, and policy makers do not necessarily have the same views on what outcome is the most important. For example, a patient might consider that the potential benefits in terms of survival might not be worthwhile in view of the potential important, even life-threatening side effects, of a given treatment. Therefore, it is important to consider patient views and expectations in cancer-related treatment recommendations in guidelines (item 5). Although patients' preferences would seem to be more routinely considered in oncology, and the oncology guidelines obtained a significantly higher score for this item, the scores were low for both types of guidelines. This could be explained by the difficulty of identifying the most appropriate methods and the lack of resources for involving patients in the process of guideline development; therefore, more research is needed in this area. This study shows much potential duplication of effort; many guidelines have been developed on the same topic by different organizations. For example, in this sample there were 24 guidelines on breast cancer. We should be examining opportunities for sharing some elements of the expensive and time-consuming process of guideline development. For example, the evidence on which the guidelines are based could be shared, but the actual recommendations could still be tailored to local and cultural contexts.33 Thus, the literature review could be shared, which coincides with the aim of the Cochrane Collaboration. When a Cochrane systematic review is not available, it could be conducted by organizations with expertise in searching and assessing evidence in the literature and then shared more broadly for local adaptation.34 In some situations, high-quality guidelines produced in one context could be locally adapted and implemented, thus reducing duplication of effort and inefficient use of resources. Given that many guideline programs have similar strategies for developing guidelines, national and international collaboration should be encouraged.35,36 For example, the collaboration between the FNCLCC guidelines project (Standards, Options, and Recommendations) and the Cancer Care Ontario Guideline Initiative provided insight into the challenges encountered when different countries cooperate, and insight into the legitimate reasons for inconsistency of guideline recommendations based on the same evidence, as a result of national and medical cultural differences.37 The practice guideline development cycle proposed by Browman et al,38 the framework for local adaptation developed by Graham et al,39 and the defined development process such as that used by the FNCLCC40 could be useful tools for elaborating a shared development strategy. These tools coupled with the quality criteria in the AGREE Instrument form the basis for a project to establish a network for clinical practice guidelines in oncology that is being spearheaded by the FNCLCC. This network is called AGREE Oncology and aims to define a common methodologic framework for developing clinical practice guidelines in oncology with a clear identification of aspects that can be shared and those that require local input. Given that the results from our analysis show that there is scope for improving the quality of guidelines, the project will provide added value for guideline development programs by building on existing experiences while taking into account the cultural and organizational diversity of the participating organizations or countries.33 Thus, the development of high-quality, locally relevant clinical practice guidelines will be encouraged.
The appendix is included in the full-text version of this article, available on-line at http://www.jco.org. It is not included in the PDF (via Adobe® Acrobat Reader®) version.
The authors indicated no potential conflicts of interest.
The AGREE Collaboration is an international group of researchers from 13 countries who designed and field tested the AGREE Instrument. Individuals participating in the AGREE Collaboration are listed in the Appendix.
Supported by the French Rhône-Alpes Region and the French Cancer League. The AGREE project was supported by a grant from the European Union BIOMED2 Programme (BMH4-98-3669). Authors' disclosures of potential conflicts of interest are found at the end of this article.
1. Woolf SH, Grol R, Hutchinson A, et al: Potential benefits, limitations, and harms of clinical guidelines. BMJ 318:527-530, 1999
2. Smith TJ, Hillner BE: Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways. J Clin Oncol 19:2886-2897, 2001 3. Woolf SH: Practice guidelines: A new reality in medicine—I. Recent developments. Arch Intern Med 150:1811-1818, 1990[Abstract] 4. Day P, Klein R, Miller F: Hurdles and levers: A comparative US-UK study of guidelines. Nuffield Trust Series No. 4. London, United Kingdom, Nuffield Trust, 1998 5. Ward JE, Grieco V: Why we need guidelines for guidelines: A study of the quality of clinical practice guidelines in Australia. Med J Aust 165:574-576, 1996[Medline]
6. Graham ID, Beardall S, Carter AO, et al: What is the quality of drug therapy clinical practice guidelines in Canada? CMAJ 165:157-163, 2001
7. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281:1900-1905, 1999 8. Grilli R, Magrini N, Penna A, et al: Practice guidelines developed by specialty societies: The need for a critical appraisal. Lancet 355:103-105, 2000[CrossRef][Medline] 9. The AGREE Collaboration: Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. http://www.agreecollaboration.org
10. AGREE Collaboration: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Qual Saf Health Care 12:18-23, 2003 11. Field MJ, Lohr KN Clinical Practice Guidelines: Directions for a New Program. Washington DC, Institute of Medicine, National Academy Press, 1990
12. American Society of Clinical Oncology: Recommended breast cancer surveillance guidelines. J Clin Oncol 15:2149-2156, 1997
13. Cluzeau F, Littlejohns P, Grimshaw J, et al: Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 11:21-28, 1999
14. Grol R, Dalhuijsen J, Thomas S, et al: Attributes of clinical guidelines that influence use of guidelines in general practice: Observational study. BMJ 317:858-861, 1998 15. Lohr KN, Field MJ: A provisional instrument for assessing clinical practice guidelines, in Field MJ, Lohr KN (eds): Guidelines for Clinical Practice: From Development to Use. Washington, DC, Institute of Medicine, National Academy Press, 1992, pp 346-410 16. Lohr KN. The quality of practice guidelines and the quality of health care, in Selbmann HK (ed): Guidelines in Health Care: Report of a WHO Conference. Baden-Baden, Germany, Nomos Verlagsgesellschaft, 1998, pp 42-52
17. Thomson R, Lavender M, Madhok R: How to ensure that guidelines are effective. BMJ 311:237-242, 1995 18. Grol R, Cluzeau FA, Burgers JS: Clinical practice guidelines: Towards better quality guidelines and increased international collaboration. Br J Cancer 89:S4-S8, 2003 (suppl 1) 19. Hayward RS, Wilson MC, Tunis SR, et al: Users' guide to the medical literature: VIII. How to use clinical practice guidelines —A. Are the recommendations valid? JAMA 274:570-574, 1995[CrossRef][Medline] 20. Begg C, Cho M, Eastwood S, et al: Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276:637-639, 1996[CrossRef][Medline] 21. Scottish Intercollegiate Guideline Network: Patient involvement in guideline development. http://www.sign.ac.uk/patients/index.html 22. National Institute for Clinical Excellence: Patient Involvement Unit for NICE. http://www.nice.org.uk/cat.asp?c=73135 23. Ray-Coquard I, Philip T, Lehmann M, et al: Impact of a clinical guidelines program for breast and colon cancer in a French cancer center. JAMA 278:1591-1595, 1997[Abstract] 24. Ray-Coquard I, Philip T, de Laroche G, et al: A controlled "before-after" study: Impact of a clinical guidelines programme and regional cancer network organization on medical practice. Br J Cancer 86:313-321, 2002[CrossRef][Medline] 25. Löwy I: Between Bench and Bedside. Cambridge, MA, Harvard University Press, 1996 26. Abbatucci JS: Long-term experience with multidisciplinary approach in oncology. Rays 8:165-172, 1983
27. Ray MD: Shared borders: Achieving the goals of interdisciplinary patient care. Am J Health Syst Pharm 55:1369-1374, 1998 28. Tripathy D: Multidisciplinary care for breast cancer: Barriers and solutions. Breast J 9:60-63, 2003[CrossRef][Medline] 29. Chang JH, Vines E, Bertsch H, et al: The impact of a multidisciplinary breast cancer center on recommendations for patient management: The University of Pennsylvania experience. Cancer 91:1231-1237, 2001[CrossRef][Medline] 30. Junor EJ, Hole DJ, Gillis CR: Management of ovarian cancer: Referral to a multidisciplinary team matters. Br J Cancer 70:363-370, 1994[Medline] 31. Fettings JH: American Society of Clinical Oncology (ed): 1994 ASCO Educational Book. Principles of practice guidelines. Patient outcomes and guidelines. Alexandria, VA, American Society of Clinical Oncology, 1994
32. American Society of Clinical Oncology: Outcomes of cancer treatment for technology assessment and cancer treatment guidelines. J Clin Oncol 14:671-679, 1996 33. Eisinger F, Geller G, Burke W, et al: Cultural basis for differences between US and French clinical recommendations for women at increased risk of breast and ovarian cancer. Lancet 353:919-920, 1999[CrossRef][Medline] 34. Browman GP: Improving clinical practice guidelines for the 21st century: Attitudinal barriers and not technology are the main challenges. Int J Technol Assess Health Care 16:959-968, 2000[CrossRef][Medline]
35. Burgers JS, Grol R, Klazinga NS, et al: Towards evidence-based clinical practice: An international survey of 18 clinical guideline programs. Int J Qual Health Care 15:31-45, 2003 36. Fervers B, Philip T, Haugh M, et al: Clinical practice guidelines in Europe: Time for European co-operation for cancer guidelines. Lancet Oncol 4:139-140, 2003[CrossRef][Medline] 37. Browman G: Background to clinical guidelines in cancer: SOR, a programmatic approach to guideline development and aftercare. Br J Cancer 84:1-3, 2001 (suppl 2)
38. Browman GP, Levine MN, Mohide EA, et al: The practice guidelines development cycle: A conceptual tool for practice guidelines development and implementation. J Clin Oncol 13:502-512, 1995 39. Graham ID, Harrison MB, Brouwers M, et al: Facilitating the use of evidence in practice: Evaluating and adapting clinical practice guidelines for local use by health care organisations. J Obstet Gynecol Neonatal Nurs 31:599-611, 2002[CrossRef][Medline] 40. Fervers B, Hardy J, Blanc-Vincent MP, et al: SOR: Project methodology. Br J Cancer 84:S8-S16, 2001 (suppl 2) Submitted June 30, 2003; accepted March 2, 2004. This article has been cited by other articles:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||
|
|||||||||||
|
|
Copyright © 2004 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|
TOP
ABSTRACT
INTRODUCTION
