This Article Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Brouwers, M. C. Articles by Browman, G. P. Search for Related Content PubMed PubMed Citation Articles by Brouwers, M. C. Articles by Browman, G. P. Related Articles Related Article

A for Effort: Learning From the Application of the GRADE Approach to Cancer Guideline Development

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario Canada; Program in Evidence-based Care, Cancer Care Ontario, Hamilton, Ontario, Canada

Mark R. Somerfield

American Society of Clinical Oncology, Alexandria, VA

George P. Browman

British Columbia Cancer Agency, Victoria, British Columbia, Canada

Few would argue against a methodology that facilitates more explicit and transparent judgments about health care research evidence and the link to practice and policy decisions. In this issue of the Journal of Clinical Oncology, De Palma et al¹ describe the application of one such method, the GRADE (Grades of Recommendation, Assessment, Development, and Evaluation) approach, for the development of clinical practice guidelines for breast, colorectal, and lung cancer treatment. The GRADE approach has emerged in response to concerns about the glut of competing grading systems, their limitations, and the confusion resulting from lack of a common rubric. GRADE (http://www.gradeworkinggroup.org/) provides an explicit method for arriving at recommendations classified according to the quality of supporting evidence.^2-8

On the basis of their experience, De Palma et al highlight several advantages of the GRADE method. First, it focuses guideline developers on key methodologic issues; second, it organizes the presentation of the evidentiary base on specific outcomes; and, third, it provides guidance for balancing tradeoffs among risks, benefits, and costs. Although these are valuable contributions, the experience of De Palma et al highlights remaining challenges in the application of GRADE.

Although most agree that GRADE's emphasis on study design, overall study quality, consistency of results, and directness of effects are critical domains on which to focus multistakeholder guideline development panels, it is clearly more challenging to operationalize these concepts. The signal value of GRADE is purported to be its explicitness, designed to improve the transparency of decision making, facilitate consistency in judgments about the quality of evidence, and link evidence to recommendations. However, the GRADE methodology has built into it vague guidance that undermines its goals of explicitness and uniformity of judgments, and leaves considerable room for individual interpretations. Even considering the least subjective component of evaluation, the quality of evidence, GRADE allows for "upgrading" or "downgrading"; however, these processes are not fully operationalized. This uncertainty is reflected in the data reported by the authors. De Palma et al document considerable variability among participants in their assessments of evidence quality, risk-benefit tradeoffs, and strength of recommendations (see their Fig 1¹). Furthermore, this variability was evident despite considerable efforts to overcome it by the authors by providing two training sessions for participants.

Decision makers seek guidance and would like to be able to rely on classification systems such as GRADE to help simplify communication and justify difficult recommendations or decisions for which they will be held accountable. This objective requires clear boundaries between "classes" of evidence and a clear link between this evidence and the "classes" of recommendations to which the evidence eventually leads. However, in the study by De Palma et al, the quality of the evidentiary base in one example was graded as "low" by three reviewers, "intermediate" by nine reviewers, and "high" by three other reviewers (see their Fig 1I), suggesting that decision makers will not find GRADE adequate for rendering complex issues more simple. If classification systems of evidence are intended to provide the foundation for recommendations, then the measurement properties of these systems (such as reliability of judgments, validity of concepts) must be adequate. If this cannot be achieved, then some could argue that the fundamental purpose of grading and categorization of evidence and recommendations—to improve the standardization and quality of decision making—is not being met.

The valuable observations provided by De Palma et al suggest that, no matter how explicit the approach used, there are likely known, and perhaps some unknowable, factors that influence how participants will interpret data and apply it in their judgments. This should not be interpreted as reflecting a problem with the panel members who participate in the guideline development processes; rather, they highlight measurement challenges inherent in any system like GRADE and the underappreciated contexts within which research evidence is factored into clinical and health policy decisions.

In applying GRADE, the most common recommendation categories one might expect in the oncology field are "Probably use it" or "Probably do not use it." This is borne out in the data from De Palma et al. Of the nine scenarios where recommendations could be made, seven (78%) recommendations fell within these categories (see their Table 3). This is not surprising, because cancer care options are rarely without important risks, benefits are often incremental, and individual patient and societal values are not necessarily aligned as in the case of expensive drugs that yield arguably modest gains. Just how one interprets these recommendation categories clinically is not known.

Further, the "Use it" and "Don't use it" categories in the GRADE approach smack of practice standards and are semantically inconsistent with the disclaimer language that accompanies most guidelines. These disclaimers typically underscore the voluntary nature of the recommendations and recognition of the central roles of clinical judgment, patient and provider preferences and values, system context—the list could go on—in applying any guideline. Indeed, recommendations, whether clinical or social, that are informed by evidence should be made within local contexts (social, political, environmental, financial, or religious) that account for circumstances that cannot always be anticipated. The GRADE approach currently does not fully capture these broader issues, resulting in potential challenges for users wanting to apply recommendations in a judicious and context-appropriate manner.

There is a misconception that the evidence-based approach, if properly applied, will eventually yield to "correct" judgments. This is a simplistic notion that fails to acknowledge research evidence as only one component of many in clinical and policy decision making. What the evidence-based approach can do is explicitly represent the issues for competing arguments and foster critical thinking by insisting on accountability to evidence. The GRADE approach shows promise in leading in this direction, but its reductionist conclusion, which employs categoric systems to classify evidence and recommendations, does not meet the necessary nuanced reality when competing interests and different interpretations of both research evidence and the context of important decisions are considered. As such, it might not be surprising that none of the most respected cancer-specific guideline groups (eg, SOR [Standards, Options, Recommendations], ASCO [American Society of Clinical Oncology], NCCN [National Comprehensive Cancer Network], and CCO [Cancer Care Ontario]) or the large national guideline groups that tackle cancer guidelines (eg, NICE [National Institute for Clinical Excellence], SIGN [Scottish Intercollegiate Guideline Network]; NHMRC [National Health and Medical Research Council]) embraces this methodology fully or relies solely on any categoric approach to define recommendations. We suspect that this has not been lost on De Palma et al. The authors state that, to avoid losing the richness of the in-depth GRADE evidence review approach, they eschewed the Yes/No conclusion and created a template for presentation of their online version of recommendations that provides greater contextualization of evidence and the addition of new labels that seems to better suit their stakeholders.

De Palma et al have provided valuable lessons to cancer guideline developers. Their claims that, in contrast to other guideline developers, their approach has several unique aspects (ie, multidisciplinary interaction, consideration of the evidence), however, cannot be justified. Even a cursory reading of any contemporary guideline in oncology that is based on a systematic (v narrative) review of the literature reveals that an in-depth assessment of the evidence by a multidisciplinary panel is a common feature. That said, De Palma et al have conducted a rigorous exercise that fairly tested the performance of an emerging methodology for guideline development; they have given the process a chance to perform under ideal conditions (eg, by training participants); they have documented and reported their findings well; and they have demonstrated what is feasible in using a relatively complex method. Most importantly, De Palma et al have illustrated the limitations of an approach that does not yet fully acknowledge the complexity of the deliberations around guideline development, including the range of factors that affect decisions. We are hopeful that, as the GRADE methodology continues to evolve and be refined, additional attention will be directed to the more difficult-to-measure factors that influence clinical and policy recommendations. At the end of the day, compared with other methods and strategies for guideline development and expressions of evidence and recommendations, does the GRADE methodology facilitate better decisions and quality of care for patients? As of today, we think the jury is still out, but at least it is still deliberating.

AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest.

AUTHOR CONTRIBUTIONS

Conception and design: Melissa C. Brouwers, Mark R. Somerfield, George P. Browman

Administrative support: Melissa C. Brouwers

Manuscript writing: Melissa C. Brouwers, Mark R. Somerfield, George P. Browman

Final approval of manuscript: Melissa C. Brouwers, Mark R. Somerfield, George P. Browman

ACKNOWLEDGMENTS

The opinions expressed in this article are those of the authors and do not necessarily represent the views of Cancer Care Ontario (MCB), the American Society of Clinical Oncology (MRS), or the BC Cancer Agency (GPB).

REFERENCES

1. De Palma R, Liberati A, Ciccone G, et al: Developing clinical recommendations for breast, colorectal, and lung cancer adjuvant treatments using the GRADE system: A study from the Programma Ricerca e Innovazione Emilia Romagna Oncology Research Group. J Clin Oncol 26:1033-1039, 2008[Abstract/Free Full Text]

2. Schünemann HJ, Best D, Vist G, et al: Letters, numbers, symbols, and words: How best to communicate grades of evidence and recommendations? CMAJ 169:677-680, 2003[Abstract/Free Full Text]

3. Atkins D, Best D, Briss PA, et al: Grading quality of evidence and strength of recommendations. BMJ 328:1490, 2004[Abstract/Free Full Text]

4. Atkins D, Eccles M, Flottorp S, et al: Systems for grading the quality of evidence and the strength of recommendations: I, Critical appraisal of existing approaches. BMC Health Serv Res 4:38, 2004[CrossRef][Medline]

5. Atkins D, Briss PA, Eccles M, et al: Systems for grading the quality of evidence and the strength of recommendations: II, A pilot study of a new system for grading the quality of evidence and the strength of recommendations. BMC Health Serv Res 5:25, 2005[CrossRef][Medline]

6. Guyatt G, Vist G, Falck-Ytter Y, et al: An emerging consensus on grading recommendations? ACP J Club 144:A08, 2006

7. Guyatt G, Gutterman D, Baumann M, et al: Grading strength of recommendations and quality of evidence in clinical guidelines: Report from an American college of chest physicians’ task force. Chest 129:174-181, 2006[CrossRef][Medline]

8. Schünemann HJ, Jaeschke R, Cook D, et al: An official ATS statement: Grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 174:605-614, 2006[Free Full Text]

Related Article

• Developing Clinical Recommendations for Breast, Colorectal, and Lung Cancer Adjuvant Treatments Using the GRADE System: A Study From the Programma Ricerca e Innovazione Emilia Romagna Oncology Research Group
  Rossana De Palma, Alessandro Liberati, Giovannino Ciccone, Elena Bandieri, Maurizio Belfiglio, Manuela Ceccarelli, Maurizio Leoni, Giuseppe Longo, Nicola Magrini, Maurizio Marangolo, and Fausto Roila
  JCO 2008 26: 1033-1039 [Abstract] [Full Text]

This Article Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Brouwers, M. C. Articles by Browman, G. P. Search for Related Content PubMed PubMed Citation Articles by Brouwers, M. C. Articles by Browman, G. P. Related Articles Related Article