This Article Full Text (PDF) Erratum (v22,p4656) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Schneider, E. C. Articles by Emanuel, E. J. Search for Related Content PubMed PubMed Citation Articles by Schneider, E. C. Articles by Emanuel, E. J.

Developing a System to Assess the Quality of Cancer Care: ASCO's National Initiative on Cancer Care Quality

¹ Department of Health Policy Management, Harvard School of Public Health, Harvard University; and the Section on Health Policy, Division of General Medicine, Brigham and Women's Hospital, Boston, MA

Jennifer L. Malin², Katherine L. Kahn²

² Department of Medicine, University of California, Los Angeles, Los Angeles; and Rand Corporation, Santa Monica, CA

Ezekiel J. Emanuel³

³ Department of Clinical Bioethics, Warren G. Magnuson Clinical Center, National Institutes of Health, Bethesda, MD

The quality of care for cancer has been questioned in recent years.^1-7 In a 1999 report, the Institute of Medicine (IOM) concluded that many patients with cancer did not receive state-of-the-art care.⁸ The IOM committee recommended a number of steps including the creation of a quality monitoring system capable of regularly reporting on the quality of care for patients with cancer.

Oncologists might find the IOM recommendation surprising. Efforts to monitor the care and outcomes of oncology patients date back at least to 1922, when the American College of Surgeons (ACS) founded its Commission on Cancer—a group specifically tasked with standardizing and improving the quality of cancer care.⁹ The Commission's National Cancer Data Base (NCDB) has provided useful data about the epidemiology of cancer and treatment patterns.¹⁰ Furthermore, oncology trials have led the way in assessment of patient outcomes such as health status and quality of life; the results from these trials have improved treatments, survival, and the quality of life for many patients with cancer.

Nevertheless, research on the quality of care throughout at least the last decade has demonstrated that increases in the knowledge of treatments with proven efficacy do not translate directly to the optimal delivery of such treatments to patients.^11-16 Accumulating evidence suggests that "underuse" and "overuse" of care may occur for patients with cancer.^17-19 Also, compared with the outcomes of patients in clinical trials, the outcomes of treatment for the general population of patients with cancer may be less favorable.²⁰

In the last few decades, the methods used to measure the quality of care have advanced.²¹ However, until recently, few programs have attempted to use these methods to measure and improve the quality of care for large populations of patients with cancer on an ongoing basis.²² Despite its appeal, the development of a national monitoring system is likely to be a highly complex undertaking with substantial implications for clinicians, patients, institutional leaders, policy makers, and other stakeholders. In this manuscript, we describe a recent effort, the National Initiative on Cancer Care Quality (NICCQ) promoted by the American Society of Clinical Oncology (ASCO), to develop a prototype for a national system that could monitor the quality of cancer care. We discuss the goals, key features, practical challenges, and key decisions that lie ahead if this program is to be expanded.

METHODS

ASCO initiated NICCQ in 2000, with the goals of developing potential measures of the quality of cancer care for two common cancers (breast and colorectal cancer), ascertaining current practice for these two diseases, and designing and implementing the first phase of a prototype quality monitoring system. ASCO engaged collaborators including professional organizations, expert advisors comprising a Task Force on Quality, and research teams at Harvard University and Rand Corp (Appendix A). Using the ACS National Cancer Database (NCDB; a national registry of incident cancer cases) and its network of participating hospital cancer registries, the NICCQ project identified and solicited participation of approximately 5,000 patients diagnosed with breast or colorectal cancer during 1998 in one of five major metropolitan areas of the United States. Detailed clinical data have been collected from patient surveys and medical records. At its completion, the project is expected to produce a detailed profile of the quality of care for breast and colorectal cancer in the five selected metropolitan areas. To date, the project has surfaced important challenges and lessons that are addressed in the following sections.

RESULTS

Goals and Key Features of a National Quality Monitoring System for Cancer Care
The goal of a quality monitoring system should be to measure and report on the quality of cancer care as accurately as possible for the lowest achievable cost. The results should support and inform quality improvement efforts. Based on the NICCQ experience, we highlight four key features of a national quality monitoring system for cancer care (Table 1), discuss each, and provide recommendations. The key features are: (1) Use of a carefully designed sampling protocol to ensure that sampled patients are representative of the population of patients with the same cancer diagnosis. (2) Procedures for protecting the privacy and confidentiality of personal information. (3) Inclusion of rigorously developed measures of the quality of cancer care that are validated and updated regularly. (4) A comprehensive and ongoing data collection protocol that relies on at least three sources of available data.

We chose incident case sampling because the clinical care events that produce the greatest impact on the patient's outcome take place shortly after diagnosis and during initial therapy. Identification of patients soon after diagnosis makes it easier to locate them for consent and survey, and makes for more complete patient recall about salient providers and events. The downside of identifying patients soon after diagnosis is the additional time required to measure outcomes for long-term survivors.

The pros and cons of sampling from cancer registries, insurance rosters, institutional rosters, and large scale ongoing health surveys have been described in detail elsewhere.^23,24 Three well known cancer registries—the Surveillance Epidemiology and End Results registry (SEER), the national program of state-based registries recently initiated by the Centers for Disease Control (CDC), and the National Cancer Database (NCDB) operated by the ACS—allow identification of patients with incident diagnoses. For a variety of reasons, including the enthusiasm of ACS about participating, we selected the NCDB. The NCDB relies on voluntary reporting of data from hospital cancer registries throughout the United States. An important consequence of relying on voluntary reporting is that registries are subject to financial turmoil of the hospital marketplace. A handful of registries had merged or ceased to operate because of merger, acquisition, or closure of their sponsoring hospitals. Despite this, the NCDB obtains information on approximately 70% of all US cancer patients, a far larger proportion than could be obtained by alternative sources.^25,26

At the initiation of NICCQ, none of the registries were able to provide fully representative national samples of newly diagnosed patients in a short time frame (eg, < 6 months) without modification. There is a substantial delay between diagnosis, complete reporting, and entry of patient data in most current national registry programs. For example, the final NICCQ project sample could not be fully defined for survey until more than two years after diagnosis. Expanding the use of rapid case ascertainment, a procedure that enables local registries to report data on new diagnoses to a central registry in as little as 3 months, could address this problem, reducing the loss of information from patients who die, relocate, or change providers shortly after incident diagnosis. However, rapid case ascertainment would also require investment of new resources at central and local registries.

Protecting Privacy and Confidentiality
Recently enacted provisions of the Health Insurance Portability and Accountability Act have heightened concerns about the privacy and confidentiality of patient information. Inevitably, there is a significant tension between protecting the privacy and confidentiality of patients and obtaining the sample of patients and the data necessary to provide meaningful and comprehensive profiles of the quality of care.^27,28 This issue is especially pertinent for patients with a cancer diagnosis who may legitimately fear inadvertent disclosure of their diagnosis to others, merely because of their participation in quality assessment.²⁹

The NICCQ experience suggests that a properly designed and conducted quality monitoring system does not pose much risk to patients. For example, no patient approached by the NICCQ project team has complained that his or her cancer diagnosis, or other medical information, was inadvertently disclosed. This achievement required substantial effort, time, and resources dedicated to protecting the privacy and confidentiality of the patients we recruited. Obtaining institutional review board (IRB) approval from 65 participating hospital cancer registries, notifying physicians, obtaining patient participation and consent to survey, obtaining medical record consent, and retrieving medical records were as challenging as might be expected in any multi-institution cohort study. To maximize the efficiency of this process, the NICCQ project team obtained review and approval for a standard study protocol and set of consent forms from the Harvard and Rand IRBs and the National Cancer Institute's IRB. Despite this approval, local differences in IRB application forms, interpretations, and recommendations about the protocol and data collection instruments led to a high degree of customization for local institutions. Thirty-five of the 65 registry IRBs required modification to consent forms and 20 registry IRBs required modification to some part of the protocol. Responding to local IRB request added more than a year to the project timeline because of the need to negotiate and ensure consistency among potentially conflicting directives.

Most registries require notification of the patient's physician before contacting the patient. Such contacts were both common courtesy and helpful in identifying deceased patients or patients who might be distressed by a research contact, however, identifying the appropriate physician for this initial contact was not trivial. Registry data may list a physician that had little or no contact with the patient, thus defeating the purpose of notification. Six registry IRBs required active consent by the listed physician before we could contact the patient, and this proved to be a formidable barrier to recruitment. Twenty percent of patients in these registries could not be approached because of physician refusal (compared with 2% of patients from registries in which physicians were notified but not required to respond). The requirement for written consent from patients to review medical records is a potential (albeit unavoidable) barrier to quality monitoring, because the quality of care for patients who refuse or are unable to consent to review might differ from the quality of care for those who grant permission. During the NICCQ project, 88% of patients who completed the survey also provided consent to review medical records. Some patients (or surrogates) declined to complete the survey, but still consented to medical record review.

Obtaining the medical records of deceased patients was especially challenging. Federal human subjects protections do not extend to deceased patients; nevertheless, one of the NICCQ project states (Kansas) legally required next-of-kin permission to review medical records. Where statute was not an issue, institutional policies frequently required next-of-kin permission for any release of records. Locating next-of-kin to obtain signed consent forms is challenging, and becomes increasingly difficult as time elapses after the patient's death.

Quality Measures to Assess Cancer Care
There are significant methodological challenges to the design of quality measures.^1,23,30-34 The measures needed to assess the quality of care are similar in some ways to the measures used in clinical trials. Quality of care measures that examine processes (or "use") of care are analogous to adherence measures from the clinical trial setting. A monitoring system should include measures of underuse of effective diagnostic tests and therapies among patients who could benefit from those therapies. It should also assess overuse of therapies among patients with limited expectation of benefit. The extent of overuse in cancer care has not been established, and such measures are especially challenging to design. In this prototype effort, we collected data that may be useful to the future development of measures of overuse.

The NICCQ project team designed candidate measures of quality for breast cancer and colorectal cancer by reviewing scientific literature and guidelines for care that would have been pertinent to the year included patients received a first diagnosis of cancer (1998). We proposed measures organized in 7 domains shown in Table 2. Quality measures to assess clinical care typically focus on those processes of care for which there is scientific evidence that the process is associated with improved patient outcomes (eg, radiation for early stage breast cancer) or focus on salient outcomes. Tests or treatments that have been shown to be efficacious in the setting of a randomized controlled trial (RCT) are ideal for the development of quality measures. Alternatively, a high degree of consensus among providers or experts that the test or treatment is likely to be beneficial may substitute if evidence is limited, especially when affirmed in guidelines or consensus statements.^35-37 Outcomes used in RCTs such as mortality, survival, health status, and quality of life can also be the basis for quality measures if there is sufficient risk adjustment.

During the NICCQ project, we found that administering a patient survey as the first step in data collection addressed a number of logistical challenges. It allowed patients to consent to participate and to provide a written medical records release. Patients could also list the names and contact information for all of the hospitals, doctors, and other providers involved in their cancer care. This proved to be the most effective way to identify relevant medical records.

As a first-time effort, the NICCQ project was subject to delays in obtaining central registry data, obtaining approvals from IRBs, and retrieving local registry data. Because of these delays, surveys could not begin until 3 years had elapsed after initial diagnosis. These delays may have affected the completeness of sample and available data. Of 5,342 patients identified, 792 (15%) were deceased, and 793 (15%) were impossible to contact because a doctor refused (4%), a valid address was unavailable (5%), or another problem (such as cognitive impairment) made the patient ineligible for survey (5%). Among 3,757 eligible patients, 2,404 (64%) responded to survey. Of the 36% who were nonrespondents, 14% had no phone number to establish follow-up to mail survey, 22% refused, and 7% did not respond to attempts by mail and telephone. Among the respondents, 2,188 (88%) consented to medical record review. At least one medical record was obtained for 96% of the eligible patients who provided the names of their providers and had a valid consent form.

We suspect that patient willingness to participate in the survey and allow medical record review could be increased by initiating data collection shortly after diagnosis. The optimal timing of survey and medical record retrieval is not obvious because cancer care frequently requires intensive, ongoing treatments over many months. Data gathered shortly after initial surgical treatment may yield information about the experience of surgery, but would yield little or no information about adjuvant chemotherapy or long-term outcomes. Multiple surveys during care could be useful but costly to administer. Some patients are unavailable for survey, such as decedents, those with significant cognitive impairment, and those with a language barrier, and this unavailability could be associated with the quality of care; so nonresponse must be addressed carefully in the analysis.

Lessons From the NICCQ
The US Congress has mandated the creation of a National Health Care Quality Report (NHQR), which was recently published.³⁹ The NHQR currently includes measures of cancer incidence and mortality, but not detailed measures of the quality of cancer care.⁴⁰ Thus the challenge of the IOM report on the quality of cancer care remains unaddressed.

Based on our experience several lessons are worth highlighting. First, the development of quality measures has not been limited by a paucity of scientific evidence. Although cancer treatments have been subject to RCTs, not all aspects of the quality of caring for cancer patients have been evaluated in trials. Accordingly, it is also desirable to include measures that are based on guidelines, other consensus statements, and expert opinion about optimal care.¹ Developing measures of overuse of care will require convening panels of clinicians and other experts.³⁶ A policy-setting group representing multiple stakeholders will be needed to review and approve measures.

Second, all currently active cancer registries have shortcomings as sampling platforms for a national quality monitoring system. To support quality monitoring, each registry would need new resources. Time delays are a key barrier that currently impedes the collection of timely information about the quality of care. Registry strategies that include rapid case ascertainment, collect a few key clinical data elements, and make use of secured, standardized, electronic data collection and reporting could address this shortcoming.

Third, financial turmoil affecting health care institutions may have an adverse impact on a national monitoring system. For example, hospital registries may find it difficult to provide data on quality of care while fulfilling their core obligations. During the NICCQ project, mergers, acquisitions, and insolvencies among hospitals were major reasons for ineligibility and nonparticipation of 23 of the 88 hospital cancer registries we approached.

Fourth, the need to obtain patient consent implies that even the best sampling strategy cannot provide perfectly representative samples of patients. Participation and response rates could be enhanced by earlier case ascertainment and streamlined IRB processes. Ideally, informed consent could be obtained at the time of diagnosis, possibly at the time patients consent to a diagnostic procedure. Nevertheless, statistical adjustment will surely be necessary to address nonparticipation and nonresponse. Expanding the NICCQ prototype to a national scale without a process to encourage local IRBs to delegate review and approval to a central organization (such as the NCI review board) could place a considerable burden on the quality monitoring system and on literally thousands of independent institutions that should participate. We also suggest that next of kin consent requirements for decedents be waived.

Fifth, quality monitoring cannot have much impact on patients' outcomes unless there are accompanying efforts to improve quality. Our results will undoubtedly identify specific aspects of oncology practice that could be improved. Using the NICCQ results to stimulate quality improvement projects targeting selected aspects of oncology practice should enhance the value of the investment in NICCQ. Previous regional quality improvement initiatives could offer a useful framework for proceeding.^41-43

Finally, as our discussion implies, the implementation of a national quality monitoring system is not a trivial undertaking. It will require substantial resources, sustained commitment by a diverse group of professional organizations, institutions that provide cancer care, registry staff, clinicians, and patients. It may not be feasible to roll out a national program in the near future, but it may be both possible and prudent to build an ongoing prototype in selected states and expand it over time.

The NICCQ project is an important response to the charge of the IOM and a first step toward an ambitious goal. We anticipate that the analysis phase will demonstrate additional challenges in the production and interpretation of quality measurement results. The lessons we have learned to date, those that lie ahead, and the results of the project, should guide policy makers in the quest to design an optimal system of monitoring. Such a system can identify opportunities to improve care, suggest which quality improvement initiatives lead to better care, and monitor the impact of quality improvement on the outcomes for patients with cancer.

Appendix A

ASCO Quality Task Force: Joseph S. Bailes, MD; Lowell E. Schnipper, MD; Chandra Belani, MD; Thomas J. Smith, MD; Charles L. Bennett, MD; Glenn D. Steele, MD, PhD; Al B. Benson, III, MD; Ellen Stovall; Susan Braun; Jane Weeks, MD; Nancy E. Davidson, MD; William C. Wood, MD; Lawrence H. Einhorn, MD; Rodger Winn, MD; Arlene A. Forastiere, MD; Michael Goldstein, MD; Craig Henderson, MD; Bruce Hillner, MD; Margaret Kemeny, MD; Nancy E. Kemeny, MD; Patricia Legant, MD, PhD; Bernard Levin, MD; William McGivney, PhD; Monica Morrow, MD; Linda Mulvihill, CTR; Catherine Park, MD; David G. Pfister, MD; Lori J. Pierce, MD; Chris Rose, MD; Carolyn Runowicz, MD

Appendix B

Sponsors: Susan G. Komen Breast Cancer Foundation, founding sponsor; American Cancer Society; Amgen; AstraZeneca Oncology; Aventis Oncology; Bristol-Myers Squibb Oncology; Immunex Corporation; Lilly Oncology; Ortho Biotech Oncology; Pfizer; National Pharmaceutical Council

Authors' Disclosures of Potential Conflicts of Interest

The following authors or their immediate family members have indicated a financial interest. No conflict exists for drugs or devices used in a study if they are not being evaluated as part of the investigation. Performed contract work within the last 2 years: Eric C. Schneider, Jennifer L. Malin, Katherine L. Kahn, Ezekiel J. Emanuel, and Arnold M. Epstein, American Society of Clinical Oncology.

Acknowledgment

The authors acknowledge grants received from the American Society of Clinical Oncology and the Damon Runyon Foundation.

REFERENCES

1. Malin JL, Schuster MA, Kahn KA, et al: Quality of breast cancer care: What do we know? J Clin Oncol 20:4381-4393, 2002[Abstract/Free Full Text]

2. Ayanian JZ, Kohler BA, Abe T, et al: The relation between health insurance coverage and clinical outcomes among women with breast cancer. N Engl J Med 329:326-331, 1993[Abstract/Free Full Text]

3. Potosky AL, Merrill RM, Riley GF, et al: Prostate cancer treatment and ten-year survival among group/staff HMO and fee-for-service Medicare patients. Health Serv Res 34:525-546, 1999[Medline]

4. Schrag D, Cramer LD, Bach PB, et al: Influence of hospital procedure volume on outcomes following surgery for colon cancer. JAMA 284:3028-3035, 2000[Abstract/Free Full Text]

5. Schrag D, Gelfand SE, Bach PB, et al: Who gets adjuvant treatment for stage II and III rectal cancer? Insight from surveillance, epidemiology, and end results: Medicare. J Clin Oncol 19:3712-3718, 2001[Abstract/Free Full Text]

6. Bach PB, Cramer LD, Warren JL, et al: Racial differences in the treatment of early-stage lung cancer. N Engl J Med 341:1198-1205, 1999[Abstract/Free Full Text]

7. Bickell NA, Aufses AH Jr, Chassin MR: The quality of early-stage breast cancer care. Ann Surg 232:220-224, 2000[CrossRef][Medline]

8. Institute of Medicine, Commission On Life Sciences National Research Council: Ensuring quality cancer care. Washington, DC, National Academy Press, 1999

9. ACS (American College of Surgeons): Commission on Cancer, American College of Surgeons, 2004

10. Mettlin CJ, Menck HR, Winchester DP, et al: A comparison of breast, colorectal, lung, and prostate cancers reported to the National Cancer Data Base and the Surveillance, Epidemiology, and End Results Program. Cancer 79:2052-2061, 1997[CrossRef][Medline]

11. Eisenberg JM, Power EJ: Transforming insurance coverage into quality health care: Voltage drops from potential to delivered quality. JAMA 284:2100-2107, 2000[Abstract/Free Full Text]

12. Kahn KL, Rogers WH, Rubenstein LV, et al: Measuring quality of care with explicit process criteria before and after implementation of the DRG-based prospective payment system. JAMA 264:1969-1973, 1990[Abstract]

13. Bates D, Cullen D, Laird N, et al: Incidence of adverse drug events and potential adverse drug events: Implications for prevention—ADE Prevention Study Group. JAMA 271:29-34, 1995

14. Schneider EC, Zaslavsky AM, Epstein AM: Racial disparities in the quality of care for enrollees in Medicare managed care. JAMA 287:1288-1294, 2002[Abstract/Free Full Text]

15. McGlynn EA, Asch SM, Adams J, et al: The quality of health care delivered to adults in the United States. N Engl J Med 348:2635-2645, 2003[Abstract/Free Full Text]

16. Epstein AM, Ayanian JZ, Keogh JH, et al: Racial disparities in access to renal transplantation—clinically appropriate or due to underuse or overuse? N Engl J Med 343:1537-1544, 2000[Abstract/Free Full Text]

17. Bennett CL, Smith TJ, Weeks JC, et al: Use of hematopoietic colony-stimulating factors: The American Society of Clinical Oncology survey: The Health Services Research Committee of the American Society of Clinical Oncology. J Clin Oncol 14:2511-2520, 1996[Abstract]

18. Bennett CL, Bishop MR, Tallman MS, et al: The association between physician reimbursement in the US and use of hematopoietic colony stimulating factors as adjunct therapy for older patients with acute myeloid leukemia: Results from the 1997 American Society of Clinical Oncology survey: Health Services Research Committee of the American Society of Clinical Oncology. Ann Oncol 10:1355-1359, 1999[Abstract/Free Full Text]

19. Kirkwood JM, Atkins MB, Johnson ER, et al: Measuring standards of care for early breast cancer in an insured population. J Clin Oncol 15:2351-2358, 1997[Abstract/Free Full Text]

20. Krzyzanowska MK, Weeks JC, Earle CC: Treatment of locally advanced pancreatic cancer in the real world: Population-based practices and effectiveness. J Clin Oncol 21:3409-3414, 2003[Abstract/Free Full Text]

21. Brook RH, McGlynn EA, Cleary PD: Quality of health care. Part 2: measuring quality of care. N Engl J Med 335:966-970, 1996[Free Full Text]

22. Lipscomb J, Snyder CF: The outcomes of cancer outcomes research: Focusing on the National Cancer Institute's quality-of-care initiative. Med Care 40:III3-III10, 2002 (suppl 6)[Medline]

23. Kahn KL, Malin JL, Adams J, et al: Developing a reliable, valid, and feasible plan for quality-of-care measurement for cancer: How should we measure? Med Care 40:III73-III85, 2002 (suppl 6)[Medline]

24. Pearson ML, Ganz PA, McGuigan K, et al: The case identification challenge in measuring quality of cancer care. J Clin Oncol 20:4353-4360, 2002[Abstract/Free Full Text]

25. Menck HR, Blankenship C, Fremgen AM: The National Cancer Data Base and physician network. Top Health Inf Manage 17:45-59, 1997[Medline]

26. Menck HR, Cunningham MP, Jessup JM, et al: The growth and maturation of the National Cancer Data Base. Cancer 80:2296-2304, 1997[CrossRef][Medline]

27. Gostin LO, Turek-Brezina J, Powers M, et al: Privacy and security of personal information in a new health care system. JAMA 270:2487-2493, 1993[CrossRef][Medline]

28. Gostin LO: National Health Information Privacy: Regulations under the Health Insurance Portability and Accountability Act. JAMA 285:3015-3021, 2001[Abstract/Free Full Text]

29. American Society of Clinical Oncology Working Group on Genetic Testing for Cancer Susceptibility: American Society of Clinical Oncology policy statement update: Genetic testing for cancer susceptibility. J Clin Oncol 21:2397-2406, 2003[Abstract/Free Full Text]

30. McGlynn EA, Asch SM: Developing a clinical performance measure. Am J Prev Med 14:14-21, 1998[CrossRef][Medline]

31. American College of Radiology: Appropriateness Criteria for Imaging and Treatment Decisions, American College of Radiology, 1998

32. Liu H, Kaplan AH, Wenger NS: Measuring patient adherence. Ann Intern Med 137:72-73, 2002[Free Full Text]

33. Shekelle PG: Are appropriateness criteria ready for use in clinical practice? N Engl J Med 344:677-678, 2001[Free Full Text]

34. Spencer BA, Steinberg M, Malin J, et al: Quality-of-care indicators for early-stage prostate cancer. J Clin Oncol 21:1928-1936, 2003[Abstract/Free Full Text]

35. Winn RJ, Botnick W, Bitsura JA: Evaluation of preliminary NCCN guidelines by external review. Oncology (Huntingt) 10:311-315, 1996

36. Shekelle PG, Kahan JP, Bernstein SJ, et al: The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med 338:1888-1895, 1998[Abstract/Free Full Text]

37. Leape L, Park RE, Kahan JP, et al: Group judgments of appropriateness: Effect of panel composition. Qual Assur Health Care 4:151-159, 1992[Medline]

38. Malin JL, Kahn KL, Adams J, et al: Validity of cancer registry data for measuring the quality of breast cancer care. J Natl Cancer Inst 94:835-844, 2002[Abstract/Free Full Text]

39. Institute of Medicine: Envisioning the National Health Care Quality Report. Washington, DC, National Academy Press, 2001

40. AHRQ: Preliminary Measure Set, National Healthcare Quality Report, AHRQ 2002

41. Kiefe CI, Allison JJ, Williams OD, et al: Improving quality improvement using achievable benchmarks for physician feedback: A randomized controlled trial. JAMA 285:2871-2879, 2001[Abstract/Free Full Text]

42. Hannan EL, Kilburn H Jr, Racz M, et al: Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 271:761-766, 1994[Abstract]

43. O'Connor GT, Plume SK, Olmstead EM, et al: A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. JAMA 275:841-846, 1996[Abstract]

This article has been cited by other articles:

				E. C. Schneider, M. R. Nadel, A. M. Zaslavsky, and E. A. McGlynn Assessment of the Scientific Soundness of Clinical Performance Measures: A Field Test of the National Committee for Quality Assurance's Colorectal Cancer Screening Measure Arch Intern Med, April 28, 2008; 168(8): 876 - 882. [Abstract] [Full Text] [PDF]

				J. O. Jacobson, M. N. Neuss, K. K. McNiff, P. Kadlubek, L. R. Thacker II, F. Song, P. D. Eisenberg, and J. V. Simone Improvement in Oncology Practice Performance Through Voluntary Participation in the Quality Oncology Practice Initiative J. Clin. Oncol., April 10, 2008; 26(11): 1893 - 1898. [Abstract] [Full Text] [PDF]

				R. Saginur, S. F. Dent, L. Schwartz, R. Heslegrave, S. Stacey, and J. Manzo Ontario Cancer Research Ethics Board: Lessons Learned From Developing a Multicenter Regional Institutional Review Board J. Clin. Oncol., March 20, 2008; 26(9): 1479 - 1482. [Abstract] [Full Text] [PDF]

				E. L. Towle, D. C. Fryefield, and J. V. Cox Good to Great: Exploring Ways to Improve Our Practice ASCO Educational Book, January 1, 2008; 2008(1): 486 - 491. [Abstract] [Full Text] [PDF]

				M. J. Leonardi, M. L. McGory, and C. Y. Ko Quality of Care Issues in Colorectal Cancer Clin. Cancer Res., November 15, 2007; 13(22): 6897s - 6902s. [Abstract] [Full Text] [PDF]

				J. Lipscomb, C. C. Gotay, and C. F. Snyder Patient-reported Outcomes in Cancer: A Review of Recent Research and Policy Initiatives CA Cancer J Clin, September 1, 2007; 57(5): 278 - 300. [Abstract] [Full Text] [PDF]

				M. L. McGory, D. S. Zingmond, E. Sekeris, and C. Y. Ko The Significance of Inadvertent Splenectomy During Colorectal Cancer Resection Arch Surg, July 1, 2007; 142(7): 668 - 674. [Abstract] [Full Text] [PDF]

				B. N. Polite, J. J. Dignam, and O. I. Olopade Colorectal Cancer Model of Health Disparities: Understanding Mortality Differences in Minority Populations J. Clin. Oncol., May 10, 2006; 24(14): 2179 - 2187. [Abstract] [Full Text] [PDF]

				J. L. Malin, E. C. Schneider, A. M. Epstein, J. Adams, E. J. Emanuel, and K. L. Kahn Results of the National Initiative for Cancer Care Quality: How Can We Improve the Quality of Cancer Care in the United States? J. Clin. Oncol., February 1, 2006; 24(4): 626 - 634. [Abstract] [Full Text] [PDF]

				J. M. Jessup, A. Stewart, F. L. Greene, and B. D. Minsky Adjuvant Chemotherapy for Stage III Colon Cancer: Implications of Race/Ethnicity, Age, and Differentiation JAMA, December 7, 2005; 294(21): 2703 - 2711. [Abstract] [Full Text] [PDF]

				A. C. Wolff and C. E. Desch Clinical Practice Guidelines in Oncology: Translating Evidence Into Practice (and back) J. Oncol. Pract, November 1, 2005; 1(4): 160 - 161. [Full Text] [PDF]

				B. K. Edwards, M. L. Brown, P. A. Wingo, H. L. Howe, E. Ward, L. A. G. Ries, D. Schrag, P. M. Jamison, A. Jemal, X. C. Wu, et al. Annual Report to the Nation on the Status of Cancer, 1975-2002, Featuring Population-Based Trends in Cancer Treatment J Natl Cancer Inst, October 5, 2005; 97(19): 1407 - 1427. [Abstract] [Full Text] [PDF]

				J. L. Malin and N. L. Keating The Cost-Quality Trade-Off: Need for Data Quality Standards for Studies That Impact Clinical Practice and Health Policy J. Clin. Oncol., July 20, 2005; 23(21): 4581 - 4584. [Full Text] [PDF]

				M. K. Krzyzanowska, J. Treacy, B. Maloney, A. Lavino, and J. O. Jacobson Development of a Patient Registry to Evaluate Hospital Admissions Related to Chemotherapy Toxicity in a Community Cancer Center J. Oncol. Pract, May 1, 2005; 1(1): 15 - 19. [Abstract] [Full Text] [PDF]

				J. Lipscomb, M. S. Donaldson, N. K. Arora, M. L. Brown, S. B. Clauser, A. L. Potosky, B. B. Reeve, J. H. Rowland, C. F. Snyder, and S. H. Taplin Cancer Outcomes Research J Natl Cancer Inst Monographs, October 1, 2004; 2004(33): 178 - 197. [Full Text] [PDF]

This Article Full Text (PDF) Erratum (v22,p4656) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Schneider, E. C. Articles by Emanuel, E. J. Search for Related Content PubMed PubMed Citation Articles by Schneider, E. C. Articles by Emanuel, E. J.