|
Advertisement |
|
|
||
 |
|
|||||
|
|||||
|
 |
||||||
|
||||||
|
||||||
|
Originally published as JCO Early Release 10.1200/JCO.2005.01.912 on April 25 2005 © 2005 American Society of Clinical Oncology.
The Cost-Quality Trade-Off: Need for Data Quality Standards for Studies That Impact Clinical Practice and Health Policy
1 Department of Medicine and Jonsson Comprehensive Cancer Center, David Geffen School of Medicine at University of California Los Angeles, Los Angeles, CA; RAND Corp, Santa Monica, CA Observational studies of cancer patients often rely on existing administrative data such as insurance claims and cancer registry data.1 Such studies provide a grainy snapshot of the use of cancer services such as surgery, radiation therapy, and chemotherapy, and allow for general comparisons of outcomes across different cohorts of patients. As researchers began to supplement administrative data with patient self-reported data or medical record review, limitations in the validity of these easily obtainable data sets began to emerge.2,3 For example, what were once thought to be global problems in the quality of breast cancer care appear to be largely the result of imprecise measurement.4,5 However, as our ability to accurately capture patient interactions with the health care system improves, so has our awareness of the importance of measuring details that simply go beyond whether a patient received radiation therapy after breast-conserving surgery or was prescribed tamoxifen.6,7 Recent efforts, such as the American Society of Clinical Oncology (ASCO) –sponsored National Initiative for Cancer Care Quality (NICCQ) and the National Cancer Institute's Cancer Care Outcomes Research and Surveillance Consortium (CanCORS) explore in greater detail than ever before the process and outcomes of cancer care in the United States.8,9 NICCQ is evaluating quality of care for breast and colorectal cancer using patient self-report three years after diagnosis as well as medical record data to assess adherence to 61 explicit process measures in the domains of data gathering, initial management, management of treatment toxicity, referrals, respect for patient preferences, and surveillance.8 The CanCORS Consortium is surveying newly diagnosed lung and colorectal patients or their surrogates at four and twelve months after diagnosis, abstracting data from medical records, and surveying the physicians delivering care.9 These intensive efforts to obtain accurate data from population-based cohorts about their myriad interactions with the health care system across the continuum of care will provide a picture of cancer care of unprecedented detail for researchers, policy-makers, clinicians, and patient advocates. However, this degree of precision comes at a high price—ASCO's NICCQ initiative cost an estimated $5 million,10 and NCI anticipates awards in excess of $34 million for the CanCORS Consortium.11 Clearly, most research studies will not be able to afford the detailed data collection strategies used by these two efforts. Therefore, it is important to consider the accuracy of data obtained from various sources and whether trading highly accurate data for lower costs is acceptable, based on the intended use of the data. In this issue of the Journal of Clinical Oncology, Phillips et al report the results of a methodological study comparing data about breast cancer treatment and recurrence from two sources: patient self-report and medical record abstraction, the latter of which is considered an accurate source of data but generally costly.12 They found that patients could very accurately report on the general type of treatment that they received (sensitivity of 100%, 99%, 99%, and 92% for surgery, radiation therapy, chemotherapy, and hormonal therapy, respectively). However, patients were less able to accurately report the technical details of treatment (sensitivity 56% to 98% and specificity 44% to 99%) or cancer recurrence (sensitivity 89% and specificity 98%). Nevertheless, these findings are encouraging for population-based researchers. Not only was there high concordance between patients' reports of breast cancer treatments and treatments identified by medical record review for many elements of care, the median recall period was 3.2 years, suggesting that such reports are reliable for at least a few years after diagnosis, though accuracy for some aspects of care declined with increasing time from diagnosis. Despite these generally promising results, the study's findings should be interpreted with some caution. The study included patients who were participants in a family study of the genetic, environmental, and lifestyle factors associated with breast cancer. These women were younger than most breast cancer patients (only 1% was age 60 or older) and, in light of their age and interest in the family breast cancer study, may have been more knowledgeable about their disease than the typical breast cancer patient.13 In addition, as with any survey, the potential for response bias must be considered,14-19 since participation may depend on a potential respondent's level of interest and availability. More recently, institutional requirements for permission from a patient's physician before contacting the patient for participation (as required in this study) or requirements for active patient consent have adversely impacted the participation of patients in survey-based research.20,21 In the study by Phillips et al,12 participants were more likely than nonparticipants to have been diagnosed more recently, to have been born in Australia, and to be married. If women who are native born also report breast cancer treatments more accurately than other women,12 studies of breast cancer care that use survey data could lead to biased conclusions about populations of patients who are not native born. Cost-Quality Tradeoffs and Uses of Data Information about patients' illnesses and treatments can be collected from a variety of sources. For breast cancer, these sources include cancer registry data, administrative data (including billing data for visits and treatments, pharmacy data, and hospital discharge data), patient self-report, medical records (from hospitals and physicians' offices), and physician report. Decisions about the use of each of these data sources often necessitate prioritizing the desired information, the availability and cost of obtaining the data, the expected validity of the source and the intended use of the data.22 Cancer registry and administrative data are often readily available at reasonable costs, but may be limited to the peridiagnosis period or may lack clinical detail. Patient and physician survey data are more expensive and are subject to response bias, which may limit generalizability, but may be the only reliable source for some information, such as patients' experiences with care. Medical record data are often the most expensive to collect and are typically quite accurate, but may suffer from incomplete documentation, or inability to locate complete patient records that may be in geographically distinct sites of care. The best data source may vary depending on the information sought,23,24 and the most complete data collection strategy typically combines data from various sources.9 When weighing the advantages and disadvantages of data from various sources, the intended use for the data becomes an important consideration. Some uses of data may require less accuracy than others. But in which situations might we consider compromising on data accuracy to improve ease of data acquisition and minimize costs? One use of data about cancer patients' disease and primary treatment is for hypothesis generation and to identify areas for further research. For this purpose, data need not be highly accurate, as treatment decisions are not likely to be made on the basis of these results alone. Data about breast cancer and its treatments may also be collected for epidemiological studies. In this case, it may be reasonable to tolerate less-than-complete accuracy, as treatment decisions are also not likely to be made on the basis of these results alone. This is particularly true if the data collected are primarily used as control variables and the accuracy does not vary across the groups being compared. If data accuracy does vary with the groups being compared, though, the potential for bias should be explored.4,24 For example, if patients who are better educated report care more accurately,12 studies of breast cancer care that rely on patient survey data could lead to biased conclusions about less well-educated patients. Cancer registry data and administrative data are used frequently in epidemiologic studies and for describing patterns of cancer care, and the findings reported by Phillips et al support the use of patient self-report data in such studies as well. However, studies focused on understanding the etiology of variation in care and treatment disparities may require more accurate data, since the potential for bias is real and because such studies are more likely to influence policies and clinical practice. Similarly, studies that collect utilization data to inform cost-effectiveness analyses may also require more accurate data, given their use in policy making. Data about cancer and its treatment are also used to evaluate the effectiveness of treatment outside of the controlled environment of a research study. Effectiveness studies require both confidence in the accuracy of data as well as detailed clinical information. For example, to evaluate whether adjuvant chemotherapy for stage III colorectal cancer is as effective in the community as predicted by randomized controlled trials requires not only information regarding the initiation of chemotherapy, but ideally also data on the doses received, whether or not treatment was completed and treatment toxicity. Although patients can accurately report receipt of chemotherapy, Phillips et al found that patients were much less able to correctly report the chemotherapy regimen received12 and were even less likely to know the doses of chemotherapy received. Patient self-report will therefore be of limited value for effectiveness studies, although it remains an excellent source of data on treatment toxicity. A final important use of data on cancer care is for quality measurement. Quality data can be used for quality improvement,26,27 to provide information for purchasers of health care,28-31 public reporting and accountability,32-34 and, more recently, for provider payment.35-38 Data used for internal quality improvement efforts may not require the same level of accuracy as needed for these other purposes, since the potential for negative consequences of poor and inaccurate data increase as data are more freely shared with external parties. Highly accurate data are particularly important when used for public reporting and provider compensation. Recognizing this, major national efforts to provide public data to employers and consumers, such as the National Committee for Quality Assurance's Health Plan Employer Data and Information Set,31 or to compensate providers based on quality, such as that planned by the Centers for Medicare and Medicaid Services37 specify that administrative data be supplemented with medical record data for quality measurement. Need for Data Quality Standards for Observational Studies That Impact Clinical Practice or Health Policy Given the variety of available data sources and the differing uses of data, minimum standards of acceptable data quality are needed to assure the validity of data collection efforts with the potential to influence clinical care or policy. The Impact Pyramid proposed by Tunis and Stryer for evaluating the impact of outcomes research on health policy25 may provide a useful framework for developing criteria for data quality based on the intended use of the data. The Tunis and Stryer framework for research studies takes the form of a four-level pyramid (Fig 1). At the base of the pyramid (level 1) are studies that contribute to our knowledge base and future research but do not directly impact on medical practice. The second level includes research that changes policy or results in the creation of a new program. Research that impacts the actual delivery of health care in a measurable way is the next level (level 3) and research that influences patient outcomes in the community is demonstrable is the top level of the pyramid (level 4).
While Tunis and Stryer used this Impact Pyramid to retrospectively assess the observed impact of outcomes research at AHCPR,25 it could be applied prospectively when evaluating the evidence of research supporting health policy decisions.39 Although the Impact Pyramid was developed primarily for research studies, it can be expanded to consider data collection efforts that are not directly related to research but may impact on policy or patient care, such as quality improvement efforts. Criteria for data reliability and validity and other methodologic considerations could be developed corresponding to each level of the Impact Pyramid (Figure). Adoption of these criteria in policy analysis would institutionalize a structured review of the data similar to the impact of grading of medical evidence, which has became standard practice in the development of clinical practice guidelines.40-42 Decisions made by payors, lawmakers and other stakeholders have the potential for tremendous impact on health outcomes and should be made with the best available evidence.43,44 Standards exist for reporting data from clinical studies (ie, CONSORT, MOOSE, STARD, QUORUM)45-48 and methods have been developed to evaluate the quality of these studies when making recommendations about clinical policy, such as practice guidelines and consensus statements.40-42 Similar criteria are urgently needed for observational studies that are used to guide health policy decisions. Authors' Disclosures of Potential Conflicts of Interest The authors indicated no potential conflicts of interest.
REFERENCES 1. Board NCP, Institute of Medicine, Hewitt M, Simone J: Ensuring Quality Cancer Care. Washington, DC, National Academy Press, 1999, pp i-236 2. Board NCP, Medicine Io, Commission on Life Sciences NRC: Enhancing Data Systems to Improve the Quality of Cancer Care. Washington, DC, National Academy Press, 2000
3. Malin JL, Schuster MA, Kahn KA, et al: Quality of breast cancer care: What do we know? J Clin Oncol 20:4381-4393, 2002
4. Malin JL, Kahn KL, Adams J, et al: Validity of cancer registry data for measuring the quality of breast cancer care. J Natl Cancer Inst 94:835-844, 2002 5. Schneider EC, Malin JL, Kahn KL, et al: Final report of the national initiative on cancer care quality (NICCQ). American Society of Clinical Oncology Annual Meeting. New Orleans, LA, 2004 6. Lipscomb J, Donaldson MS: Outcomes research at the National Cancer Institute: Measuring, understanding, and improving the outcomes of cancer care. Clin Ther 25:699-712, 2003[CrossRef][Medline] 7. Freeman HP C: Voices of a broken system: Real people, real problems, in Reuben S (ed): President's Cancer Panel Report of the Chairman 2000-2001. Bethesda, MD, National Cancer Institute, National Institutes of Health, 2002
8. Schneider EC, Epstein AM, Malin JL, et al: Developing a system to assess the quality of cancer care: ASCO's national initiative on cancer care quality. J Clin Oncol 22:2985-2991, 2004
9. Ayanian JZ, Chrischilles EA, Wallace RB, et al: Understanding cancer treatment and outcomes: The Cancer Care Outcomes Research and Surveillance Consortium. J Clin Oncol 22:2992-2996, 2004 10. FAQs: ASCO's National Initiative on Cancer Care Quality (NICCQ), American Society of Clinical Oncology. http://www.asco.org/ac/1,1003,_12-002177-00_18-0011725-00_19-0011728-00_20-001,00.asp 11. Cancer Care Outcomes Research and Surveillance Consortium., in Sciences. CCaP (ed): Health Services and Economics. Bethesda, MD, National Cancer Institute. http://healthservices.cancer.gov/cancors
12. Phillips KA, Milne RL, Buys S, et al: Agreement between self-reported breast cancer treatment and medical records in a population based breast cancer family registry. J Clin Oncol 23:4679-4686, 2005 13. Finney LJ, Iannotti RJ: The impact of family history of breast cancer on women's health beliefs, salience of breast cancer family history, and degree of involvement in breast cancer issues. Women Health 33:15-28, 2001[CrossRef][Medline] 14. Mazor KM, Clauser BE, Field T, et al: A demonstration of the impact of response bias on the results of patient satisfaction surveys. Health Serv Res 37:1403-1417, 2002[CrossRef][Medline] 15. Solberg LI, Beth Plane M, Brown RL, et al: Nonresponse bias: Does it affect measurement of clinician behavior? Med Care 40:347-352, 2002[CrossRef][Medline] 16. Partin MR, Malone M, Winnett M, et al: The impact of survey nonresponse bias on conclusions drawn from a mammography intervention trial. J Clin Epidemiol 56:867-873, 2003[CrossRef][Medline] 17. Kahn KL, Liu H, Adams JL, et al: Methodological challenges associated with patient responses to follow-up longitudinal surveys regarding quality of care. Health Serv Res 38:1579-1598, 2003[CrossRef][Medline]
18. Voigt LF, Koepsell TD, Daling JR: Characteristics of telephone survey respondents according to willingness to participate. Am J Epidemiol 157:66-73, 2003 19. Ronckers C, Land C, Hayes R, et al: Factors impacting questionnaire response in a Dutch retrospective cohort study. Ann Epidemiol 14:66-72, 2004[CrossRef][Medline] 20. Gurwitz JH, Guadagnoli E, Landrum MB, et al: The treating physician as active gatekeeper in the recruitment of research subjects. Med Care 39:1339-1344, 2001[CrossRef][Medline] 21. Nelson K, Garcia RE, Brown J, et al: Do patient consent procedures affect participation rates in health services research? Med Care 40:283-288, 2002[CrossRef][Medline] 22. Kahn KL, Malin JL, Adams J, et al: Developing a reliable, valid, and feasible plan for quality-of-care measurement for cancer: How should we measure? Med Care 40:III73-III85, 2002[Medline]
23. Fowles JB, Rosheim K, Fowler EJ, et al: The validity of self-reported diabetes quality of care measures. Int J Qual Health Care 11:407-412, 1999 24. Keating NL, Landrum MB, Landon BE, et al: Measuring the quality of diabetes care using administrative data: Is there bias? Health Serv Res 38:1529-1545, 2003[CrossRef][Medline] 25. Tunis S, Stryer D: The outcomes of outcomes research at AHCPR., Agency for Healthcare Research and Quality, 1999 26. Petitti DB, Contreras R, Ziel FH, et al: Evaluation of the effect of performance monitoring and feedback on care process, utilization, and outcome. Diabetes Care 23:192-196, 2000[Abstract]
27. Kiefe CI, Allison JJ, Williams OD, et al: Improving quality improvement using achievable benchmarks for physician feedback: A randomized controlled trial. JAMA 285:2871-2879, 2001
28. Epstein A: Performance reports on quality–prototypes, problems, and prospects. N Engl J Med 333:57-61, 1995
29. Iglehart JK: The National Committee for Quality Assurance. N Engl J Med 335:995-999, 1996
30. Marshall MN, Shekelle PG, Leatherman S, et al: The public release of performance data: What do we expect to gain? A review of the evidence. JAMA 283:1866-1874, 2000 31. National Committee for Quality Assurance.The state of managed care quality 2000. Washington, DC, National Committee for Quality Assurance, 2000
32. Epstein AM: Public release of performance data: A progress report from the front. JAMA 283:1884-1886, 2000
33. Jencks SF, Cuerdon T, Burwen DR, et al: Quality of medical care delivered to Medicare beneficiaries: A profile at state and national levels. JAMA 284:1670-1676, 2000 34. Bost JE: Managed care organizations publicly reporting three years of HEDIS measures. Manag Care Interface 14:50-54, 2001
35. Pedersen CA, Rich EC, Kralewski J, et al: Primary care physician incentives in medical group practices. Arch Fam Med 9:458-462, 2000 36. Sussman AJ, Fairchild DG, Coblyn J, et al: Primary care compensation at an academic medical center: A model for the mixed-payer environment. Acad Med 76:693-699, 2001[Medline] 37. Tieman J: Experimenting with quality. CMS-Premier initiative to reward best, punish worst. Mod Healthcare 33:6, 2003 38. Peterson ED: Should we link payment to quality? Am Heart J 148:S56-S58, 2004[CrossRef][Medline] 39. Lipscomb J, Donaldson MS, Hiatt RA: Cancer Outcomes Research and the Arenas of Application. J Natl Cancer Inst Monographs 33:1-7, 2004 40. West SL, King V, Carey TSea: Systems to Rate the Strength of Scientific Evidence. Rockville, MD, Agency for Healthcare Research and Quality, 2002 41. Harris RP HM, Woolf SH, et al: Current methods of the US Preventive Services Task Force: A review of the process. Am J Prev Med 20:21-35, 2001
42. Lohr KN: Rating the strength of scientific evidence: Relevance for quality improvement programs. Int J Qual Health Care 16:9-18, 2004 43. Dobrow MJ, Goel V, Upshur RE: Evidence-based health policy: Context and utilisation. Soc Sci Med 58:207-217, 2004 44. Cangialose CB, Blair AE, Borchardt JS, et al: Purchasing oncology services. Kerr L. White Institute/American Cancer Society Task Force on Purchasing Oncology Services. Cancer 88:2876-2886, 2000[CrossRef][Medline] 45. Moher D, Cook DJ, Eastwood S, et al: Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354:1896-1900, 1999[CrossRef][Medline]
46. Moher D, Schulz KF, Altman D: The CONSORT statement: Revised recommendations for improving the quality of reports ofparallel-group randomized trials. JAMA 285:1987-1991, 2001
47. Stroup DF BJ, Morton SC, et al: Meta-analysis of observational studies in epidemiology: A proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283:2008-2012, 2000 48. Group TS: The STARD Initiative–Towards Complete and Accurate Reporting of Studies on Diagnostic Accuracy, 2001 Related Article
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||
|
|||||||||||
|
|
Copyright © 2005 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|
