|
Advertisement |
|
|
||
 |
|
|||||
|
|||||
|
 |
||||||
|
||||||
|
||||||
|
Originally published as JCO Early Release 10.1200/JCO.2008.19.7756 on March 9 2009 © 2009 American Society of Clinical Oncology.
Evaluating the Impact of Organizational Changes in Health Care Delivery: Challenges in Study DesignDana-Farber Cancer Institute, Boston, MA The tenets of evidence-based medicine are by now so canonic that even organizational interventions to improve health outcomes require systematic evaluation to demonstrate their value. Of course, the most compelling strategy for establishing benefit is the randomized trial. In most cases, this is a welcome advance, one made in recognition of the reality that even interventions posing minimal health risks involve the use of scarce resources, the depletion of which can be measured in lost time, dollars, or opportunities. Behavioral and organizational interventions—such as installation of an exercise program or employment of patient navigators by a practice—typically target units larger than the individual. In this context, researchers are increasingly employing a design strategy known as the group randomized trial, also called the cluster randomized trial. Cluster randomized trials make sense when random assignment of individual patients is compromised because shared exposure may lead to contamination and thereby potential for resulting bias. Consider the example of an educational intervention carried out by infusion room chemotherapy nurses to foster adherence to antiemetic regimens. Randomization at the individual patient level might prompt providers with knowledge of the study intervention to provide their patients with additional education. Similarly, patients not exposed directly to the intervention might learn about it second hand from other study participants in the waiting room. Randomization at the level of the clinic mitigates these potential biases, but it creates other design challenges. In this issue of Journal of Clinical Oncology (JCO), Passalacqua et al1 describe the results of a cluster randomized trial designed to minimize anxiety and satisfy information needs of more than 3,000 patients with cancer treated at 38 Italian hospitals. Passalacqua et al developed an intervention to address the information needs of oncology inpatients in Italian hospitals. This intervention, known as the Point of Information and Support (PIS) system, was motivated by survey research evidence suggesting that information deficits were a source of considerable anxiety, which often led patients to adopt alternative unproven medical treatments. The PIS system involved creating a library with information materials and training hospital nurses to staff a resource room adjacent to the hospital ward. Each hospital was required to meet minimum staffing, opening hours, and publicity requirements but otherwise had considerable leeway in designing and implementing its own PIS system. This flexibility in design is pragmatic in that it allows for local variation in patterns of care delivery; however, the lack of standardization also creates a vulnerability that proved severe in this case. Six months after the PIS systems were integrated into the hospitals, more than 3,000 patients rated their satisfaction with information provided during their hospitalization according to the Hospital Anxiety and Depression Scale. Passalacqua et al1 found no differences in levels of anxiety, depression, or satisfaction between patients treated at the 21 hospitals that did and the 17 hospitals that did not implement a PIS system in the context of this study. JCO readers are likely to empathize with the frustration that these investigators must have felt when their intervention failed to improve outcomes. The main reason the study results were negative was that only 10 of the 21 hospitals assigned to develop a PIS system actually did so properly. This study failed to improve outcomes because the intervention could not be implemented or standardized across sites. The enormous amount of effort that went into the design of this study1 prompts us to question how this trial could have been designed differently to avoid such a disappointing result. What alternative strategies might have been pursued to optimize the value of the information gained? Passalacqua et al are to be congratulated for applying the same critical and systematic approach to evaluation of an organizational intervention that we apply to treatment trials. However, their study and its pitfalls highlight some important challenges confronted when evaluating organizational interventions. These issues merit consideration before the launching of a cluster randomized trial. Interventions vary enormously in the degrees of risk and cost that they create for patients and health care systems. In contrast to administering chemotherapy, which is high cost and high risk, placing a box of tissues in each examination room is a low-risk, low-cost intervention. The tissues are likely to be harmless to most patients, and even if rarely used, their relatively low cost means that the resources expended to keep the examination rooms stocked are unlikely to detract significantly from the ability of clinics to pursue other interventions of priority. No additional resources are required to maintain tissue boxes in good condition, and there are negligible adverse consequences if they go unused. This absurd example illustrates that some interventions are so innocuous that they do not merit investment of resources requisite for evaluation in the context of a trial. A logical first question to ask is whether the intervention under consideration is so similar to a box of tissues that it is reasonable to deploy it immediately and evaluate patterns of use downstream. In the case of the study by Passalacqua et al,1 if training clinical staff to attend to unmet patient psychological and information needs could have been accomplished without substantial investment of additional clinical resources, it is conceivable that the cost and risk involved could have been low enough to make the intervention resemble the proverbial box of tissues. Conducting in-depth qualitative research interviews at the Italian clinics before the creation of PIS systems might have helped Passalacqua et al1 identify local barriers to implementation or local needs that could have informed design of the randomized trial.2 Qualitative interviews might have illuminated the reasons behind the resistance of some clinic sites to patient libraries. They might have identified viewpoints of key clinic leaders as well as opposition or barriers to implementation of the PIS system. Even when interventions involve taking medications with ingredients and dosages that can be standardized across sites, we know that treatment administration can vary as a result of differing instructions and interpretation of study protocols. Behavioral and organizational interventions are particularly prone to variance in how they are implemented across study sites. When evaluating the impact of an organizational or behavioral intervention, investigators must consider how faithfully the intervention can be reproduced across study sites. The result of the randomized trial by Passalacqua et al1 was also negative because a large effect size was postulated. Evaluation of study outcomes before and after the launch of a PIS system at a single center could have provided investigators with some reasonable preliminary estimates of the magnitude of the impact of the intervention. In this case, knowing how much implementation of a PIS system had improved patient satisfaction and anxiety or depression at several types of clinics would have allowed a preliminary estimate of effect size and thereby informed the sample-size calculation. Of course, pre- and postintervention assessment is a flawed approach if change in outcomes results from clinic interventions other than the PIS system. Nevertheless, some compelling preliminary evidence that an intervention could improve outcomes is advantageous. Typically, the decision to conduct a randomized clinical trial is made after considering the strength of preliminary data and requisite sample size and prioritizing competing research proposals. The expected value of information approach uses methods from econometrics and decision analysis to formalize this process.3 Specifically, it involves considering the cost of not having the information to be gained from the trial as well the value of perfect information. By using structured equations to estimate the consequences of proceeding or not proceeding with a study in terms of both lives and monetary resources, this approach helps to make the tradeoffs explicit. Typically, sensitivity analyses are performed across the range of plausible estimates for unknown parameters to approximate the expected value of the research. In the study by Passalacqua et al,1 information about the cost of developing and maintaining each PIS system as well as the consequences of having anxious and ill-informed patients would have had to be estimated. Increasingly, this approach is being used to prioritize clinical trials.3 It is plausible that patients treated at a particular clinic site are more similar to one another (in terms of attributes such as age, level of education, and information needs) than they are to patients treated at another clinic site. Within a cluster of individuals with shared attributes, outcomes have a tendency to be correlated. The greater the degree of correlation present, the more the sample size of a clinical trial needs to be increased to achieve the same degree of power.4 Because the degree of clustering is not usually known a priori, it is reasonable to estimate the sample size over a plausible range. Failure to account for correlation creates the risk of an intervention being declared to have had no benefit simply because power was not great enough to detect meaningful differences. Readers of the clinical cancer research literature should applaud the subjecting of behavioral and organizational interventions to the same rigorous scrutiny that medical interventions such as chemotherapy treatments undergo. These interventions may command considerable resources, and therefore, systematic evaluation should be a prerequisite to large-scale implementation. Cluster randomized trials are the optimal technique for evaluating system-level changes in which contamination precludes the ability to randomize at the individual level. They are sometimes termed pragmatic trials because their purpose is to evaluate the effectiveness of interventions in everyday settings (rather than to evaluate efficacy for specific patient populations meeting stringent eligibility criteria). A recent systematic review by Murray et al5 demonstrated that the analytic methods used to design and report these studies are often flawed by a lack of transparency. In December 2008, long after the Passalacqua et al1 study was submitted for publication, the CONSORT group published an extension of its criteria for randomized trial reporting to include cluster randomized trials.6 This consensus statement is important because as organizational and behavioral interventions are increasingly recognized as important strategies for optimizing cancer outcomes, we are likely to see more frequent use of cluster randomized designs. In this context, having a clear set of standards for design, interpretation, and reporting will become all the more valuable. As more transparent standards are adopted, readers, reviewers, and editors will become more familiar with and more discerning about this important study design. AUTHOR'S DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST The author(s) indicated no potential conflicts of interest. REFERENCES
1. Passalacqua R, Caminiti C, Campione F, et al: Prospective, multicenter, randomized trial of a new organizational modality for providing information and support to cancer patients. J Clin Oncol 27:1794–1799, 2009. 2. Giacomini MK, Cook DJ: Users' guides to the medical literature: Qualitative research in health care B—What are the results and how do they help me care for my patients? Evidence-Based Medicine Working Group. JAMA 284:478–482, 2000. 3. Claxton KP, Sculpher MJ: Using value of information analysis to prioritise health research: Some lessons from recent UK experience. Pharamcoeconomics 24:1055–1068, 2006.[CrossRef] 4. Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research. London, United Kingdom: Arnold, 2000. 5. Murray DM, Pals SL, Blitstein JL, et al: Design and analysis of group-randomized trials in cancer: A review of current practices. J Natl Cancer Inst 100:483–491, 2008. 6. Zwarenstein M, Treweek S, Gagnier JJ, et al: Improving the reporting of pragmatic trials: An extension of the CONSORT statement. BMJ 337:a2390; 2008.
Related Article
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||
|
|||||||||||
|
|
Copyright © 2009 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|
