|
|||||
|
|
||||||
Journal of Clinical Oncology, Vol 24, No 7 (March 1), 2006: pp. 1136-1144 © 2006 American Society of Clinical Oncology. DOI: 10.1200/JCO.2005.03.1179 Quality of Randomized Controlled Trials Reporting in the Primary Treatment of Brain TumorsFrom the Department of Clinical Epidemiology and Biostatistics, and Department of Computing and Software, McMaster University; Juravinski Cancer Center; and Center for the Evaluation of Medicine, St Joseph Healthcare, Hamilton, Ontario, Canada. Address reprint requests to: Rose Lai, MD, MSc, The Neurological Institute of Columbia University, Brain Tumor Center, 710 W 168th St, Room 204, New York, NY 10032; e-mail: rlai{at}neuro.columbia.edu
PURPOSE: To assess the reporting quality of randomized controlled trials (RCTs) in the primary treatment of brain tumors and to identify significant predictors of quality. PATIENTS AND METHODS: Two investigators searched MEDLINE, EMBASE, and bibliographies of retrieved articles for RCTs in the primary treatment of brain tumors published between January 1990 and December 2004. We assessed the quality of overall reporting and key methodologic factors reporting (allocation concealment, blinding, and intention to treat [ITT]). Two investigators also rated articles independently using items from the revised Consolidated Standards of Reporting Trials statement. A generalized estimated equation was used to generate regression models that identified significant factors associated with quality of reporting. RESULTS: We retrieved 74 relevant RCTs that randomly assigned 14,498 brain tumor patients. The quality of overall reporting has improved during the last 15 years, but eight of the 15 methodologic items were reported in less than 50% of trials. In the appraisal of the reporting quality of key methodologies, allocation concealment, blinding, and adherence to the ITT principle were reported in less than 30% of articles. Multivariable regression models revealed that an impact factor more than 1.66, publication after 1995, and sample size more than 280 were significant factors associated with better overall reporting, whereas complete industrial funding, impact factors more than 2.64, and positive primary outcomes were predictors of higher ratings of the three most important methodologic qualities. CONCLUSION: Despite improvement in general reporting quality, key methodologies that safeguard against biases may still benefit from better description. Significant factors associated with better reporting may act as surrogates for other characteristics.
Adult brain tumors are relatively rare diagnoses with poor prognoses.1 Advances in their treatments, especially those of glioma, have been slow because the tumors are diffusely infiltrating and molecularly heterogeneous, rendering local therapy and conventional chemotherapy ineffective in sustaining tumor control. The challenge of penetrating the blood-brain barrier adds to the obstacles of finding efficacious agents. Nevertheless, there have been some major therapeutic breakthroughs recently via high-quality randomized controlled trials (RCTs).2 Similar to other areas of medicine, RCTs remain the gold standard for evidence-based practice in neuro-oncology because of their established validity in drawing causal inferences. Many physicians judge the merit of RCTs based on the quality of published reports. Although it is an imperfect proxy for true methodologic quality, reporting quality continues to influence our interpretation of evidence for a number of reasons. First, contacting authors to clarify methodologic issues can be time consuming and inconvenient. Second, practice guideline developers often rely on the quality of publications to make their recommendations, and published RCTs are the most important resources. Third, RCTs of sufficient quality are needed to conduct unbiased meta-analysis.3 Furthermore, the quality of reporting is essential for guiding journal peer-review decisions. To our knowledge, no one has systematically evaluated the quality of RCT reporting in the primary treatments of brain tumors. Because neuro-oncology is an evolving area of medicine, this evaluation may help to facilitate the future development of evidence-based guidelines and practice by identifying reporting deficiencies that can benefit from improvement. At present, there are few guidelines for the treatments of brain cancers, and clinical practice may not follow the best evidence.4 In addition, good reporting quality ensures the validity of meta-analysis, which is particularly useful in neuro-oncology because many individual RCTs are underpowered to detect small effect sizes. Therefore, we conducted this observational study to appraise the quality of RCT reporting in brain tumor therapy. Our goals were to assess the overall quality of published articles using standard guidelines, to specifically evaluate the reporting of those key methodologic qualities that safeguard against biases, and to investigate factors associated with better reporting quality.
Study Selection We searched MEDLINE, EMBASE, and bibliographies of retrieved articles for RCTs in brain tumor treatments published between January 1, 1990, and December 31, 2004, using the terms "brain neoplasms," "brain tumors," "brain cancers," and "randomized controlled trials." We included phase III RCTs published in English on the primary treatment of adult brain tumors and considered all histologic types. The exclusion criteria were RCTs on secondary treatments (eg, treatment of symptoms from tumor or therapy), and phase II RCTs, because they often require additional studies to provide conclusive evidence.5 When the same trial was published more than once, we retained the report based on the primary outcome or the article with the most updated results. Therefore, each study represented a unique trial. Two trained investigators (R.L. and M.F.) independently retrieved articles based on the above inclusion and exclusion criteria. Any discrepancies were resolved through consensus and reference to the abstracts or articles.
Rating of Overall Reporting Quality
Rating of Key Methodologic Qualities Similarly, ITT may have different meanings for different investigators. In this study, we adopted the most common interpretation, which is the inclusion of all patients randomly assigned in the analysis (regardless of whether they actually satisfied the entry criteria; eg, central neuropathology review), the treatment actually received, and subsequent withdrawal or deviation from the protocol.12 To examine further the application of ITT analysis, we characterized the handling of deviations from randomized allocation, false inclusions, and missing outcomes for each trial. In the assessment of blinding, the terms double blind and single blind were judged insufficient for readers to determine who was blinded, given that blinding can occur at the levels of patients, treating physicians, judicial outcome assessors, and data analysts.13 However, blinding of patients or treating physicians is often not feasible in many RCTs of brain tumor therapy because trials may involve the comparisons of different treatment modalities or therapeutics with diverse adverse effect profiles. Nevertheless, the judicial outcome assessors (central neuro-radiology review) who evaluate progression-free survival or time to progression could often be blinded. Therefore, we considered blinding to have occurred if trials specified at least one blinded group. In addition, we compared blinding rates that are feasible with those actually achieved for patients, treating physicians, and judicial outcome assessors. For each of the above-described three factors, we adopted relevant items from the CONSORT statement for evaluation. Each item was scored with 1 point if the method was appropriate and received 0 points if the reporting was unclear or the method was inappropriate. A combined key methodologic index score was calculated for each trial by combining the scores of these three factors (possible range, 0 to 3). An inter-rater agreement was calculated for each key methodology.
Statistical Analyses
To identify factors associated with overall quality of reporting, we used the overall quality reporting score as the outcome variable. We first performed univariate analyses using trial characteristics as covariates. Variables that were significant at the
Characteristics of Retrieved RCTs The RCT selection process is outlined in Figure 1. The inter-rater agreement between the two investigators who selected articles was 0.86 (95% CI, 0.81 to 0.91). Table 1 summarizes the characteristics of included trials. A total of 14,498 patients were randomly assigned, and of the 74 RCTs that formed the basis of this study, more than 90% were therapeutic trials on glioma and CNS metastases. The 45 RCTs on glioma can be further subclassified into four trials on low-grade glioma, 37 trials on malignant glioma, and four trials on a mixture of low- and high-grade histology. Six journals published more than half of all brain tumor RCTs. The primary end point was survival for 63 studies (85.1%) and response rate for the remaining 11 trials (14.9%).
Ratings of Overall Quality and Key Methodologic Qualities The ratings of overall quality of reporting are listed in Table 2. The overall inter-rater agreement was 0.83 (95% CI, 0.79 to 0.87). When the 74 RCTs are considered together, eight of the 15 items were reported in less than 50% of studies. Table 2 also lists the results for CNS metastases and glioma separately. The rating of each item did not differ significantly between the two subtypes of brain tumors (data not shown). Their combined overall reporting quality scores also did not differ significantly from each other (F = 0.02; P = .96).
For the rating of key methodologic factors, the was 0.82 (95% CI, 0.79 to 0.88) for allocation concealment, 0.88 (95% CI, 0.85 to 0.91) for blinding, and 0.90 (95% CI, 0.87 to 0.93) for ITT. When we considered the entire cohort of RCTs, less than 30% of them scored positive in each factor (Table 3). Similar to overall reporting, the scores for each factor and the combined scores did not differ significantly between CNS metastases and glioma (F = 0.47; P = .49). Therefore, all additional analyses involving overall and key methodologic reporting qualities combined the 74 RCTs.
Among those trials that described allocation concealment, the most common method was centralized randomization (Table 4). Blinding of the treating physicians, patients, or outcome assessors was uncommon, even though it could have been accomplished in at least one group in more than 62% of RCTs (Table 5). More than two thirds of RCTs did not mention whether there was any loss to follow-up and did not describe ways of handling missing outcomes (Table 6); about half of all trials excluded patients who were falsely included, most often due to wrong neuropathologic diagnoses.
Figure 2 shows a gradual improvement in overall reporting quality every 5 years, but the greatest progress in scores was from the period 1990 to 1994 compared with 1995 to 1999. Table 7 lists the distribution of 74 brain tumor RCTs by the reporting of key methodologies and overall quality reporting scores, which were divided into quartiles. Only 41.7%, 20.8%, and 45.8% of RCTs in the top quartile of overall quality scores described allocation concealment, mentioned blinding, and performed ITT analyses, respectively. Therefore, good overall reporting does not invariably guarantee the reporting of the most important methodologic qualities that safeguard against bias.
Factors Associated With Reporting Quality Tables 8 and 9 list the univariate analyses of factors associated with overall quality reporting scores and each category's median score. The final multivariable regression model shows that impact factor above 1.660, publication after 1995, and sample size more than 280 were significantly associated with better overall reporting quality (Table 10).
A somewhat different set of factors was associated with the reporting of key methodologic factors (Table 9). The highly significant category described as completely funded by industries remained significant (incidence rate ratio, 2.42; 95% CI, 1.57 to 3.73; P < .0001) when the analysis excluded trials that did not report funding. In the final multivariable regression model (Table 10), RCTs completely funded by industries, trials with positive primary outcome, and impact factor of published journal greater than 2.636 remained significant predictors; the impact factor category more than 4.284 was borderline significant. The R2MARG for the regression model of overall reporting quality was 0.57, which means 57% of the variance in the outcome was explained by the model. Similarly, the R2MARG for the model of key methodologic reporting was 0.62.
This assessment of reporting quality of brain tumor RCTs showed general improvement during a span of 15 years. Some specific areas, such as allocation concealment, blinding, analysis according to the ITT principle, and details of the randomization scheme, were still poorly reported even in recent years. These findings seem to support the impression that it may be easier for authors to describe clinical features of their studies, such as the inclusion criteria and therapeutic regimens, than to report trial methodology and statistical features.17 An awareness of the importance of these features may help; one study showed that RCTs published by principle investigators with epidemiology affiliations scored significantly higher on reporting quality.18 To our knowledge, there is one other assessment of randomized trials in the primary treatment of an entire oncology subspecialty (breast oncology), but it was published before the establishment of guidelines.19 The most common deficiencies identified in that study, such as descriptions related to the randomization process, allocation concealment, and the handling of withdrawals (part of ITT), were also shortcomings found in this appraisal. Although our assessment was conducted nearly 20 years later, the reporting of these essential methodologies does not seem to have changed. The finding that factors predictive of key methodologic qualities were different from those associated with overall reporting quality is intriguing. RCTs with positive outcomes may reflect an increased effort into the reporting of important methodologic features to minimize the scrutiny of biases, given that studies that excluded patients after random assignment or did not report allocation concealment yield more positive results.9 Similarly, better reporting associated with complete industrial funding may arise from the desire of sponsors to alleviate the concern that trials funded exclusively by pharmaceutical companies may result in biased findings. This view is supported by another investigation, which found that trials supported solely by profit-making organizations had a trend toward higher quality scores.20 Nevertheless, this significant factor needs to be interpreted with caution, given that there were only five RCTs completely financed by industrial sponsors, and the funding of almost 30% of trials was unknown. Our results remain to be validated in the future when there are more available studies. Although high journal impact factor is associated with better reporting, it may act as a surrogate for some other characteristics that have a more immediate effect on quality. It is a measure of a journal's prestige, and more influential journals may have more rigorous peer-review processes and attract the submission of better quality papers. Given that many brain tumor therapies are expected to have small benefit, large sample size may be a proxy for meticulous trial planning, which in turn may correlate with better reporting. One limitation of this study is that we cannot verify trial methodology reported in these articles. In fact, the degree to which reporting quality reflects true methodologic quality of an RCT is the subject of continuous debate. Two studies support the idea that bad reporting does not mean bad methodology. The first study was conducted by the Radiation Therapy Oncology Group on a selection of 56 of the consortium's RCTs.21 They found that many aspects of trial methodology not reported in the final articles were stated in the protocols. If this finding is validated in a larger study, the results may apply to the eight Radiation Therapy Oncology Group trials in our cohort. Another study that sampled 35 RCTs from ACP Journal Club, Journal Watch, and Internal Medicine Alert compared published reports with data provided directly by the authors; the results suggested that blinding and allocation concealment were under-reported but were often conducted.22 However, two investigations furnished contradictory evidence. One study that correlated published randomized trials conducted in Denmark between 1994 and 1995 and their protocols found that many trials with unclear allocation concealment in the article also had an unclear description in the protocols.23 In another survey that examined 63 RCTs of breast cancer treatment, data from the published reports were compared with information provided by the principle investigators; the conclusion was that deficient reporting did reflect flawed methods.19 Small sample size, the reliability of surveyed investigators, variation in the quality of sampled journals, and the number of nonresponders may account for discrepancies of these methodologic studies. In the future, the availability of all RCT protocols in the public domains may help to facilitate the assessment of trial methodology.24 This study has other limitations. The appraisal of reporting quality was limited to those aspects related to the study design and analysis; we have chosen not to assess the quality of discussion and the validity of authors' conclusions because previous evaluations found that these assessments were too subjective, with poor correlations between raters.25 Furthermore, our assessment offers no insight into the ratings of other aspects of clinical trial methodology, such as the external validity and appropriateness of the comparison arm, because they are not part of CONSORT and have not been validated extensively in other scales. The inclusion of all brain tumor histology in a wide range of journals and the coverage of a relatively long period of time do allow substantial generalization of our quality ratings. This first survey, therefore, represents one of the initial steps toward improving evidence-based practice in neuro-oncology; in the future, we suggest that all RCTs of brain tumor therapies should report the methodologic items outlined in the revised CONSORT statement, especially those pertaining to allocation concealment, blinding, and ITT.
The following RCTs were included in this study (in alphabetical order by the first author):
The authors indicated no potential conflicts of interest.
Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.
1. Deangelis LM: Brain tumors. N Engl J Med 344:114-123, 2001 2. Stupp R, Mason WP, Van Den Bent MJ, et al: Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. N Engl J Med 352:987-996, 2005 3. Moher D, Pham B, Jones A, et al: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609-613, 1998[CrossRef][Medline] 4. Chang SM, Parney IF, Huang W, et al: Patterns of care for adults with newly diagnosed malignant glioma. JAMA 293:557-564, 2005 5. Wieand HS: Randomized phase II trials: What does randomization gain? J Clin Oncol 23:1794-1795, 2005 6. Moher D, Schulz KF, Altman DG: The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:1191-1194, 2001[CrossRef][Medline] 7. Huwiler-Muntener K, Juni P, Junker C, et al: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287:2801-2804, 2002 8. Balk EM, Bonis PA, Moskowitz H, et al: Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 287:2973-2982, 2002 9. Schulz KF, Chalmers I, Hayes RJ, et al: Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273:408-412, 1995 10. Schulz KF, Altman DG, Moher D: Allocation concealment in clinical trials. JAMA 288:2406-2407, 2002 11. Alderson P, Green S, Higgins JPT: Assessment of study quality, in Higgins JPT, Green S (ed): Cochrane Handbook for Systematic Reviews of Interventions 4.2.5. Chichester, UK, John Wiley & Sons Ltd, 2005, pp 87-95 12. Hollis S, Campbell F: What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 319:670-674, 1999 13. Devereaux PJ, Manns BJ, Ghali WA, et al: The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials 23:380-388, 2002[CrossRef][Medline] 14. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977[CrossRef][Medline] 15. Zeger SL, Liang KY, Albert PS: Models for longitudinal data: A generalized estimating equation approach. Biometrics 44:1049-1060, 1988[CrossRef][Medline] 16. Zheng B: Summarizing the goodness of fit of generalized linear models for longitudinal data. Stat Med 19:1265-1275, 2000[CrossRef][Medline] 17. Junker CA: Adherence to published standards of reporting: A comparison of placebo-controlled trials published in English or German. JAMA 280:247-249, 1998 18. Bhandari M, Richards RR, Sprague S, et al: The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am 84:388-396, 2002 19. Liberati A, Himel HN, Chalmers TC: A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 4:942-951, 1986 20. Djulbegovic B, Lacevic M, Cantor A, et al: The uncertainty principle and industry-sponsored research. Lancet 356:635-638, 2000[CrossRef][Medline] 21. Soares HP, Daniels S, Kumar A, et al: Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 328:22-24, 2004 22. Devereaux PJ, Choi PT, El-Dika S, et al: An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol 57:1232-1236, 2004[CrossRef][Medline] 23. Pildal J, Chan AW, Hrobjartsson A, et al: Comparison of descriptions of allocation concealment in trial protocols and the published reports: Cohort study. BMJ 330:1049, 2005 24. De Angelis CD, Drazen JM, Frizelle FA, et al: Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. N Engl J Med 352:2436-2438, 2005 25. Krzyzanowska MK, Pintilie M, Brezden-Masley C, et al: Quality of abstracts describing randomized trials in the proceedings of American Society of Clinical Oncology meetings: Guidelines for improved reporting. J Clin Oncol 22:1993-1999, 2004 Submitted June 15, 2005; accepted December 14, 2005.
This article has been cited by other articles:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||
|
Copyright © 2006 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|