|
Advertisement |
|
|
||
 |
|
|||||
|
|||||
|
 |
||||||
|
||||||
|
||||||
|
Journal of Clinical Oncology, Vol 22, No 10 (May 15), 2004: pp. 1993-1999 © 2004 American Society of Clinical Oncology. DOI: 10.1200/JCO.2004.07.199
Quality of Abstracts Describing Randomized Trials in the Proceedings of American Society of Clinical Oncology Meetings: Guidelines for Improved ReportingFrom the Department of Medical Oncology and the Department of Biostatistics, Princess Margaret Hospital, University of Toronto, Toronto, Canada. Address reprint requests to Ian F. Tannock, MD, PhD, Princess Margaret Hospital, 610 University Ave, 5-208, Toronto, ON, M5G 2M9, Canada; e-mail: ian.tannock{at}uhn.on.ca
PURPOSE: To evaluate the quality of reporting in abstracts describing randomized controlled trials (RCTs) included in the Proceedings of American Society of Clinical Oncology (ASCO) meetings and to propose reporting guidelines for abstracts that are submitted to future meetings.
METHODS: Guidelines for reporting of RCTs in abstracts were developed by extracting key elements from published guidelines for full reports of RCTs, and modified based on an expert survey. Abstracts presenting results of RCTs with sample size RESULTS: Brief description of the intervention, explicit identification of the primary end point, and presentation of results accompanied by statistical tests were regarded by experts as the most important items to include in an abstract, whereas presentation of secondary and subgroup analyses was the least important. Deficiencies in reporting were present in almost all of the 510 abstracts; for example, only 22% of the abstracts provided explicit identification of the primary end point. The median quality score was 5.5 (range, 2.0 to 8.5); the quality score improved with time (P < .0001) and was better for oral or plenary presentations (P = .0003). CONCLUSION: The quality of reporting of RCTs in abstracts submitted to Annual Meetings of ASCO is suboptimal. Although space precludes the inclusion of details required in the final report, abstracts could be improved through the use of explicit minimal guidelines, which are suggested in this article.
Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. Frequently, these trials are first presented at medical conferences, such as the Annual Meeting of the American Society of Clinical Oncology (ASCO), the world's largest meeting of oncology professionals. Trials presented at the ASCO conference may influence clinical practice before the publication of the full report. This is because oncologists wish to offer their patients the best possible treatment, even if the results of clinical trials describing apparent advances have not yet been published in full in a peer-reviewed publication. The situation is compounded by delays in publication1,2 and sometimes by complete lack of publication.3 In a previous study, we found that the median time from presentation of a randomized trial at ASCO to its publication was 2.7 years and that at 5 years, 26% of the trials were still not published.4 Therefore, conference abstracts represent important, albeit abridged, records of the design, analysis, and main results of studies. To improve the quality of medical research and to assist physicians in evaluating clinical data, guidelines for the conduct and reporting of randomized trials have been developed. 5-7 Such guidelines have the potential to improve the quality of reporting.8 Abstracts are of necessity brief and cannot provide many of the details that are required in the final report of a study. It is therefore imperative that the limited space be used to best advantage to report the essential elements of the study. The quality of conference abstracts has received little attention, although preliminary reports suggest that it is poor.9-11 Improving the quality of abstracts would assist physicians in deciding on the merits of a trial and could also facilitate the peer review process. In the present study, we extracted key elements from guidelines for full reports of trials and modified them based on a survey of expert oncologists, to develop a set of minimum guidelines for reporting of RCTs in abstracts submitted to ASCO meetings. We also evaluated the quality of abstracts describing large RCTs submitted to the annual ASCO meetings during a 10-year period and rated them using a quality score that indicates their conformity with our guidelines. Based on our observation that the quality of reporting was rather poor, we suggest use of minimal guidelines for selection of future abstracts describing RCTs.
Development of Guidelines To develop a set of guidelines for reporting of randomized trials in conference abstracts, we reviewed published guidelines for full reports of RCTs6,7,12,13 along with literature on the quality of abstracts.14,15 We decided to use the revised Consolidated Standards of Reporting Trials (CONSORT) statement7 as the primary source for a list of items that are important to include in a brief abstract that would be submitted to a medical meeting. Because of the required brevity of abstracts, coupled with our experience of ASCO abstracts in a previous study4 as well as the experience of Timmer et al,15 several CONSORT items were not included in the list while others were abbreviated. Excluded items were "method of randomization" and most of the discussion items, such as "generalizability" and "overall evidence." Description of participants and details of statistical analysis are examples of items that were abbreviated. An item relating to source of funding was added. We then conducted a survey by e-mail of senior members of ASCO. Our expert survey included ASCO Board members, track leaders of the 2003 Education and Program Committees of ASCO, and the editor and associate editors of the Journal Of Clinical Oncology. Each respondent was asked to rate 17 items derived predominantly from the CONSORT statement that might be included in an abstract (Fig 1) using a scale from 1 (not necessary to include in abstract) to 5 (absolutely required in abstract). We also asked them to provide additional comments regarding issues that were not addressed in our survey, but would be important to consider in developing these guidelines. We sent a reminder by e-mail to the nonrespondents 2 weeks after the initial contact. We tabulated the expert responses by calculating the mean score for each of the 17 items then used the score to divide the items into those that were essential to be included in an abstract reporting an RCT (mean score, > 4.0), those items that should be reported (mean score, 3.5 to 4.0), and those that may be reported if space permits (mean score, < 3.5).
Assessment of the Quality of Reporting in ASCO Abstracts We reviewed Proceedings of the ASCO Annual Meeting for years 1989 to 1998 to identify a cohort of large randomized trials (sample size  200). We selected this time period because we designed the study to also evaluate subsequent publication of trials with positive and negative results; details of the process we used for identification of trials have been published previously.4 For trials presented at more than one ASCO meeting during the study period (n = 28), we retained the abstract that pertained to the earliest presentation and excluded all subsequent ones. This left us with a final cohort of 510 abstracts, each representing a unique trial. We sought to evaluate the quality of reporting in abstracts, which is related to but not the same as the methodologic quality of a trial.16 We defined "quality of reporting" as the degree to which information about the rationale, design, execution and results of the trial was provided in the abstract, consistent with the guidelines that we developed. We designed a data abstraction form, which we modified after pilot-testing it on a sample of 40 abstracts (Table 1). An abstraction manual was developed with the form (available on request). The data abstraction was performed by two of the authors (C.B.-M. and R.D.). To evaluate interrater agreement, 10% of the abstracts (n = 52) were coded by both individuals and results were compared. Disagreements were resolved by discussion among three members of the study team (M.K.K., C.B.-M., and R.D.).
We attempted to quantify the standard of reporting in abstracts by defining a quality score, which was based on conformity with our guidelines (Table 2). The quality score was generated by assigning one point for each of the 10 items that are "absolutely essential" or "should be included" in the abstract (or a half-point if there was partial indication of the item). One point was subtracted if the abstract included an unplanned subgroup analysis provided that this did not reduce the quality score below zero (primary definition). Hence, the quality score could range from 0 to 10. We also calculated the score without penalizing abstracts for reporting unplanned subgroup analyses (alternate definition). Because analyses that used these alternative definitions of the quality score gave similar results, only results using the primary definition are presented in the manuscript.
Statistical Considerations Summary statistics were used to describe characteristics of the trial cohort and the results of the abstraction process. Interrater agreement in data abstraction was assessed by calculating percent agreement and the 95% CI. For items in the expert survey, the mean values and 95% CIs were calculated for each item. Regression with quality score as the outcome variable was used to identify factors associated with the quality of reporting. While routine statistical techniques indicated that the quality score, which is a count variable, did not depart strongly from normality, regression analyses were performed using parametric as well as nonparametric techniques and Poisson modeling. The explanatory variables considered included year of presentation (grouped into 3 categories: 1989-1991, 1992 to 1995, and 1996 to 1998); type of presentation (plenary or oral, poster, or published); sample size (continuous); type of cancer (breast, gastrointestinal cancer, hematologic cancer, lung cancer, or other); type of funding (cooperative group, industry, other, or not-specified); and geographic location of the trial (Europe, North America, or other). Because the definition of the quality score included whether the source of funding was specified and the type of funding variable had a category "funding not specified," this could artificially result in this variable's being significant. Therefore, type of funding variable was first explored in the subset of abstracts where funding was specified. Because it was not significant in univariate analyses, it was not considered further.
Analysis of variance and the Kruskal-Wallis test (nonparametric) were used for univariate analyses. Variables that were significant at the P
Results of the Expert Survey and Development of Guidelines The revised CONSORT statement served as the principal source of items for the expert survey.7 The list of items included in the survey are presented in Figure 1. Of the 69 experts contacted, 39 returned the survey for a response rate of 57%. Figure 1 shows the mean expert score and 95% CI for each of the 17 items included in the survey. Description of the intervention, identification of the primary end point and results accompanied by statistical tests were regarded as the most important items to include in an abstract. Details of study design were slightly less important and results of secondary and subgroup analyses were the least important. Several of the respondents provided additional comments. Two experts indicated that results of unplanned subgroup analyses should not be reported. Others stated that the total number of patients accrued should be reported (two respondents) and that results must be included in the abstract at the time of submission (three respondents). Additional comments related to more detail about data management and analysis (five respondents), identification of sponsorship (one respondent), and the necessity to report potential conflicts of interest (one respondent). Based on our original extraction of important items from guidelines for full reports of trials, and on the results of the above survey, we propose in Table 2 guidelines for reporting of abstracts that describe RCTs.
Quality of Reporting in ASCO Abstracts
The inter-rater agreement in data abstraction based on 10% of the sample (52 abstracts) that was abstracted by two individuals is presented in Table 4. The percent agreement for most of the items was very good (percent agreement > 80%). Items for which the agreement was good (percent agreement between 70% and 80%) were explanation of study rationale, description of participants, and identification of unplanned subgroup analyses. The only item for which the agreement was less than 70% was whether conclusions agreed with results. The data abstractors had problems classifying this item as many of the abstracts did not have an identifiable primary end point or provided vague conclusions. This item was omitted from quantitative analyses.
The quality of reporting in ASCO abstracts is described in Table 1. A rationale for the study was not presented in one third of the abstracts. Items relating to study design were the most poorly reported. For example, the primary end point was explicitly stated in only 22% of the abstracts, sample size considerations were reported in less than 5%, and there was a clear indication of whether an intent-to-treat analysis was performed in less than 10% of the abstracts. Reporting of results was better, with the majority of studies providing a P value and/or 95% CI for at least one end point. Multiple end points were reported in 74% of the trials and a substantial proportion of the abstracts presented subgroup analyses, without any indication that these were planned in the protocol. The source of funding was either not specified or could not be inferred from the title or list of authors in 26% of the trials. The distribution of quality scores is presented in Figure 2. The scores were symmetrically distributed around a median score of 5.5 (range, 2.0 to 8.5). The median quality scores broken down by abstract characteristics are presented as part of Table 3; they all range from 5 to 6, suggesting minimal differences in quality in relation to the factors considered. When formally evaluated using regression techniques, factors that were associated with quality score in univariate analyses included year of presentation (P < .0001), type of presentation (P < .0001), and type of cancer (P = .016). The multivariate model is presented in Table 5. Factors that remained significant in the multivariate analysis were type of presentation and year of presentation. There was no evidence of an interaction between type of presentation and year of presentation (P = .78). Abstracts that were presented as plenary or oral presentations and more recent abstracts had higher quality than earlier abstracts and those not presented at the meeting. The R2 for the model is 0.08, suggesting that the factors considered explain only a small proportion of the variability in scores.
Next, we explored the relationship between the quality of reporting and whether or not the study was positive (defined as study results favoring experimental therapy) on the likelihood of acceptance for an oral or plenary presentation. While the quality of reporting was associated with acceptance for an oral presentation (P < .0001), study results were not (P = .36). When we evaluated whether the quality score was associated with subsequent publication of a full report, we found that it was not predictive of time to publication.
Our review of more than 500 ASCO abstracts presenting results of large randomized trials revealed that many of the abstracts were missing information on essential elements of a clinical trial, especially those elements relating to study design and analysis. Given that formal guidelines are an established mechanism to improve the quality of reporting, we developed a set of guidelines for reporting of RCTs in conference abstracts that may be used by future program committees of ASCO. The proposed guidelines are based on guidelines for full reports of randomized trials, and on input from leaders in the field of clinical oncology. Our findings related to the quality of reporting in abstracts are not surprising. Several studies have shown that the quality of reporting of randomized trials in full reports is deficient.17-20Much less is known about the quality of reporting in abstracts presented at medical conferences, but the limited evidence that is available suggests that quality of reporting in abstracts is also less than optimal.9-11,21 In contrast to our study, none of the prior studies have specifically evaluated abstracts reporting randomized trials, none evaluated clinical trials in oncology, and many were undertaken before dissemination of guidelines for full reports of clinical trials. Also, most of the studies have assessed quality of reporting at a more global level, whereas our study provides insight into very specific elements of reporting, which assisted us in developing guidelines that might be used by ASCO and related organizations. In our cohort of 510 abstracts, items relating to study design were least often reported. These items are important when reviewing abstracts for potential acceptance at a meeting. Whereas our experts ranked inclusion of study results above inclusion of details of study design, the latter were judged to be very important and were ranked higher than items such as presentation of secondary or subgroup analyses. Word limitations placed on abstracts are a frequently cited reason why essential information maybe missing from abstracts. Although none of the abstracts in our cohort scored a perfect 10, a small proportion of abstracts achieved scores of 8 or more, indicating that it is feasible to include the majority of necessary information in an abstract. This suggests that with guidance, quality of reporting may be improved. We found that quality of reporting was associated with acceptance for presentation at the meeting, but was not associated with subsequent publication of a full report. Reasons for this observation may be that quality of reporting in abstracts is not reflective of actual quality of the study or that other factors such as study results may be more predictive of publication. Publication in a journal is based on quality of reporting in a manuscript, and we are not aware of any studies linking quality of reporting in a meeting abstract with quality of study or quality of reporting in a full publication. In contrast, acceptance for presentation at the meeting occurs on the basis of the abstract, thus it is likely the more appropriate comparison. One potential advantage of improving the quality of reporting in abstracts is facilitation of the peer review process. Peer review at medical conferences has been plagued by inconsistent results among reviewers.22 Although diversity in experience and point of view among reviewers accounts for some of the inconsistencies seen in the review process, suboptimal quality of reporting likely contributes to this problem. In addition to benefiting reviewers, improving the quality of abstracts will likely benefit authors. Both in our study as well as in previous ones,9,11,15 better quality of reporting was associated with a higher likelihood of acceptance for presentation. Most important, well-written abstracts that contain all the key information about a trial are also likely to be better appreciated by readers. Can quality of reporting be improved? In the last decade, guidelines for the conduct and reporting have been proposed for full articles describing results of randomized trials,7 meta-analyses,23 and cost-effectiveness analyses.24 Pre- and post-comparisons of these types of studies show that the overall quality of studies, including the quality of reporting, improved following publication of the guidelines.8,25 The improvement in quality over time that we observed in this study may be an indirect result of the dissemination of guidelines for full reports, "policing" by ASCO, or possibly increased competition. Most likely, it is the result of a combination of these factors. The only study that has assessed the impact of instructions on quality of reporting in abstracts looked at the accuracy of reporting in abstracts of manuscripts being considered for publication.26 The investigators did not find that their intervention had an impact, but the object of that study was to improve accuracy of reporting when compared to the full report and not the quality of reporting in itself. We are not aware of studies that assess whether quality of reporting in conference abstracts can be improved through guidelines. Several limitations of our study should be noted. First, we evaluated the quality of reporting, which is not the same as the methodologic quality of the study.16 It is possible that a poorly reported study is well designed and executed, and a well-reported one may have several shortcomings. Our decision to assess the quality of reporting rather than the methodologic quality of the trial was a pragmatic one. Limited space precludes inclusion of some of the key items required for an assessment of methodologic quality (eg, details of the randomization process).15,20,27 Disentangling quality of reporting from quality of study using an abstract is probably not feasible, although most of the items in our guidelines (such as explicit definition of a primary end point) are essential components of high quality studies. Second, it should be kept in mind that all guidelines have a subjective component and thus may reflect authors' biases. We tried to minimize this bias by using feedback from experts when drafting the guidelines, but we generated the list of items that was sent to experts. To facilitate identification of factors that may be associated with quality of reporting, we calculated a quality score. Although our quality score appears to have face and content validity, we did not formally validate the score, therefore, we do not recommend using the score without such validation. Third, abstracts are short, often preliminary, and deficiencies may be rectified in full publication. Fourth, we cannot assess the accuracy of the data in abstracts compared to final results. There is evidence that discrepancies exist,20,28 but their impact on clinical practice is not known. Fifth, we only reviewed the quality of reporting in abstracts describing randomized trials; therefore, we cannot comment on the quality of reporting in other types of ASCO abstracts. Furthermore, our review was limited to RCTs related to cancer and to the meetings of one society; thus our study results do not provide insight into reporting in other disciplines or at other conferences. Nevertheless, there is a strong probability that similar problems of reporting apply to other areas of medicine. Finally, any multivariate model developed using stepwise methods may not be stable and should be confirmed in an independent data set. We strongly support an initiative by ASCO to introduce guidelines for abstracts, including guidelines for abstracts that report randomized trials. We believe that this initiative will be welcomed by authors. The guidelines that we have proposed should be validated and modified if necessary and could then be used to determine whether the introduction of guidelines results in better quality of reporting and facilitates peer review.
The authors indicated no potential conflicts of interest.
We thank Fion Tang for help with data management. We also thank Antje Timmer, MD, MSc, Germany, for providing us with a copy of his quality assessment instrument and manual, all of the experts who participated in our survey, and the reviewers for their constructive comments.
Supported in part by a Cancer Care Ontario Fellowship (M.K.K.) and an ASCO Young Investigator Award (M.K.K.). Presented in part at the 39th Annual Meeting of the American Society of Clinical Oncology, Chicago, IL, May 31, 2003. Authors' disclosures of potential conflicts of interest are found at the end of this article.
1. Stern JM, Simes RJ: Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. BMJ 315:640-645, 1997 2. Hopewell S, Clarke M, Stewart L, et al: Time to publication for results of clinical trials (Cochrane Methodology Review), in: The Cochrane Library. Oxford, UK, Update Software, issue 4, 2002 3. Scherer RW, Langenberg P: Full publication of results initially presented in abstracts (Cochrane Methodology Review), in: The Cochrane Library. Oxford, UK, Update Software, issue 4, 2002
4. Krzyzanowska MK, Pintilie M, Tannock IF: Failure to publish large randomized trials presented at an oncology meeting. JAMA 290:495-501, 2003
5. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature: Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Ann Intern Med 121:894-895, 1994
6. Begg C, Cho M, Eastwood S, et al: Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276:637-639, 1996
7. Altman DG, Schulz KF, Moher D, et al: The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Intern Med 134:663-694, 2001
8. Moher D, Jones A, Lepage L: Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA 285:1992-1995, 2001 9. Waxman BP, Dudley HA: A critical assessment of the submitted abstracts for the 1982 Winter Meeting of the Surgical Research society. Br J Surg 70:182, 1983[Medline] 10. Gourgoulianis K, Panagakis A, Tsakraklides V: Anatomy of the IXth European Congress of Pathology abstracts. Pathol Res Pract 180:246-247, 1985[Medline] 11. Panush RS, Delafuente JC, Connelly CS, et al: Profile of a meeting: How abstracts are written and reviewed. J Rheumatol 16:145-147, 1989[Medline] 12. Moher D, Jadad AR, Nichol G, et al: Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 16:62-73, 1995[Medline] 13. Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials: Current issues and future directions. Int J Technol Assess Health Care 12:195-208, 1996[Medline]
14. Winker MA: The need for concrete improvement in abstract quality. JAMA 281:1129-1130, 1999 15. Timmer A, Sutherland LR, Hilsden RJ: Development and evaluation of a quality score for abstracts. BMC Med Res Methodol 3:2, 2003[CrossRef][Medline]
16. Huwiler-Muntener K, Juni P, Junker C, et al: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287:2801-2804, 2002
17. Junker CA: Adherence to published standards of reporting: A comparison of placebo- controlled trials published in English or German. JAMA 280:247-249, 1998
18. Djulbegovic B, Adams JR, Lyman GH, et al: Evaluation and appraisal of randomized controlled trials in myeloma. Ann Oncol 12:1611-1617, 2001 19. Hill CL, LaValley MP, Felson DT: Secular changes in the quality of published randomized clinical trials in rheumatology. Arthritis Rheum 46:779-784, 2002[CrossRef][Medline] 20. Bhandari M, Devereaux PJ, Guyatt GH, et al: An observational study of orthopaedic abstracts and subsequent full-text publications. J Bone Joint Surg Am 84-A:615-621, 2002 21. Timmer A, Hilsden RJ, Cole J, et al: Publication bias in gastroenterological research: A retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol 2:7, 2002[CrossRef][Medline] 22. Rubin HR, Redelmeier DA, Wu AW, et al: How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine. J Gen Intern Med 8255-258, 1993[Medline] 23. Moher D, Cook DJ, Eastwood S, et al: Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement—Quality of Reporting of Meta-Analyses. Lancet 354:1896-1900, 1999[CrossRef][Medline]
24. Siegel JE, Weinstein MC, Russell LB, et al: Recommendations for reporting cost-effectiveness analyses: Panel on Cost-Effectiveness in Health and Medicine. JAMA 276:1339-1341, 1996 25. Phillips KA, Chen JL: Impact of the U.S. panel on cost-effectiveness in health and medicine. Am J Prev Med 22:98-105, 2002[CrossRef][Medline]
26. Pitkin RM, Branagan MA: Can the accuracy of abstracts be improved by providing specific instructions? A randomized controlled trial. JAMA 280:267-269, 1998
27. Klassen TP, Wiebe N, Russell K, et al: Abstracts of randomized controlled trials presented at the society for pediatric research meeting: An example of publication bias. Arch Pediatr Adolesc Med 156:474-479, 2002 28. Weintraub WH: Are published manuscripts representative of the surgical meeting abstracts? An objective appraisal. J Pediatr Surg 22:11-13, 1987[Medline] Submitted July 29, 2003; accepted February 27, 2004.
This article has been cited by other articles:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||
|
|||||||||||
|
|
Copyright © 2004 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|
TOP
ABSTRACT
INTRODUCTION
.05 level were then entered into a multivariate model. Those that became nonsignificant in the multivariate model were removed individually, and the model was re-evaluated. The multivariate analyses were performed by assuming both that the quality score follows a normal distribution (linear regression) and a Poisson distribution (Poisson regression). The modeling was performed using proc GLM (assuming normality) and proc GENMOD (assuming Poisson distribution) in SAS version 8.2 (SAS Institute, Cary NC). Because no substantial differences were seen between the two analyses, only results of linear regression are presented. Logistic regression was used to explore the relationship between type of result (positive versus negative) and the quality score on acceptance for presentation at the meeting. Cox proportional hazards models were used to examine the association between quality of reporting and subsequent publication. 
