This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Roche, K. Articles by Walker, H. Search for Related Content PubMed PubMed Citation Articles by Roche, K. Articles by Walker, H. Social Bookmarking                            What's this?

Factors Affecting Workload of Cancer Clinical Trials: Results of a Multicenter Study of the National Cancer Institute of Canada Clinical Trials Group

From the Sunnybrook Regional Cancer Centre, Toronto; National Cancer Institute of Canada Clinical Trials Group, Kingston; London Regional Cancer Centre, London, Ontario; and British Columbia Cancer Agency, Vancouver Clinic, Vancouver, British Columbia, Canada.

This study was made possible through the commitment of clinical research associates commitee members at participating institutions across Canada (Appendix C).Address reprint requests to Kathyrn Roche, Manager of Clinical Trials and Epidemiology, Toronto-Sunnybrook Regional Cancer Centre, 2075 Bayview Ave, Toronto, Ontario, Canada M4N 3M5; email: kathie.roche{at}tsrcc.on.ca

	ABSTRACT

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
PURPOSE: Increasingly, cancer treatment centers need to be able to estimate specific costs and resources associated with clinical trials. Because the time requirements of trial coordination and data collection are not well known, the Clinical Research Associates (CRA) Committee of the National Cancer Institute of Canada Clinical Trials Group carried out a multicenter study to measure trials’ task times and evaluate the effects of certain factors.

METHODS: A data collection instrument was designed and validated before its implementation in the study. Eighty-three CRAs from 24 cancer treatment institutions across Canada collected timing observations of 41 tasks (156 subtasks). Information from all stages of trials activity (protocol management, eligibility and entry, treatment, and follow-up and final stage) was obtained, from initial negotiations to follow-up after study closure.

RESULTS: After controlling for stage, phase and sponsor were found to be significant independent factors. Analysis within the stages showed similar patterns. New drug inclusion as a factor was confounded with phase. Industry-sponsored studies had significantly higher overall mean times than did local and cooperative group studies. Early-phase studies required more time than did phase III trials. External sponsorship of any kind increased CRA time more than that necessary for locally coordinated studies, except during the protocol management stage. The burden of a phase I study increased to greater than average once underway and accruing patients.

CONCLUSION: Our data demonstrated that sponsor and study phase are important factors to be taken into consideration when estimating clinical trial costs and resource use.

	INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
CANCER CLINICAL trials have become more rigorous and demanding in their execution, in large part because of an increasingly stringent regulatory environment.^1,2 The result is that the costs to undertake clinical trials have escalated, but the financial environment is such that health care budgets are being reduced, and institutions are under pressure to manage resources more efficiently.¹ Future research is threatened as arguments over who should pay for the costs of large randomized studies of experimental treatments compared with standard therapy escalate.^3-6 Pharmaceutical industry and cooperative group sponsors extend reimbursement at varying levels, but it is far from clear how much of the effort is covered by these funds. Cancer trial units are finding it essential to be able to forecast the specific costs of research and thus define and target necessary resources and budgets. Unfortunately, little is known about the costs of running a clinical trial.⁷ Although it is recognized that a full assessment of the overall cost of clinical trials would take into consideration research infrastructure, data collection, and both indirect and direct costs of medical care,⁸ recent investigations have concentrated on the latter. In the United States, clinical trial costs have not been reimbursed by Medicare or private insurers, although recent legislation has been directed at removing that discrimination.⁹ Concern that payment for care as part of clinical research is becoming less predictable as a result of managed care has fueled a number of studies examining the relative costs of clinical trials and routine patient care.^8,10-14

The most obvious and explicitly extra costs incurred in clinical trials are those that arise from the workload of clinical research associates (CRAs), the individuals responsible for the management and administration of the study as well as the data collection effort, but these costs have not been systematically investigated. There are no standard, accepted formulas for calculating CRA workload; even so, the Society for Clinical Research Associates (SoCRA) reports that its members are facing tremendous pressure from a variety of sources to account for their time.¹⁵ Workload measurement instruments in health care disciplines such as nursing and pharmacy are relatively sophisticated, yielding information now used routinely in administrative decision-making, but they are of limited use in application to the clinical trials research setting. Nursing systems relate primarily to methods of patient care, incorporating measurements that stem from the physical and emotional needs of the patient.^16-19 Pharmacy workload measurement research has generated calculations in unit values of the time required to perform the functions of hospital pharmacy practice.^20-22 Because CRAs are involved in a wide range of activities, including regulatory compliance, indirect and direct patient and protocol management, and administration, it is difficult to estimate CRA workload using existing formulas that account only for patient-related or unit-based activity.¹⁵ Although a major component of clinical trial funding is the cost of CRA time, that amount is usually established by guesstimation.²³

The lack of information about required CRA time for trial coordination and data management presents a substantial obstacle to cancer research units attempting to estimate accurately the budget and resource requirements of proposed studies. The ability to predict all the factors that influence the time spent by CRAs on a project is critical. At Toronto-Sunnybrook Regional Cancer Centre, a complexity template based on a rating of studies by local CRAs as low, moderate, or high intensity, in correlation with their collected task-timing data, has been used to support cost estimates of proposed new trials (K. Roche, personal communications, 1995 and 1998). The CRA Committee of the NCIC CTG wished to build on the preliminary work performed in Toronto and at other Canadian centers²⁴ (personal communications: B Smuck, 1995 and 1998; M.A. Hiatt, 1995; M.A. Brittain, J. Manzo, and S. Webster, 1995) and, therefore, designed and carried out a multicenter workload measurement study (WM1). The goal of WM1 was to learn more about the time requirements for managing, administering, and gathering data for clinical trials in Canadian cancer treatment centers, thereby assisting institutions in their cost estimates of trial participation and efforts to forecast and organize resource and personnel use. The study objectives were (1) to calculate time unit measures for standard trial coordination and data management tasks carried out by CRAs; (2) to identify and assess the effect of varying circumstances and other factors that may have an impact on those task times; and (3) using this information to develop a workload measurement tool for institutions to use in their trial cost and feasibility estimations.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Study Participants
The study was carried out between April and October 1996. Eighty-three CRAs from 24 NCIC CTG member centers collected information on the time spent on various clinical trial–related tasks. Each CRA recorded task times for 30 consecutive days during a 3-month period. CRAs were defined according to the SoCRA definition at the time of the study: "A CRA may function as a coordinator, consultant, educator and/or researcher in clinical research. These persons will be responsible for and participate in the management of one or more of the following aspects of clinical trials research: data collection, analysis, or monitoring; case management of protocol patients; recruitment or enrollment of human subjects; protection of subjects and subjects’ rights through ethics board relations, development of informed consents; preparation of adverse event experience reports; maintenance of drug records and/or inventory; grant and budget development; report preparation; education of other health care professionals, patients, or family; and protocol development" (B. Davis, SoCRA, personal communication, 1994). (SoCRA has since modified the terms of its definition to apply to clinical research professionals to accommodate the confusion of terminology relating to job titles. CRAs are known by a variety of names, including clinical research coordinator, data manager, clinical research nurse, study coordinator, and clinical research assistant). CRAs with at least 3 months of experience and working at CRA functions at least 0.2 full-time equivalents (FTEs), were eligible to participate. CRAs could report timings on all or a subset of their trials. The participating member institutions were primarily academic teaching hospitals and provincial cancer centres. Six community hospitals and health care facilities were included. Before joining the study, each institution submitted to the NCIC CTG central office a blind-coded list of studies to be referenced by its CRAs, categorized by phase, type, primary sponsor, and new drug component (Appendix A). It should be noted that studies sponsored locally or by cooperative groups may or may not have industry support. For the purpose of this analysis, we did not attempt to distinguish these indirect sponsor subgroups. The sponsor "other group" refers to three cooperative groups: the Radiation Therapy Oncology Group (RTOG), the Gynecologic Oncology Group, and the National Surgical Adjuvant Breast and Bowel Project.

Data Collection Form
The study data collection form incorporated components from various instruments already developed by members of the planning committee for their own local investigations (Fig 1). Four stages of trial activity were identified: (1) protocol management (administration and tasks trial-specific but not patient-specific), (2) eligibility and entry, (3) treatment, and (4) follow-up and final stage (tasks relating to the entry and management on study of particular patients). Task categories were assigned to each of the four stages, with each category divided into subtasks to ensure all CRA activities would be captured. Protocol management included overall study coordination tasks, some that related to set-up and closure, thus occurring only once per trial, and others that involved trial maintenance, thus recurring regularly. Timings for trial tasks, such as study meetings and batch screening, were also reported in this stage. Unlike three sequential stages that follow, protocol management involved activities that span the entire life of a trial. The intent of the eligibility and entry stage was to capture data related to screening and eligibility assessment activities performed, whether the patient was successfully entered onto a study or not. These tasks are performed only once per patient per study. Both the treatment and the follow-up and final stages included repeating tasks performed multiple times per study patient (Appendix B).

View larger version (14K): [in this window] [in a new window]   Fig 1. Sample data collection form page.

Statistical Analysis
Data were entered into an Oracle database (Computertime Network Corp, St-Laurent, Quebec) with customized data entry screens, verified, and then transferred to an SAS data set (SAS Institute, Cary, NC) for batch checking and analysis. Task times were consistently categorized by center, CRA, trial code, and patient initials; however, we differentiated between discrete tasks and continuing tasks. For the study stages treatment and follow-up and final stage, task times were considered discrete, by day. For the stages protocol management and eligibility and entry, daily task times were added together because the nature of these tasks is such that they often extend over a number of days before completion (eg, the task of reviewing the informed consent document with a patient and eventually obtaining consent).

The primary outcome of the study was the time of each of the trial coordination and data management tasks as recorded by the CRAs. The total time spent for carrying out trial activities for a particular trial stage (ie, protocol management, eligibility and entry, treatment, follow-up and final stage) was calculated by summing all of the corresponding task times within that stage. The average times for a particular trial stage were compared using least square means under the general linear model. The trial factors of interest in this report were phase of study (phases I, I/II, II, and III), sponsor (NCIC CTG, other cooperative group, pharmaceutical industry, and local), and inclusion of investigational drugs. The specific trial factors were first assessed on the basis of the overall F-test from the general linear model, after controlling for trial stages. Least square means for various trial factors were calculated and compared within the two-way analysis of variance models. Further exploratory analyses were carried out on the basis of the least square means and the significance values for each of the trial stages, controlling the effect of tasks in a two-way analysis of variance model. A Bonferroni correction for the significance values was applied when we assessed the least square means for the pair-wise comparisons. For example, a P < .0083 was considered significant when a factor with four levels, such as trial type or sponsor, was being assessed.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Participants averaged 6.8 years of clinical research experience (oncology experience, 6.4) and 34.5 hours per week, with education levels of master’s degree (4%), bachelor’s degree (29%), college diploma (31%), nursing diploma (31%), and high school diploma (5%). Each was responsible for, on average, an overall number of 28 clinical trials (mean open, 12; mean closed, 14; mean planned, 4). Number of submitted data items per CRA ranged from 33 to 678.

After controlling for stage (protocol management, eligibility and entry, treatment, and follow-up and final), the analysis, including phase, new drug inclusion, and sponsor together, showed that phase and sponsor were significant independent factors. The analysis within the stages showed similar patterns. New drug inclusion as a factor was confounded with phase and was not significant in some of the multifactor models. Results by phase of study (P < .0001; df, 3) and the nature of sponsor (P < .0001; df, 1), stratified by trial stages, are shown below. In Fig 2, mean stage times are compared by phase and by sponsor. Tables 1 through 4 define the component tasks of the stages and indicate the number of timings recorded. The mean value for each task is given, along with the maximum and the SD (the minimum range values were typically 1 to 5 minutes, and thus are not tabulated here). Figures 3, 4, 5, and 6 illustrate comparability of task times across sponsor and phase.

View larger version (70K): [in this window] [in a new window]   Fig 2. Mean times of trial stages: comparisons by phase and sponsor.

View larger version (73K): [in this window] [in a new window]   Fig 3. Protocol management stage: mean task time comparisons by phase and sponsor.

View larger version (70K): [in this window] [in a new window]   Fig 4. Eligibility and entry stage: mean task time comparisons by phase and sponsor.

View larger version (69K): [in this window] [in a new window]   Fig 5. Treatment stage: mean task time comparisons by phase and sponsor.

View larger version (67K): [in this window] [in a new window]   Fig 6. Follow-up and final stage: mean task time comparisons by phase and sponsor.

Protocol Management
Local, industry-sponsored, and other cooperative group studies all took significantly more CRA time in the protocol management stage than did NCIC CTG studies (P < .006 for all three pair-wise comparisons) (Table 1; Fig 3). It is notable that task H (monitoring) took strikingly more time for other cooperative group studies, probably because of site audits taking place at the involved centers, but for all tasks other than H, the factor of other group as sponsor seems comparable to NCIC CTG as sponsor.

Eligibility and Entry
Both industry and other cooperative group trials required significantly more time for the Eligibility and Entry stage compared with NCIC CTG and local studies (P < .006 for the particular pair-wise comparisons) in a two-way analysis of variance model controlling tasks (Table 2; Fig 4). No significant difference between the studies by phase was demonstrated in the eligibility and entry stage. The high phase I/II mean time for task J (documenting) was extrapolated from only 15 timings out of a total of 332. The highest individual timing for that task was for a phase III trial sponsored by industry.

Treatment
Trials sponsored by industry took more time at the treatment stage than did trials with other sponsors (P < .0002 for all three pair-wise comparisons), with local studies requiring significantly less time (P < .0001 for all three pair-wise comparisons) (Table 3; Fig 5). Phase I, I/II, and II design was associated with higher task times than was phase III design at this stage of a clinical trial (P < .0001 for all three pair-wise comparisons).

Follow-Up and Final
Phase I studies in the follow-up and final stage were more time-consuming than phase I/II, II, and III trials (P < .0001 for all three pair-wise comparisons) (Table 4; Fig 6). Sponsor also was an important factor: industry studies took significantly more time than trials coordinated locally or by cooperative groups (P < .0001 for all three pair-wise comparisons).

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Traditionally, clinical research has been subsidized by cancer care facilities, and carried out as a labor of love by enthusiastic investigators.² It is understood that many clinical trials are underwritten by participating institutions as a result of inadequate funding, which fails to take into account various kinds of incremental costs. A retrospective analysis carried out in 1993 of two NCIC CTG phase II trials in previously untreated small-cell lung cancer demonstrated that the major cost drivers were laboratory and imaging tests.⁷ Recently, the assumption that enrollment of patients onto clinical trials necessarily incurs incremental costs, such as an increased frequency of tests, has been challenged by several studies that conclude that treating patients in a trial setting does not in fact constitute an extra burden for the taxpayer or insurer.^8,10-13

However, although CRA time is clearly incremental, it is still left out of the equation. The author of a study that concluded that participation in cancer clinical trials at a large health maintenance organization did not result in substantial increases in the direct costs of medical care states unequivocally that a full accounting of the costs "would assess not only direct medical care costs but also the burden of recruiting patients, assuring that treatment protocols are followed, collecting and managing data, and supporting the infrastructure for research."⁸ Notably, institutions that have either directly calculated their own costs of clinical research or recently negotiated contracts for such work report much higher estimates of costs than do those that have not undertaken formal costing procedures.²⁵

CRA activities, which include a wide range of functions within the areas of regulatory compliance, protocol management, direct and indirect patient care, data collection, and general administration, form the cornerstone of clinical trials research. However, we are aware of no published research that attempts to quantify the time requirements of these activities or analyze the influence of relevant factors on those requirements. Work to date has been preliminary, generating more questions than answers and/or based on subjective evaluations or generalized relative ratings rather than actual measurement and real data. A group at the Methodist Hospital of Indiana, for example, has designed a protocol complexity-rating questionnaire for study nurses to complete in their ranking estimates of the relative workloads of clinical trials.²⁶ The authors of a formula for the estimation of CRA time for trial funding purposes claim that their simple method provides results that correlate to the elaborate guesstimation usually involved in such calculations, but their formula is only applicable in situations where an initial estimate of set-up time in hours can readily be assumed.²³ In the analysis of the two NCIC CTG phase II lung studies, data management costs were determined from time estimates only.⁷

The paucity of reliable information in the literature is disturbing, according to researchers with the RTOG who recently carried out a workload survey of their CRAs, because administrators are left with unproved methods and practices for determining workload, which in turn can result in unrealistic performance expectations.²⁷ For example, an algorithm developed in 1992 by the Cancer Clinical Investigations Review Committee (CCIRC), now called Study Section H, of the National Cancer Institute for use by cooperative groups determining budget allotments for their member institutions has inappropriately been adopted by some investigative sites to assess productivity and make day-to-day staffing decisions.²⁷ The CCIRC algorithm, devised after committee discussions but not validated, assigns the following credits: 1.00 point for a phase II or III study registration; 0.5 points for a phase I or ancillary (biology) study registration; 0.25 points for follow-up of each subject after active treatment; and 0.15 points for subjects enrolled or followed up by affiliated institutions. A sum of 40 credits during a 12-month period is judged to be equal to one FTE. The RTOG researchers argue that staffing decisions on the basis of this formula are bound to emphasize subject enrollment and follow-up information to the exclusion of many other aspects of a CRA’s responsibilities. Even so, in their survey, most respondents reported more than the target number of credits (mean, 57.4; median, 51.0).²⁸ Clearly, CRAs "have their hands full."²⁹

At the 1999 Annual Meeting of the American Society of Clinical Oncology (ASCO), then-president Allen S. Lichter reported on a major survey of oncologists with regard to recruitment to clinical trials. Respondents (38.4%) claimed that insufficient data management support constituted a significant barrier to their participation. More than 80% said that more data management help would assist their enrollment efforts.²⁵ A recent Canadian survey of investigators concerning the criteria for the determination of per case funding showed a similar response: data management was identified 34% of the time as the most important determinant of the costs of conducting a phase II study.⁷ The findings of a project, also presented as part of the ASCO Presidential Symposium, comparing institutional estimates of industry sponsorship and government sponsorship of a mock protocol judged to be of moderate intensity (phase III randomized, double-blind, chemotherapy for hormone-refractory prostate cancer) were that study sites require almost two FTEs to conduct such a clinical trial with 20 enrollees (200 hours per patient).²⁵ Although there was a high level of variation in the resource intensity of clinical research at the 25 United States sites who participated in the exercise, the authors noted that even the minimum study site estimates were high and concluded that CRA work accounted for the majority of staff time on clinical research activities.

The RTOG authors stated that their survey results are useful for generating questions but have limited application, in part because the modified CCIRC algorithm they used assumes that all treatment trials have comparable levels of workload intensity, whereas in fact the frequency and complexity of individual tasks can vary greatly across trials.²⁸ They emphasize that their findings should provide impetus for studies to explore workload measurement tools and protocol complexity rating systems. As thought-provoking as the ASCO mock protocol results are, it is important to note they are based on assumptions about complexity rather than calculations, estimates of workload rather than actual data.

The workload measurement study carried out by the CRA Committee of the NCIC CTG involved numerous health care professionals at cancer care facilities across Canada collecting timing information on a daily basis during a several-week period on a large number of carefully predefined tasks. The measurement tool we developed was both comprehensive and precise. It performed well in the multicenter setting.³⁰ WM1 thus represented a unique effort to date, in size and scope, to categorize CRA workload and measure time requirements while assessing the effects of certain trial features.

Our data suggest that both pharmaceutical company sponsorship and local management of a study increase start-up time and continuing administrative burden (protocol management stage). A possible explanation is that membership in cooperative groups, including NCIC CTG, provides centers with ongoing external support (financial, intellectual, and administrative), as well as consistent expectations from study to study. Protocol and case report form formats are standardized. Streamlined procedures can be established to minimize repetition of effort where possible, whereas a trial unit coping with a single study from a company with which it has no history will have to dedicate time and energy to proving its credentials, meeting personnel, negotiating contracts and other arrangements, learning procedures, and so on. A different challenge is faced by trial units that carry out studies locally. Without either the mantle of cooperative group membership and its familiar infrastructure or the regulatory expertise and authority of an experienced pharmaceutical company, such a unit is literally on its own, meeting the responsibilities of sponsor in addition to those of participating center. At the eligibility and entry stage, however, studies with no external sponsorship took the least time. It is probable that a trial designed in-house would have in place patient screening and registration procedures based on local practice and, therefore, easily incorporated into the natural flow of events. The process of determining patient eligibility for a multicenter cooperative group or industry study is comparatively structured and inflexible. For example, the registration/randomization step alone (task C) can involve a back-and-forth communication between the center and the sponsor sometimes spanning several hours or more across time zones and typically based on a substantial list of inclusion and exclusion criteria written for a mixed group of institutions, thus requiring interpretation for local application. For local studies, the study personnel authorized to make interpretative rulings are conveniently located on site.

Why industry trials took more time at both the treatment and the follow-up and final stages than did studies with other sponsorship is not clear. One hypothesis is that these trials are routinely more complex; that is, each task involves more steps and/or components when a pharmaceutical company is involved. For example, treatment task A, toxicity assessment, consists of several subtasks, as follows: (A) interim counts, (B) treatment day counts, (C) previous toxicity, (D) new toxicity, (E) grading, (F) dose modification, (G) document toxicity, and (H) other. On the data collection form, a CRA who reported a timing for the task also would have ticked all the subtasks performed. We calculated a complexity index for the task for each sponsor type by assessing the percentage of task time events in which more than the majority (five or more of the eight subtasks) were ticked and found that, at 22%, industry trials had the highest percentage of tasks with the majority ratio. NCIC CTG and other cooperative group trials were a close tie for second place (16% and 15%, respectively), although for local studies, the task toxicity assessment was remarkably lacking in complexity (only 3% of the time the task was performed did it involve five or more subtasks). Intrigued by our findings, we calculated complexity indices for the first tasks of the other three stages: (1) protocol management task A (center-approval procedures): industry, 14%; NCIC CTG, 3%; other group, 0%; local, 0%; (2) eligibility and entry task A (screen for eligibility): industry, 23%; NCIC CTG, 16%; other group, 14%; local, 14%; and (3) follow-up and final task A (patient assessment): other group, 32%; industry, 25%; NCIC CTG, 25%; local, 22%. Taking the first task as a marker of the nature of the stage as a whole, we might conclude that tasks at every stage except follow-up and final are most complex when industry is a sponsor.

Differences in time requirements by study phase seen in our analysis were not unexpected. Phase III trials should have simpler and fewer data management and trial coordination expectations than earlier phase studies because the regulatory scrutiny necessary for investigation involving human subjects is more stringent during the earlier phases. Comparisons are less reliable when the numbers are small, as they are in some of the phase I and I/II breakdowns, especially in the protocol management stage, which involves fewer CRAs doing tasks that are relatively time-consuming, but we elected to maintain the distinction between phases I, I/II, and II to highlight apparent differences in the time requirements of some of the tasks (eg, protocol management tasks A through D; follow-up and final tasks C and D).

Our results are in accord with a general consensus among members of the CRA committee that certain features of trials increase their workload, for example, pharmacokinetic blood sampling in phase I studies, tumor measurements in phase II studies, and multiple monitoring visits by industry personnel. Pilot workload studies done in Canadian cancer centers typically incorporated assessment of factors such as sponsor and phase of study²⁴ (K. Roche, B. Smuck, M.A. Brittain, personal communications, 1995). We contend that the CCIRC algorithm undervalues the workload of phase I and I/II studies. Interestingly, in the ASCO mock protocol project, industry versus government sponsorship was not found to affect substantially the cost and time required to perform a clinical trial of moderate intensity, yet respondents to the ASCO survey of oncologists reported that they were more likely to participate in a cooperative group study than one sponsored by industry if the same level of reimbursement were offered.²⁵ Although there may of course be other factors that influence their level of enthusiasm, our finding that pharmaceutical industry sponsorship of trials does in fact increase CRA workload at every stage would seem to confirm the general suspicion that such trials are not necessarily a major source of supplemental funding, as has been claimed,² but rather that more money is rightly being paid for more work.

Although it would be tempting to adopt the mean task times generated from our analysis for direct application to clinical trial workload and cost calculations, we caution against this. The mean task times reported in this article are useful to assess the relative effects of sponsor and phase on workload, but the numbers on their own must be regarded as underestimates. A clear limitation of the WM1 study was our inability to define a complete task, because the data reflect a snapshot in time, and many tasks extend over 2 or more days (eg, the informed consent process). Although we did differentiate between discrete tasks and continuing tasks in our analysis, the data do not allow us to determine whether the task was actually begun before the data collection period or completed after it was over. Further, because each task consists of a number of subtasks, the time a given task takes is likely to vary considerably by center and even CRA, depending on local procedures and job descriptions. The separate, detailed analysis of four sample tasks we carried out to explore changes in their composition according to trial characteristics involved estimates of subtask parameters in minutes, as well as analysis of frequencies of those subtasks.³¹ In our examination of protocol management task A (center-approval procedures), for example, we found that the overall mean for the task category was substantially lower than would be expected on the basis of the parameter estimates of the individual component subtasks. Large variations seen in the task category special procedures (eligibility and entry, treatment, follow-up and final stage) can be explained by the different types of subtasks included under its rubric (eg, arranging pathology review, which takes only a few minutes, v radiotherapy quality assurance, a rare but time-consuming occurrence).

The more precise subtask data to be obtained after an analysis of the entire WM1 data set might be more reliable for planning and cost-estimating purposes, as institutions would then be able to reconstruct each task according to the particular demands of planned trials and the actual mix of subtasks carried out by their CRAs. The time required for repeating tasks could be approximated on the basis of patient accrual plus expected number of treatment visits and duration of follow-up, adding and weighting the estimates according to trial sponsorship and phase.

To more fully appreciate the effort involved in the coordination and data management of clinical trials, and thus to understand how time and money are being spent, researchers in the future would do well to direct their attention to close examination of the individual tasks and resist the tendency to generalize from limited information. To address the question of the time taken by continuing tasks, we recommend that a smaller scale study, among a limited number of centers, be undertaken to collect accurate data on such tasks from beginning to end. The sample tasks we examined varied in complexity among trial sponsors in ways that affect the CRA time required for their completion. The reasons for those variations should be explored, so we recommend further work in this area. We also compared the WM1 data collected on two tasks relating to screening for eligibility and obtaining consent with the NCIC CTG trials database to approximate the ratio of time spent on pre-entry work-up to the actual number of patients successfully registered.³² The results of this investigation showed that most of the patients CRAs approach are never registered, and that for every patient successfully enrolled, roughly 5 hours are expended, not counting time spent conducting general, across-trials screening. We concluded that trial units that estimate required CRA time for planned studies and sponsors hopeful of rapid accrual should be aware of the iceberg effect of the screening and consent effort and incorporate the expected costs into their budgets. Trial success is obviously threatened by excessive or difficult screening requirements, which should thus be minimized wherever possible.

APPENDIX A

The following individuals and institutions participated in the study: D. Green, Dr H. Bliss Murphy Cancer Centre, St John’s, Newfoundland; V. Powell, M. Yazer, Nova Scotia Cancer Centre, Halifax, Nova Scotia; K. Chenier, St John Regional Hospital, New Brunswick; H. Boudreau, Dr Leon Richard Oncology Centre, Moncton, New Brunswick; E. Plitt, Jewish General Hospital, McGill University, Montreal, Quebec; J. Dionne, L. Gagne, M. Methot, I. Nadeau, Hopital Notre Dame, Montreal, Quebec; D. Danis, D. Fry, G. Gauthier, D. Lister, D. St-Cyr, B. Waterfield, F. Aspelund, M. DuManoir, M. Laurin-Wold, K. Schenk, Ottawa Regional Cancer Centre, Civic and General Divisions, Ontario; M. Klassen, E. Ravelle, D. Woodland, Hotel Dieu Hospital, St Catharines, Ontario; N. Deyoe, York County Hospital, Newmarket, Ontario; N. Johnston, R. Mulock, Humber River Regional Hospital, Toronto, Ontario; J. Sinclair, St Joseph’s Hospital, Toronto, Ontario; I. Barak, J. Hayes, K. Hewitt, S. Knight, M. MacDonald, P. McKinlay, B. Nayler, L. Paolucci, K. Roche, G. Stelmach, Toronto-Sunnybrook Regional Cancer Centre, Ontario; M.A. Brittain, I. Malanyaon, J. Manzo, D. Requiestas, D. Tsuji, S. Webster, P. Wilkowski, Princess Margaret Hospital, Toronto, Ontario; R. Fraser, Credit Valley Hospital, Mississauga, Ontario; I. Driscoll, T. Englbrecht, S. Grant, A. Gratton, A. Malpage, B. Smuck, D. tenHaaf, London Regional Cancer Centre, Ontario; M.E. Arena, S. Cecchetto, S. Gerard, G. Jenkins, T. Koski, M. Laamanen, J. Paquette, Northeastern Ontario Regional Cancer Centre, Sudbury, Ontario; P. Bettello, K. Bishop, Algoma District Health Centre, Sault Ste Marie, Ontario; L. Greeves, D. MacCormack, K. Marek, K. McDonald, E. Stiles, D. Timchishen, T. White, CancerCare Manitoba, Winnipeg; B. Bowman, R. Cooper, H. Darwin, W. Valstar, Tom Baker Cancer Centre, Calgary, Alberta; G. Amyotte, M. Belzile, A. Bulloch, D. Drebit, Cross Cancer Institute, Edmonton, Alberta; D. Ayers, C. Ng, R. Page, C. Schwindt, L. Unger, Vancouver Cancer Centre; M. Fourt, British Columbia Cancer Agency, Fraser Valley Cancer Centre, Surrey; C. Fietz, British Columbia Cancer Agency, Vancouver Island Cancer Centre, Vancouver, British Columbia, Canada.

	ACKNOWLEDGMENTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Supported by the National Cancer Institute of Canada, Kingston, Ontario, Canada.

We thank LeeAnne Hall for data entry and administrative support and Lam Pho for database management.

	REFERENCES

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
1. Schilsky RL: Message from the chair: Priority on institutional compliance. CALGB Newsletter 3: 2–3, 1999

2. Scott J, Cooper M, Larson T: Making clinical trials financially viable. Oncol Issues 15: 14–16, 2000

3. Souhami RL: Clinical trials in cancer: Who should pay? Clin Oncol 5: 269–271, 1993[CrossRef]

4. O’Brien C: British health service balks at cost of cancer trials. Nat Med 3: 707, 1998

5. Ault A: Who should subsidize NCI clinical trials? Nat Med 4: 992, 1998[CrossRef]

6. Mack C: Federal support for clinical trials. Acad Med 73: 774–775, 1998[Medline]

7. Evans WK, Dahrouge S, Stapleton J, et al: An estimate of the cost of conducting phase II trials in lung cancer. Lung Cancer 28: 85–95, 2000[CrossRef][Medline]

8. Fireman BH, Fehrenbacher L, Gruskin EP: Cost of care for patients in cancer clinical trials. J Natl Cancer Inst 92: 136–142, 2000[Abstract/Free Full Text]

9. Arnold K, Vastag B: Medicare to cover routine care costs in clinical trials. J Natl Cancer Inst 92: 1032, 2000[Free Full Text]

10. Wagner JL, Alberts SR, Sloan JA, et al: Incremental costs of enrolling cancer patients in clinical trials: A population-based study. J Natl Cancer Inst 91: 847–853, 1999[Abstract/Free Full Text]

11. Brown ML: Cancer patient care in clinical trials sponsored by the National Cancer Institute: What does it cost? J Natl Cancer Inst 91: 818–819, 1999[Free Full Text]

12. Bennett CL, Stinson TJ, Vogel V, et al: Evaluating the financial impact of clinical trials in oncology: Results from a pilot study from the Association of American Cancer Institutes/Northwestern University Clinical Trials Costs and Charges Project. J Clin Oncol 18: 2805–2810, 2000[Abstract/Free Full Text]

13. Quirk J, Schrag D, Radzyner M, et al: Clinical trial costs are similar to and may be less than standard care and inpatient charges at an academic medical center are similar to major, minor, and non-teaching hospitals. Proc Am Soc Clin Oncol:19:433a, 2000 (abstr 1696)

14. Sherman EJ, Pfister DG, Rubin D, et al: Using patients as their own controls to evaluate the cost of phase I trials. Am Soc Clin Oncol : 433a, 2000 (abstr 1697)

15. Gwede C: CRA workload assessment tools needed: You can help. SoCRA Source 9: 16, 1996

16. Thibault C: Workload measurement systems in nursing. Montreal Quebec Canada, Association des Hopitaux du Quebec, 1990

17. Prescott PA: Nursing intensity: Needed today for more than staffing. Nurs Econ 6: 409–414, 1991

18. Cockerill R, Pallas LO, Bolley H: Measuring nursing workload for case costing. Nurs Econ 11: 342–349, 1993[Medline]

19. Carr-Hill RA, Jenkins-Clarke S: Measurement systems in principle and in practice: The example of nursing workload. J Adv Nurs 22: 221–225, 1995[CrossRef][Medline]

20. Health and Welfare Canada : National hospital productivity improvement program: Pharmacy workload measurement system. Ottawa Ontario Canada, Government of Canada, 1985

21. Hamilton D, Brown G, da Silva J: Clinical pharmacy workload measurement: pharmacokinetic and drug information services. Can J Hosp Pharm 5: 203–210, 1990

22. Choich R Jr: Relationship of productivity analysis to departmental cost-accounting systems. Am J Hosp Pharm 45: 1103–1110, 1988[Abstract]

23. Rajab MH, Kirkpatrick P, Fritz E: Standardizing estimate of study coordinator time to enhance joint collaboration and funding of clinical trials. Control Clin Trials 21: 76S, 2000 (suppl 2, abstr)

24. Russell R, Holland C, Baldwin A, et al: Workload measurement guidelines for clinical trials nurses at the Hamilton Regional Cancer Centre. Network News, NCIC CTG Data Management Newsletter, 1993 (suppl 12)

25. Lichter AL, Schnipper L, Emanual E: Presidential symposium: Report of the Clinical Trials Subcommittee—ASCO 1999 virtual meeting/lecture web site. Http://www.conference-cast.com/asco.org, Accessed March 2000.

26. Hancock RD, Wiland S, Brown NA: Development and testing of a complexity rating scale for clinical trial protocol management. Monitor 1995, pp 36–39

27. Gwede CK, Johnson D, Trotti A: Measuring the workload of clinical research coordinators: Part 1—Tools to study workload issues. Appl Clin Trials 9: 40–44, 2000

28. Gwede CK, Johnson D, Trotti A: Measuring the workload of clinical research coordinators: Part 2—Workload implications for sites. Appl Clin Trials 9: 42–47, 2000

29. Gwede CK: Measurement of CRA workload and productivity: Quest for the magic number! SoCRA Source 18: 9, 1998

30. Paul N, Roche K, Smuck B, et al: Lessons learned from conducting a study to measure workload of cancer clinical trials. Control Clin Trials 29: 57S, 1999 (suppl 2, abstr)

31. Roche K, Paul N, Whitehead M, et al: Predicting trial complexity and time consumption: Further results of the workload measurement study of the National Cancer Institute of Canada Clinical Trials Group. Proc Am Soc Clin Oncol 19: 449a, 2000 (abstr 1763)

32. Paul N, Roche K: Screening patients for clinical trials: Factors affecting the ratio of time spent to accrual. Control Clin Trials 21: 95S, 2000 (suppl 2, abstr)

Submitted February 15, 2001; accepted August 27, 2001.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Facebook    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				S. L. Hillman, M.-W. An, M. J. O'Connell, R. M. Goldberg, P. Schaefer, J. C. Buckner, and D. J. Sargent Evaluation of the Optimal Number of Lesions Needed for Tumor Evaluation Using the Response Evaluation Criteria in Solid Tumors: A North Central Cancer Treatment Group Investigation J. Clin. Oncol., July 1, 2009; 27(19): 3205 - 3210. [Abstract] [Full Text] [PDF]

				E. Atallah, J. Cortes, S. O'Brien, S. Pierce, M. B. Rios, E. Estey, M. Markman, M. Keating, E. J. Freireich, and H. Kantarjian Establishment of baseline toxicity expectations with standard frontline chemotherapy in acute myelogenous leukemia Blood, November 15, 2007; 110(10): 3547 - 3551. [Abstract] [Full Text] [PDF]

				C. M. Moinpour, A. M. Denicoff, D. W. Bruner, A. B. Kornblith, S. R. Land, A. O'Mara, and E. Trimble Funding Patient-Reported Outcomes in Cancer Clinical Trials J. Clin. Oncol., November 10, 2007; 25(32): 5100 - 5105. [Abstract] [Full Text] [PDF]

				J. Y Friedman, J. Sugarman, J. K Dhillon, V. DePuy, C. K Pierre, M. A Dinan, J. S Allsbrook, K. A Schulman, and K. P Weinfurt Perspectives of clinical research coordinators on disclosing financial conflicts of interest to potential research participants Clinical Trials, June 1, 2007; 4(3): 272 - 278. [Abstract] [PDF]

				M. R. Mahoney, D. J. Sargent, M. J. O'Connell, R. M. Goldberg, P. Schaefer, and J. C. Buckner Dealing With a Deluge of Data: An Assessment of Adverse Event Data on North Central Cancer Treatment Group Trials J. Clin. Oncol., December 20, 2005; 23(36): 9275 - 9281. [Abstract] [Full Text] [PDF]

				J. R. Wright and M. N. Levine Researching the Cost of Research J. Clin. Oncol., November 15, 2003; 21(22): 4081 - 4082. [Full Text] [PDF]

				T. G. Roberts Jr, T. J. Lynch Jr, and B. A. Chabner The Phase III Trial in the Era of Targeted Therapy: Unraveling the "Go or No Go" Decision J. Clin. Oncol., October 1, 2003; 21(19): 3683 - 3695. [Abstract] [Full Text] [PDF]

				D. P. Goldman, S. H. Berry, M. S. McCabe, M. L. Kilgore, A. L. Potosky, M. L. Schoenbaum, M. Schonlau, J. C. Weeks, R. Kaplan, and J. J. Escarce Incremental Treatment Costs in National Cancer Institute-Sponsored Clinical Trials JAMA, June 11, 2003; 289(22): 2970 - 2977. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Roche, K. Articles by Walker, H. Search for Related Content PubMed PubMed Citation Articles by Roche, K. Articles by Walker, H. Social Bookmarking                            What's this?