This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Monsky, W. L. Articles by Alberts, D. S. Search for Related Content PubMed PubMed Citation Articles by Monsky, W. L. Articles by Alberts, D. S. Social Bookmarking                            What's this?

Comparison of Young Clinical Investigators’ Accuracy and Reproducibility When Measuring Pulmonary and Skin Surface Nodules Using a Circumferential Measurement Versus a Standard Caliper Measurement: American Association for Cancer Research/American Society of Clinical Oncology Clinical Trials Workshop

From the Department of Radiology, Beth Israel Deaconess Medical Center, Boston, MA; The Arizona Cancer Center, University of Arizona Health Sciences Center, and Cancer Technologies, Inc, Tucson, AZ; and University of Texas Health Science Center at San Antonio and Cancer Therapy and Research Center, San Antonio, TX.

Address reprint requests to David S. Alberts, MD, Arizona Cancer Center, PO Box 245024, 1515 North Campbell Ave, Tucson, AZ 85724-5024; email dalberts{at}azcc.arizona.edu

	ABSTRACT

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
PURPOSE: The clinical investigator must understand that errors in measuring tumors can greatly affect such clinical-trial end points as tumor response. We performed a prospective, controlled study of tumor measurements that compared circumferential measurements made with a loop planimeter with linear measurements made with a standard caliper.

METHODS: Using a cross-over design, 76 clinical oncology fellows/junior oncology faculty members attending a Methods in Clinical Cancer Research Workshop sponsored by the American Association for Cancer Research and the American Society of Clinical Oncology measured five pulmonary nodule phantoms that ranged in size from 1.76 to 13.21 cm² and five surface nodule phantoms with sizes ranging from 2.3 to 12.9 cm². To perform these measurements, they used both a loop planimeter and a caliper. Forty-two and 40 participants repeated measurements 3 days later on pulmonary and surface nodules. Accuracy, reproducibility, and time efficiency were evaluated.

RESULTS: The linear caliper measurements overestimated pulmonary nodule and surface nodule size by a median of 37% and 23%, respectively. Circumferential loop planimeter measurements overestimated pulmonary nodule size and surface nodule size by a median of 8% and 17%, respectively. Interobserver reproducibility for the planimeter was greater than that for the caliper, as evidenced by thinner measurement interquartile ranges. Furthermore, intraobserver reproducibility was higher for the planimeter, with its variability being only 31.4% and 25.5% as large as that of the caliper when measuring the pulmonary and surface nodules, respectively.

CONCLUSION: Circumferential measurements provide better accuracy, reproducibility, and speed in measuring both pulmonary and surface nodules than do perpendicular diameters.

	INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
THE CLINICAL TRIAL literature shows great disparity in reported rates of tumor response. For example, the reported rate of response for colorectal cancer patients treated with fluorouracil ranges from 8% to 85%.¹ Furthermore, the reported rate of response for head and neck cancers treated with methotrexate ranges from 8% to 60%.² Some of this variability is due to differences in patient characteristics and therapy, but some is artifactual and is related to the criteria used for assessing tumor response and the associated errors in measurement.^3-5 The clinical investigator must understand the critical need for accurate and reproducible tumor measurements when evaluating tumor response to therapy.

Surrogate end points such as tumor response (tumor regression or time to tumor progression) in place of, or together with, survival as the primary and/or secondary end points are commonly used in clinical trials. Although measures of palliation and survival may be ideal end points for clinical trials investigating patient benefit, most such trials use the objective measurements of tumor response as a surrogate end point for therapeutic efficacy. These surrogate end points may be controversial and fraught with error.^4-6 The evaluation of tumor response usually depends on measurements of tumor lesions, either directly or with the aid of radiographic studies, at intervals throughout the clinical trial. Serial measurement errors may result in inappropriate designation of progression and abandonment of a potentially effective agent, as well as the false measurement of tumor regression, leading to use of an ineffective agent.⁵ Thus, these measurements should be standardized and accurate, with high inter- and intraobserver reproducibility.

The chest radiograph accounts for half of all medical imaging. Clinical trials investigating antineoplastic agents for primary lung cancer and metastatic tumors to the lungs may use the chest radiograph to evaluate tumor size as a surrogate end point. Although computed tomography (CT) scans are used more often, issues of measurement accuracy and reproducibility apply equally to both methods.⁷ The current standard for measuring a surface nodule or radiographic image of a pulmonary nodule is to use a standard Vernier-type caliper or a ruler to measure the longest perpendicular diameters and then to approximate the surface area of the best-fit rectangle, the product of these diameters. When using the products of the longest perpendicular diameters to calculate tumor surface area, a 27% overestimation is predicted because of the incorrect assumption of rectangular proportions for ellipsoid masses.⁸ This is because tumors are usually spherical or ellipsoid and the corners of the best-fit rectangle may not contain tumor or tumor shadow, in the case of the radiographic image. Furthermore, perception of the longest perpendicular diameters is observer-dependent and is difficult to standardize. In this study, we conducted a prospective controlled study of measurements made by new clinical oncology investigators, who used the loop planimeter and a standard caliper to measure phantoms representing surface nodules and radiographic pulmonary nodules.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Study Design
Seventy-six clinical oncology fellows and junior oncology faculty members attending the American Association for Cancer Research and the American Society of Clinical Oncology Methods in Clinical Cancer Research Workshop volunteered to measure a set of five pulmonary nodule phantoms on anteroposterior chest radiographs and five surface nodule models (see Tumor Models, below). Measurements of all 10 nodules were made with the loop planimeter and with the caliper of a Tumorimeter (Cancer Technologies, Inc, Tucson, AZ) (see Tumorimeter Design, below). This was a randomized cross-over comparison in that participants used the loop planimeter first for all measurements if they were born in an odd-numbered month or the caliper first if their month of birth had an even number. All linear measurements of the longest diameter and its perpendicular diameter made with the caliper were multiplied to yield a rectangular tumor area, as called for in most clinical trial protocols. The times to measure each nodule with the loop planimeter and the caliper were recorded, not including the time needed to multiply perpendicular diameters when using the caliper. Three days after their initial measurements, 42 of 76 participants repeated measurements on the pulmonary nodules, and 40 repeated those on the surface nodules, with 39 participants common to the two sets of repeated measurements.

Tumor Models
Paraffin sheets were molded into roughly spheroid phantom nodules to mimic pulmonary nodules. These phantoms, which approximated soft tissue opacity on chest radiographs, were placed in a human chest phantom model available at the Beth Israel Deaconess Medical Center Department of Radiology and frontal (posteroanterior) radiographs were obtained. The actual sizes of these phantoms were determined by measuring spiral-CT–acquired (General Electric, Fairfield, CT) contiguous axial images with CT-associated measuring software to obtain circumferential measurements. The magnification of the phantoms on the chest radiographs was also accounted for (true area = measured area/[distance from x-ray tube to film/distance from x-ray tube to center of the nodule phantom]).² The phantom pulmonary nodules measured 1.76, 4.72, 8.20, 13.21, and 13.21 cm² (Fig 1 A).

View larger version (93K): [in this window] [in a new window]   Fig 1. (A) Chest radiograph shows phantom nodules placed in a human chest phantom model. (B) Simulated surface nodules were placed on a foam cushion and covered with felt chamois.

Tumorimeter Design
The Tumorimeter (Fig 2) is a plastic device that has a separate caliper attached to a sliding scale for linear measurements in centimeters as well as a loop planimeter constructed of flexible plastic tubing attached to a sliding scale for circumferential measurements in square centimeters, as described by Dorr and Alberts.⁸

View larger version (49K): [in this window] [in a new window]   Fig 2. The Tumorimeter has a flexible loop for approximating surface area in square centimeters and a caliper for linear measurements in centimeters. As the thumbwheel extends the loop and the caliper, the slider with the calibrated scale indicates tumor surface area and caliper distance.

To see whether the measurement techniques differed from each other in their estimates of tumor size, differences were computed between the caliper and planimeter measurements for each nodule. Because the distributions were nonnormal and asymmetric, the sign test was used to determine whether the median differences varied from zero.

The accuracy of each of the measurement techniques was assessed by computing the percentage of overestimation of tumor size for each nodule with each measurement technique, defined as the measured size minus the "true" size, divided by the "true" size, expressed as a percentage. The sign test was used to determine whether the median percentage of overestimations differed from zero. To provide an overall estimate of the accuracy of each technique for a given tumor variety, the median percentage overestimation arising from measurements across all nodules and trials, within a given tumor variety, was computed for each measurement technique.

The interobserver reproducibility of each of the techniques was examined by computing interquartile ranges for each of the nodule measurements made by each of the techniques. The typical approach of computing the variance was inappropriate because of excessive skewness and/or outliers in the measurements. The intraobserver reproducibility of each of the techniques was approximated by computing the variance of the differences between nodule measurements made on two separate trials of the same tumor nodule by the same participant. (Thus, only participants who performed replicate analyses were used in the determination of reproducibility; n = 42 for the pulmonary nodules and n = 40 for the surface nodules.) It is understood that this is an underestimate of the variance because it assumes independence of the replicate measurements. From these variances, approximate estimates of the coefficients of variation were computed for each measurement technique on a given nodule. Because the differences between replicate measurements were approximately normal, a separate multivariate repeated-measures response model⁹ was fit to the differences for each of the pulmonary and surface nodule varieties, allowing for estimation of effects on the mean difference and on the relative variability of the differences attributable to measurement technique and nodule size. The same multivariate repeated-measures response model, fit to the natural log of the measurement times, was used to determine how the first- versus the second-trial measurement technique and nodule affected the time to complete each measurement. Throughout this article, statistical significance is defined using two-sided tests.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
Difference Between Caliper and Planimeter Measurements
Differences between techniques (caliper measurement minus planimeter measurement) were computed for each nodule measured by each participant. The median differences and respective sign tests for each nodule were determined. For the pulmonary nodule measurements for both trials, the median differences were all positive and all significantly greater than zero (P < .01 for each of the nodules). For the surface nodule measurements, only the two largest nodules had positive medians that were significantly greater than zero during both trials (P < .03 for the two largest nodules). Thus, measurements of pulmonary nodules on radiographs and the two largest surface nodules obtained with the caliper are significantly larger than those made by the loop planimeter (Fig 3).

View larger version (33K): [in this window] [in a new window]   Fig 3. Median percentage overestimations of tumor size comparing the loop planimeter and caliper measuring devices. The number 1 designates that the median is not different from zero. The number 2 designates that the medians are not different between caliper and planimeter.

When the five surface nodules with true surface areas of 2.3, 3.0, 3.1, 4.8, and 12.9 cm² were measured using calipers, the median overestimations of nodule size, for first-time measurements, were 4%, 27%, 23%, 30%, and 27%, respectively (P < .0001 for the four largest nodules). The loop planimeter measurements of these surface nodules had median overestimations of 13%, 23%, 23%, 21%, and 16%, respectively (P < .0001 for the four largest nodules) (Fig 3). Similar results were obtained during the repeated measurements. Thus, the loop planimeter provided significantly less overestimation of the two largest surface nodule phantoms.

The overall median percentage overestimations for the caliper were 36.6% and 22.6% for the pulmonary and surface nodules, respectively, compared with 8.0% and 16.7% for the planimeter.

Reproducibility for the Caliper and Planimeter Techniques
The interobserver reproducibility of a given measurement technique was quantified using the interquartile ranges of the measurements. Figure 4 shows plots of the interquartile ranges for each of the pulmonary and surface nodule measurements. Except for the 8.20-cm² nodule, the interquartile ranges on the pulmonary nodules were larger for the caliper than for the planimeter during both the initial and repeated measurements. The initial and repeated surface nodule measurements yielded similar interquartile ranges between the caliper and planimeter techniques (differing by less than 0.15 cm²) for the 2.3-, 3.0-, and 3.1-cm² nodules, but the larger surface nodules, particularly the 12.9-cm² nodule, differed by as much as 4.8 cm². Thus, the loop planimeter seems to provide greater interobserver reproducibility than the caliper for measuring all pulmonary nodules and the two largest surface nodules.

View larger version (31K): [in this window] [in a new window]   Fig 4. Interquartile ranges (in square centimeters) of tumor measurements, comparing the loop planimeter and caliper measurement devices.

For both tumor types, measurement technique and tumor size had no effect on the mean differences; however, the measurement technique and tumor size significantly influenced variability in the differences between replicates, which reflects intraobserver reproducibility. Specifically, when pulmonary nodules were measured with the planimeter, the variability was only 31.4% as large as that obtained with the caliper. For the surface nodules, use of the planimeter resulted in variability that was only 25.5% as large as that with the caliper. Figure 5 provides plots of the mean differences between replicates ± 2 SDs. Evidence also suggests that the variability increases with increasing surface nodule size but not with increasing pulmonary nodule size. Finally, Fig 6 provides approximate coefficients of variation. The planimeter measurements of the pulmonary nodules had a consistently lower coefficient of variation. For the surface nodule measurements, the differences between the techniques were quite pronounced, with the planimeter’s coefficients typically lower by 30%.

View larger version (14K): [in this window] [in a new window]   Fig 5. Mean differences between replicate measurements (in square centimeters) ± 2 SD.

View larger version (16K): [in this window] [in a new window]   Fig 6. Approximate coefficients of variation (%) comparing the loop planimeter and caliper measurement devices.

The time for making each measurement was also recorded. Mean measurement times with the planimeter were 5 to 8 seconds less than those with the caliper (P < .0001 for each of the nodules), not including the time taken to multiply the two perpendicular diameters of a nodule. A learning curve was identified in that the repeated measurements, taken 3 days after the first measurements, took less time on average than the first measurements, regardless of the measurement technique (P = .05 for each of the nodules).

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
When conducting clinical trials using tumor response, tumor regression, or time to tumor progression as an end point, measurement error may significantly influence the interpretation of the results. Every effort should be taken to ensure that measurements are standardized and accurate with high inter- and intraobserver reproducibility, because serial measurement errors may cause incorrect designation of disease progression and false measurement of tumor regression. In this study, we demonstrated that pulmonary nodules on chest radiographs and solid-surface nodules larger than 4.5 cm are more accurately measured with circumferential measurements using a loop planimeter than with perpendicular diameters obtained with a caliper. Furthermore, empirical evidence suggests that interobserver reproducibility is greater when making circumferential measurements with the loop planimeter and that intraobserver reproducibility is significantly greater with the planimeter.

Linear tumor measurement error directly affects the interpretation of tumor response rate. Moertel and Hanley⁴ showed that if a 50% reduction in the product of perpendicular diameters is the criterion, the response rate due to measuring error alone (without change in lesion size) is 7.8%, whereas if a 25% reduction criterion is used, the placebo response rate is 19%. Furthermore, a reduction of nodule size to 40% of the initial area was detected only 73% of the time. Previously, it was demonstrated that linear measurements of diameter deviate from the true diameter by 12% on average.^4,10 Moreover, Dorr and Alberts⁸ reported a 26% overestimation in pulmonary nodule size when making perpendicular width measurements with calipers. Here we further demonstrate overestimation of the size of pulmonary nodules on a frontal chest radiograph of up to 54% (median, 36.6%) when using linear measurements. In contrast, the error associated with circumferential measurements of pulmonary nodule size varied between -9% and 22% (median, 8%).

Besides overestimating nodule size, investigators using linear caliper measurements demonstrate substantial measurement bias when approximating lesion dimensions to even integers so that they can simplify the multiplication of the perpendicular diameters.^4,6 These linear-measurement errors are compounded further when multiplying the perpendicular diameters. The current results with circumferential measurements using the planimeter are consistent with those using transparent tracings or precut clear gauges, which have been documented to have greater accuracy.^11,12 Measurement of actual malignant lesions has a higher rate of error than does that of simulated nodules.⁵ This may be due to interobserver differences in ability to detect the edges of an actual nodule and the longest perpendicular diameter. To limit the differences in observer performance when evaluating accuracy and reproducibility, we used phantom pulmonary nodules and a surface nodule model.

Besides accuracy, we assessed the inter- and intraobserver reproducibility. Interobserver reproducibility for the planimeter seems greater than that for the caliper for the pulmonary nodules and the two largest-surface nodules, as evidenced by thinner interquartile ranges. Additionally, intraobserver reproducibility, as assessed by the variability in the differences between replicate measurements, was significantly higher when the planimeter was used. Pulmonary nodule measurements made with the planimeter had a variability that was only 31.4% as large as that with the caliper. Variability of surface nodule measurement with the planimeter was 25.5% as large as that with the caliper.

On average, a mean of 5 to 8 seconds less time was required to measure tumor nodules using the loop planimeter. Although this is statistically significant, it is probably not of practical importance. However, calculations of the product of the perpendicular diameter were not included in the timing when the caliper was used for area measurements. These multiplication steps are time-consuming and subject to additional error.

The results reported here demonstrate that young clinical investigators can improve accuracy and reproducibility by using a loop planimeter to make circumferential measurements of pulmonary and surface nodules. This reduction in error can lead to more accurate interpretation of clinical trial end points, such as tumor response.

	ACKNOWLEDGMENTS

 
Research support provided by grant no. 1R25CA68647 for the American Association for Cancer Research/American Society of Clinical Oncology Workshop for Methods in Clinical Cancer Research, a National Cancer Institute Research Fellowship (T32) granted through the Department of Radiology, Beth Israel Deaconess Medical Center, and grant no. CA-23074 (Arizona Cancer Center Core Grant) from the National Institutes of Health–Health and Human Services, Bethesda, MD.

We acknowledge the American Association for Cancer Research and the American Society of Clinical Oncology for organizing the Methods in Clinical Cancer Research Workshop and for their commitment to training young clinical investigators.

	REFERENCES

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

 
1. Moertel CG, Thynne GS: Large bowel, in Holland JF, Frei E III (eds): Cancer Medicine (ed 2). Philadelphia, PA,Lea & Febiger, 1982, pp 1830-1859

2. Tannock IF: Chemotherapy for head and neck cancer. J Otolaryngol 13:99-104, 1984[Medline]

3. Tonkin K, Tritchler D, Tannock I: Criteria of tumor response used in clinical trials of chemotherapy. J Clin Oncol 3:870-875, 1985[Abstract/Free Full Text]

4. Moertel CG, Hanley JA: The effect of measuring error on the results of therapeutic trials in advanced cancer. Cancer 38:388-394, 1976[Medline]

5. Warr D, McKinney S, Tannock I: Influence of measurement error on assessment of response to anticancer chemotherapy: Proposal for new criteria of tumor response. J Clin Oncol 2:1040-1046, 1984[Abstract]

6. Lavin PT, Flowerdew G: Studies in variation associated with the measurement of solid tumors. Cancer 46:1286-1290, 1980[Medline]

7. Hopper KD, Kasales CJ, Van Slyke MA, et al: Analysis of interobserver and intraobserver variability in CT tumor measurements. Am J Roentgenol 167:851-854, 1996[Abstract/Free Full Text]

8. Dorr RT, Alberts DS: Quantitation of ellipsoid tumor areas using a circumferential measuring device. Med Oncol Tumor Pharmacother 5:249-251, 1988[Medline]

9. Galecki AT: General class of covariance structures for two or more repeated factors in longitudinal data analysis. Commun Stat Theory Methods 23:3105-3119, 1994

10. Kodlin D, Cohn I Jr: Simulation experiments of tumor measurements in clinical trials. Treat Rep 62:2077-2083, 1978

11. Brigham BA, Bunn PA Jr, Minna JD, et al: Growth rates of small cell bronchogenic carcinomas. Cancer 42:2880-2886, 1978[Medline]

12. Gordon AB: A tumour gauge to measure breast lumps for use with mammograms. Surg Cancer 11:197-198, 1984

Submitted May 4, 1999; accepted August 11, 1999.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Facebook    Reddit    Technorati    Twitter    What's this?

This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Monsky, W. L. Articles by Alberts, D. S. Search for Related Content PubMed PubMed Citation Articles by Monsky, W. L. Articles by Alberts, D. S. Social Bookmarking                            What's this?