Originally published as JCO Early Release 10.1200/JCO.2008.19.9182 on December 22 2008

This Article Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Anderson, J. R. Articles by Krailo, M. D. Search for Related Content PubMed PubMed Citation Articles by Anderson, J. R. Articles by Krailo, M. D. Related Articles Related Articles Related Reply Social Bookmarking                            What's this?

Problems Identified With Phase II Stopping Rules That Employ Response and Early-Progression Rates

College of Public Health, University of Nebraska Medical Center, Omaha, NE

Mark D. Krailo

College of Medicine, University of Southern California, Los Angeles, CA

To the Editor:

Goffin and Tu¹ report using simulation to create phase II stopping rules that depend both on response and early-progression rates. However, there appear to be some problems with the rules as defined in Tables 1, 2, and 3 of their report.

If the study were to progress to stage 2, 30 patients would be studied. The drug would be rejected (null hypothesis accepted) if the number of responders were three (10%) or fewer, and the number of early progressive diseases were 16 (53%) or greater. The drug would be accepted (null hypothesis rejected) if the number of responders were eight (27%) or greater, or the number of early progressive diseases were 10 (33%) or fewer. Note, however, that there is a region indicating no decision at stage 2 between the drug acceptance and drug rejection regions. For instance, if the number of responders were between zero and seven, and the number of early progressive diseases were between 11 and 15, the drug would be neither rejected nor accepted.

Furthermore, some of the rules in Tables 1 to 3 do not appear to have the operating characteristics reported by Goffin and Tu.¹ We performed a simulation study (100,000 simulations) in which data were generated from the null and alternative hypotheses specified in this letter, and conducted a two-stage phase II study and applied the stopping rules as outlined by Goffin and Tu in Tables 1 to 3.

For the Table 1 rule (full-space method), the drug acceptance rate ( error) at the null hypothesis is estimated to be .003 (Table 1). However, the no-decision rate is .085, so the do-not-reject rate is about .088. The rate of stopping the trial at stage 1 under the null hypothesis is estimated to be 88%, less than the 98% reported in the authors' Table 1. For response and early-progression rates on the border of the alternative hypothesis, the no-decision rates range from 33% to 38%. If power is defined as the percentage of time that the study results fall into the drug acceptance region under various alternative hypotheses, power for the full-space method rule is estimated to be between 23% and 40%. If it is defined as the percentage of time that the study results fall into the do-not-reject region, power for the full-space method rule is between 60% and 78%, less than the 93.5% value for power reported in the authors' Table 1.

For the Table 2 rule (borderline-value method), the no-decision rates are much smaller (because the no-decision region is smaller), and are estimated to be less than 4% in all simulations. The error for the Table 2 rule is estimated to be approximately 11% (excluding the no decisions). This is substantially greater than the value of 2.5% reported in Table 2. The probability of stopping at stage 1 is estimated to be 68%, again, less than the 90% reported by the authors. Power (defined as the percentage of time that the study results fall into the drug acceptance region) for the borderline-value method rule is estimated to be between 74% and 91%.

The Table 3 rule (interesting and uninteresting drugs that have border values) does not have a no-decision region. We were able to confirm the error of .111, and the probability of stopping at stage 1 under the null hypothesis of .68, reported in Table 3. Power for this rule is estimated to be between 73% and 91%.

Goffin and Tu¹ use a nonstandard definition of power in their report. Their power definition has a Bayesian flavor to it. Alternatives are picked at random from all or part of the alternative hypothesis space. Averaging over a particular alternative hypothesis space seems to amount to the expected value of power over the values for the response and early-progression rates thought to be consistent with a good drug. For their Table 1, the alternative values for the response and early-progression rates seem to be any in the alternative hypothesis space (response rate, 0% to 20%; early-progression rate, 40%). For their Table 2, it seems the alternative values are restricted to the border of the alternative hypothesis space closest to the null hypothesis (response rate, 0% to 20%; early-progression rate, 40%; response rate, 20%; early-progression rate, 40% to 80%).

Power is usually defined as the probability of rejecting the null hypothesis at a fixed alternative in the alternative hypothesis space. Many rules for the fixed alternatives listed in the simulation results table have inadequate power.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest.

REFERENCES

1. Goffin JR, Tu D: Phase II stopping rules that employ response rates and early progression. J Clin Oncol 26:3715–3720, 2008.[Abstract/Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Facebook    Reddit    Technorati    Twitter    What's this?

Related Articles

• In Reply
  John R. Goffin
  JCO 2009 27: 647 [Full Text]
• Phase II Stopping Rules That Employ Response Rates and Early Progression
  John R. Goffin and Dongsheng Tu
  JCO 2008 26: 3715-3720 [Abstract] [Full Text]

Related Reply

• In Reply
  John R. Goffin
  JCO 2009 27: 647 [Full Text]

This Article Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Anderson, J. R. Articles by Krailo, M. D. Search for Related Content PubMed PubMed Citation Articles by Anderson, J. R. Articles by Krailo, M. D. Related Articles Related Articles Related Reply Social Bookmarking                            What's this?