This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Raab, S. S. Articles by Grzybicki, D. M. Search for Related Content PubMed PubMed Citation Articles by Raab, S. S. Articles by Grzybicki, D. M. Social Bookmarking                            What's this?

The "Big Dog" Effect: Variability Assessing the Causes of Error in Diagnoses of Patients With Lung Cancer

Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Uma Krishnamurti, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki

From the Department of Pathology, University of Pittsburgh School of Medicine; Western Pennsylvania Hospital; Department of Family Medicine and Clinical Epidemiology, University of Pittsburgh School of Medicine, Pittsburgh, PA; Henry Ford Health System, Detroit, MI; University of Iowa Healthcare, Iowa City, IA; Wake Forest University Medical Center, Winston-Salem, NC; and the Loyola University Medical Center, Maywood, IL

Address reprint requests to Stephen S. Raab, MD, Department of Pathology, 5230 Centre Avenue, UPMC Shadyside Hospital, Pittsburgh, PA 15213; e-mail: raabss{at}upmc.edu

	ABSTRACT

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 
PURPOSE: The frequency of diagnostic error in patients who have a lung mass and a pathology specimen is as high as 15%. This study examined the role of inter-pathologist agreement in identifying the cause of error in these patients.

METHODS: Pathologists from six institutions reviewed the slides of 40 patients who had a pulmonary specimen false-negative diagnosis. The initial assessment of error cause arose from cytologic-histologic correlation slide review of discrepant diagnostic samples in patients who had both a bronchial brushing cytologic and surgical specimen. The cause of error was attributed either to clinical sampling (diagnostic material obtained in one but not the other sample) or interpretation (pathologist failed to identify the salient diagnostic features). The pairwise kappa () statistic was used to calculate interobserver agreement between the review and original diagnoses and between the separate review diagnoses.

RESULTS: The pairwise statistic ranged widely from –0.154 to 1.0, and the pairwise statistic of the slides from one institution was undetermined because that institutional pathologist never made the assessment that error was secondary to interpretation. Agreement for observers within the same institution was better than agreement between observers from different institutions.

CONCLUSION: Pathologists exhibit poor agreement in determining the cause of error for pulmonary specimens sent for cancer diagnosis. We developed a psychosocial hypothesis (the "Big Dog" Effect) that partially explains biases in error assessment. This lack of agreement precludes confident targeting of these errors for quality improvement interventions with prospects of success across a variety of institutions.

	INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 
Cancer diagnoses are usually based on the cytologic or histologic evaluation of tissues.¹ The reproducibility of pathologist error assessments in cancer diagnosis from one institution to another has not been systematically studied. However, this reproducibility has implications for the evaluation, selection, and monitoring of error reduction in cancer diagnosis and for the medical legal assessment of failure or delay in diagnosis as a form of malpractice.

The frequency and type of errors in cancer diagnosis depend on the method of detection.¹ A commonly used method is cytologic-histologic (CH) correlation,^2,3 in which a pathologist reviews the diagnoses of cytologic and surgical specimens from the same anatomic site (eg, fine needle aspiration and surgical excision of the thyroid gland). In this exercise, the correlated diagnoses are classified as either concordant or discordant. Raab et al reported that up to 11.8% of all nongynecologic CH correlation pairs were discordant.¹ The first step of a root cause analysis, as performed by most laboratories, involves the review of the slides from the discordant cases to determine whether the source of error is sampling (ie, diagnostic material obtained in one but not the other sample) or interpretation (diagnostic material is present in both samples but misinterpreted in one sample).^2,4 For 15 years, CH correlation has been mandated by federal regulation on the assumption that the detection and investigation of discrepancies improves diagnostic accuracy and clinical care.⁵

In a single institutional study of diagnostic cancer errors detected by the CH correlation method, Clary et al reported that on the original review, 66% of nongynecologic errors were secondary to sampling and 34% of errors were secondary to interpretation.² The interobserver variability of the rereview assignment of error cause was calculated using the pairwise kappa () statistic and indicated poor agreement (range, 0.015 to 0.156).² In a combined set of gynecologic and nongynecologic CH correlation errors, Raab et al reported that the pairwise statistic ranged widely (0.118 to 0.737).¹

This report summarizes the first multi-institutional study examining the inter- and intraobserver agreement in the attribution of the causes of errors in lung cancer diagnosis, as they were detected by CH correlation. The six institutions that participated in this project are members of a consortium studying patient safety under an Agency for Healthcare Research and Quality (AHRQ)-funded quality improvement initiative.⁶

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 
Background and Design
In 2002, AHRQ provided funding to four institutions to: determine baseline error frequencies detected by different methods; determine the clinical impact of diagnostic errors; perform root cause analysis to derive error reduction strategies, and; assess the success of these error reduction strategies.⁶ Subsequently, two additional institutions joined in the consortium project. Institutional review board approval for performance of this project was obtained at all sites.

Description of the CH Correlation Review Process
Because the federal regulations, developed as stipulated by the Clinical Laboratory Improvement Amendments of 1988, do not mandate how CH correlation is to be done,⁵ laboratories perform CH in a variety of ways.⁷ In the beginning of the project, we standardized CH correlation in the participating institutions. The top of Figure 1 shows the patient events that would lead to a CH correlation being performed. This laboratory examined their information system on the first of every month and detected this cytology and surgical pathology specimen pair as originating from the same anatomic site.

View larger version (16K): [in this window] [in a new window]   Fig 1. Patient and laboratory events involved in same site cytology and surgical pathology sampling and cytologic-histologic correlation.

In our standardized CH process, the review pathologist examined all microscopic slides and determined if the cytologic diagnosis, surgical pathology diagnosis, neither diagnosis, or both diagnoses were the cause of discrepancy.⁶ The review pathologist then assigned an underlying root cause for the error using two categories—interpretation or sampling. An interpretation error was an error in categorization. Interpretive errors further were classified as overcalls if the review diagnosis was more than two categoric steps lower than the original or undercalls if the review diagnosis was more than two categoric steps higher than the original. A sampling error was an error in which diagnostic material was not present on the discrepant slide, even on review. In this study, we examined review by a single pathologist because such review is the most common way that American laboratories perform their regulatory duty of CH correlation.⁷

Case Selection
In our multi-institutional AHRQ data set, pulmonary CH discrepancies were the most frequently detected nongynecologic error.¹ Specimen pairs of a bronchial brushing and a surgical lung biopsy made up the most commonly reported pair of discrepant pulmonary specimens. Table 1 shows the frequencies of pulmonary CH correlation discrepancies in our data collected for 2002. The frequency of pulmonary error is the number of discrepant bronchial brushing/surgical specimens divided by the total number of correlating pairs (discrepant and nondiscrepant). Sites B and D could not obtain denominator data because of laboratory information system inability to retrieve necessary data elements. Raab et al¹ reported that a statistically significant association existed between institution and error cause (P < .001). For most institutions, errors were not assigned to surgical pathology misinterpretation, and institutions A, B, and D attributed the majority of errors to cytology sampling.

Data Analysis
All statistical analyses were performed using the statistical software package Statistical Package for the Social Sciences, version 12.0 (SPSS, Chicago, IL) by one of the coauthors (J.E.J.). We used the statistic to measure interobserver agreement. The 95% CI for each statistic was calculated to assist with the interpretation of the statistic. We did not perform hypothesis testing, as our purpose in measuring agreement was to assess its level and not to compare levels between pathologist pairs.

The pairwise statistic measured the agreement between the pathologists' reviews and the original reason for discrepancy. For the cases in which each project site reviewed slides that they contributed, the pairwise statistic represented intrasite agreement. Otherwise the extent of agreement was inter-site. We also performed a pairwise statistic to determine the agreement between all review pathologist pairs for four separate slide study sets submitted by project sites A, B, C, and D. The pairwise statistic was calculated to measure pathologist agreement for categorizing reasons for errors into one of the two categories of sampling or interpretation. If a project site classified every error into one class (eg, interpretation), and another project site classified errors into two classes, the pairwise statistic was undetermined. If the statistic was undetermined, we added a value of 1.0 to each cell in order to determine the statistic; in these cases we reported the original undetermined statistic and the statistic after the recalculation. We could not further perform the statistic for the dichotomous separation of cytologic or surgical specimen error because of low case numbers within the sample type categories.

We interpreted the pairwise statistic as follows: 0.800 to 1.0, excellent agreement; 0.600 to 0.800, good agreement; 0.400 to 0.600, fair agreement; 0.400, poor agreement.^8,9 To illustrate how pairwise statistics correspond to the actual adjudication process, three examples are provided in Table 3. In these scenarios, two pathologists blindly adjudicated the cause of discrepancy, as either interpretation or sampling, in 10 fictitious cases. The adjudicated causes of discrepancy resulting in pairwise statistics of 0.800 (excellent agreement), 0.600 (good agreement), and 0.400 (poor agreement) are shown.

View this table: [in this window] [in a new window]   Table 3. Examples Illustrating How Pairwise Statistics Correspond to Excellent, Good, and Poor Agreement

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

View this table: [in this window] [in a new window]   Table 4. Pairwise Statistic and 95% CIs for the Review Reason for Discrepancy Compared With the Original Reason for Discrepancy

Table 5 shows the pairwise statistic for the review pathologists as they classified the four study sets. The pairwise statistic varied by project site study set and ranged from –0.250 to 1.0, although most values indicated poor agreement. The inter site agreement was highly variable between study sets. Overall, the highest agreement was seen with the project site B slide study set. For the project site C study set, the pairwise statistic was less than 0.400 for every comparison; this study set had contained the majority of interpretive errors.

View this table: [in this window] [in a new window]   Table 5. Pairwise Statistic and 95% CIs for the Individual Pathologists Reviewing the Four Project Sites' Study Sets

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

We hypothesize that the lack of inter site agreement is secondary to fundamental differences in how pathologists evaluate instances of diagnostic discrepancy. We term this phenomenon as the "Big Dog" effect; senior experienced pathologists at each institution serve as the final arbitrator for error cause and use different methods and approaches to decide whether discrepancies exist and their causes. When Big Dogs are confronted with differing assessments from other Big Dogs, they remain reticent to change their opinions. As part of this project, the Big Dogs conjointly evaluated errors on several occasions. The locally dominant pathologists in every case did not change their assessments to a significant extent. Pathologists tend to operate in local environments with little exposure to outside opinion; this has led to poor inter-institutional diagnostic agreement for particular case types. The local nature of CH correlation reference standards is a potential source of error that rarely is addressed.

A corollary to the Big Dog effect is the "Little Dog" effect, which operates at the local institutional level. Most institutions have only one Big Dog to whom other pathologists (Little Dogs) at the same institution defer diagnostic judgment (bite). Institution B exhibited the highest intrasite value (0.800), because the pathology group maintained the same hierarchical structure (same Big Dog) for the original and rereview CH process. Institution C had the next highest value, because slides were drawn from two hospitals, and the Big Dog exhibited control at only one of the hospitals (disagreements were with slides from the other hospital). Institution D had a new Big Dog who performed the rereview and disagreed with the previous Big Dog ( value undetermined). Institution A had maintained the same Big Dog, but the CH process was performed differently; originally a cytology fellow had made the error cause assessment and was reluctant to attribute error to interpretation; the Big Dog had no such qualms.

In our study, the pairwise statistic was undetermined for the original error assignment for one institutional pathologist, because that pathologist reported that poor sampling caused all errors. This assignment was caused by biases in the institutional CH correlation process.⁷ The institutional criteria for actual CH correlation slide review are varied. Vrbin et al⁷ showed that no two American laboratories measured the same metrics during the CH correlation process. The Clinical Laboratory Improvement Amendments of 1988⁵ has, thus, mandated that all laboratories measure an incommensurate phenomenon. In the process of conducting this study, we discovered that some laboratories, in fact, did not review the surgical slides and automatically attributed the error to cytology sampling, thus precluding their ability to identify any surgical pathology error.

Although the CH correlation process has been mandated by law on the assumption that it detects the causes of errors, some institutions use it in ways that systematically bias assignment of error to the preanalytic phase of testing (sampling).¹ For example, cytology trainees are biased not to attribute interpretive error to their mentors. Similar to nonpathology error detection methods, the CH correlation process is driven by fear of disclosure and fear of blame.^10-12 Consequently, laboratories generally avoid using CH correlation data for error reduction.^13,14 Most laboratories expend considerable resources performing CH correlation purely because they are required to do so by law and not because they have reasonable hopes of learning to prevent errors based on the findings of this exercise.

Noncorrelating CH correlation cases tend to be challenging.² Nodit et al¹⁵ reviewed 32 pulmonary CH correlation pairs, in which the cytology case was a bronchial washing or brushing; root cause analysis showed that an interpretive error was the originally assigned cause in 50% of cases. However, the diagnosis of malignancy was straightforward on rereview in only one case, indicating that less than optimal sampling contributed to even those errors that were considered interpretive.¹³ Most pulmonary errors are caused by the interplay of both poor sampling and misinterpretation. This factor may have contributed to the lack of agreement in this study; some pathologists were more likely than others to attribute error to interpretation even if they saw only a few malignant cells.

Although the statistic can verify that agreement exceeds chance levels, there is controversy over its use to quantify the level of agreement among two or more observers.^16-18 For example, the statistic can be affected in complex ways by bias between observers. Although we recognize that there are limitations in interpreting the statistic, this study suggests that there is marked variability in the assessment of assigning root cause of error in this quality assurance method.

Conclusion
The largest source of error within pathology is interobserver variability,^1,19-21 but pathologists are reluctant to devise interventions that detect these errors. For controversial areas such as proliferative and preneoplastic breast disease, authors showed that diagnostic variability was reduced if reviewers applied accepted criteria.^22-25 However, a problem in many areas of pathology is the lack of outcomes data to drive the acceptance of one set of criteria over another. Consequently, there is no consensus on the optimal diagnostic criteria. In the cases typical of this study, there is a lack of standardization on how to perform CH correlation review,⁷ a lack of criteria for nongynecologic specimen adequacy, and a lack of diagnostic uniformity for malignancy in cases containing few malignant cells or cases of well differentiated malignancy. The latter may be related to experience and/or the willingness of some pathologists to diagnose cancer when the evidence is scanty. These issues need to be addressed in order to reduce the variability in assignment of error cause in CH correlation review cases.

	Authors' Disclosures of Potential Conflicts of Interest

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 
Although all authors completed the disclosure declaration, the following authors or their immediate family members indicated a financial interest. No conflict exists for drugs or devices used in a study if they are not being evaluated as part of the investigation. For a detailed description of the disclosure categories, or for more information about ASCO's conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors.

Authors Employment Leadership Consultant Stock Honoraria Research Funds Testimony Other Stephen S. Raab AHRQ (B) Frederick A. Meier AHRQ (B) Richard J. Zarbo AHRQ (B) D. Chris Jensen AHRQ (B) Kim R. Geisinger AHRQ (B) Uma Krishnamurti AHRQ (B) Chad H. Stone AHRQ (B) Janine E. Janosky AHRQ (B) Dana M. Grzybicki AHRQ (B)

Dollar Amount Codes (A) < $10,000 (B) $10,000-99,999 (C) $100,000 (N/R) Not Required

	Author Contributions

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 

Conception and design: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Uma Krishnamurti, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki Financial support: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Uma Krishnamurti, Janine E. Janosky, Dana M. Grzybicki Administrative support: Stephen S. Raab, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Uma Krishnamurti Provision of study materials or patients: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Uma Krishnamurti, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki Collection and assembly of data: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Uma Krishnamurti, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki Data analysis and interpretation: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki Manuscript writing: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki Final approval of manuscript: Stephen S. Raab, Frederick A. Meier, Richard J. Zarbo, D. Chris Jensen, Kim R. Geisinger, Christine N. Booth, Uma Krishnamurti, Chad H. Stone, Janine E. Janosky, Dana M. Grzybicki

 

	NOTES

 
Supported by Agency for Healthcare Research and Quality Grant No. R01 HS13321-01.

Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.

	REFERENCES

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION Authors' Disclosures of... Author Contributions REFERENCES

 
1. Raab SS, Grzybicki DM, Janosky JE, et al: Clinical impact and frequency of anatomic pathology errors in cancer diagnosis. Cancer 104:2205-2213, 2005[CrossRef][Medline]

2. Clary JM, Silverman JF, Liu Y, et al: Cytohistologic discrepancies: A mean to improve pathology practice and patient outcomes. Am J Clin Pathol 117:567-573, 2002[Abstract/Free Full Text]

3. Jones BA, Novis DA: Cervical biopsy-cytology correlation: A College of American Pathologists Q-Probes study of 22,439 correlations in 348 laboratories. Arch Pathol Lab Med 120:523-531, 1996[Medline]

4. Joste NE, Crum CP, Cibas ES: Cytologic/histologic correlation for quality control in cervicovaginal cytology: Experience with 1,582 paired cases. Am J Clin Pathol 103:32-34, 1995[Medline]

5. Department of Health and Human Services, Health Care Financing Administration: Clinical laboratory improvement amendments of 1988: final rule, 57 Federal Register 7146. 1992, codified at 42 CFR 493

6. Raab SS: Improving patient safety by examining pathology errors. Clin Lab Med 24:849-863, 2004[CrossRef][Medline]

7. Vrbin CM, Grzybicki DM, Zaleski MS, et al: Variability in cytologic-histologic correlation practiced and implications on patient safety. Arch Pathol Lab Med 129:893-898, 2005[Medline]

8. Cicchetti DV: Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 6:284-290, 1994[CrossRef]

9. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977[CrossRef][Medline]

10. Leape LL: A systems analysis approach to medical error. J Eval Clin Pract 3:213-222, 1997[CrossRef][Medline]

11. Leape LL, Berwick DM: Five years after to err is human: What have we learned? JAMA 293:2384-2390, 2005[Abstract/Free Full Text]

12. Waring JJ: Beyond blame: Cultural barriers to medical incident reporting. Soc Sci Med 60:1927-1935, 2005[CrossRef][Medline]

13. Wakefield BJ, Blegen MA, Uden-Holman T, et al: Organizational culture, continuous quality improvement, and medication administration error reporting. Am J Med Qual 16:128-134, 2001[Abstract/Free Full Text]

14. Eccles M, Grimshaw J, Campbell M, et al: Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 12:47-52, 2003[Abstract/Free Full Text]

15. Nodit L, Balassanian R, Sudilovsky D, et al: Improving the quality of cytology diagnosis: Root cause analysis for errors in bronchial washing and brushing specimens. Am J Clin Pathol 124:883-893, 2005[CrossRef][Medline]

16. Byrt T, Bishop J, Carlin JB: Bias, prevalence and kappa. J Clin Epidemiol 46:423-429, 1993[CrossRef][Medline]

17. Lantz CA, Nebenzahl E: Behavior and interpretation of the statistic: Resolution of the two paradoxes. J Clin Epidemiol 49:431-434, 1996[CrossRef][Medline]

18. Nelson JC, Pepe MS: Statistical description of interrater variability in ordinal ratings. Stat Methods Med Res 9:475-496, 2000[Abstract/Free Full Text]

19. Llewellyn H: Observer variation, dysplasia grading, and HPV typing: A review: Am J Clin Pathol 114:S21-35, 2000[Medline]

20. Leslie KO, Fechner RE, Kempson RL: Second opinions in surgical pathology. Am J Clin Pathol 106:S58-S64, 1996[Medline]

21. Page DL, Dupont WD, Jensen RA, et al: When and to what end do pathologists agree? J Natl Cancer Inst 90:88-89, 1998[Free Full Text]

22. Scott MA, Lagios MD, Axelsson K, et al: Ductal carcinoma in situ of the breast: Reproducibility of histological subtype analysis. Hum Pathol 28:967-973, 1997[CrossRef][Medline]

23. Wells WA, Carney PA, Eliassen MS, et al: Pathologists' agreement with experts and reproducibility of breast ductal carcinoma-in-situ classification schemes. Am J Surg Pathol 24:651-659, 2000[CrossRef][Medline]

24. Bethwaite P, Smith N, Delahung B, et al: Reproducibility of new classification schemes for the pathology of ductal carcinoma in situ of the breast. J Clin Pathol 51:450-454, 1998[Abstract]

25. Sloane JP, Amendoeira I, Apostolikas N, et al: Consistency achieved by 23 European pathologists in categorizing ductal carcinoma in situ of the breast using five classifications: European Commission Working Group on Breast Screening Pathology. Hum Pathol 29:1056-1062, 1998[Medline]

Submitted September 25, 2005; accepted March 21, 2006.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Facebook    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				A Schattner The unbearable lightness of diagnostic testing: time to contain inappropriate test ordering Postgrad. Med. J., December 1, 2008; 84(998): 618 - 621. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Purchase Article View Shopping Cart Alert me when this article is cited Alert me if a correction is posted Services Email this article to a colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Save to my personal folders Download to citation manager Rights & Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Raab, S. S. Articles by Grzybicki, D. M. Search for Related Content PubMed PubMed Citation Articles by Raab, S. S. Articles by Grzybicki, D. M. Social Bookmarking                            What's this?