OSA's Digital Library

Journal of the Optical Society of America A

Journal of the Optical Society of America A

| OPTICS, IMAGE SCIENCE, AND VISION

  • Editor: Stephen A. Burns
  • Vol. 24, Iss. 12 — Dec. 1, 2007
  • pp: B70–B80

Multireader multicase variance analysis for binary data

Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers  »View Author Affiliations


JOSA A, Vol. 24, Issue 12, pp. B70-B80 (2007)
http://dx.doi.org/10.1364/JOSAA.24.000B70


View Full Text Article

Enhanced HTML    Acrobat PDF (701 KB) Open Access





Browse Journals / Lookup Meetings

Browse by Journal and Year


   


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools

Share
Citations

Abstract

Multireader multicase (MRMC) variance analysis has become widely utilized to analyze observer studies for which the summary measure is the area under the receiver operating characteristic (ROC) curve. We extend MRMC variance analysis to binary data and also to generic study designs in which every reader may not interpret every case. A subset of the fundamental moments central to MRMC variance analysis of the area under the ROC curve (AUC) is found to be required. Through multiple simulation configurations, we compare our unbiased variance estimates to naïve estimates across a range of study designs, average percent correct, and numbers of readers and cases.

© 2007 Optical Society of America

OCIS Codes
(000.5490) General : Probability theory, stochastic processes, and statistics
(110.3000) Imaging systems : Image quality assessment
(330.5510) Vision, color, and visual optics : Psychophysics

History
Original Manuscript: April 13, 2007
Revised Manuscript: July 7, 2007
Manuscript Accepted: August 2, 2007
Published: September 28, 2007

Virtual Issues
Vol. 3, Iss. 1 Virtual Journal for Biomedical Optics

Citation
Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers, "Multireader multicase variance analysis for binary data," J. Opt. Soc. Am. A 24, B70-B80 (2007)
http://www.opticsinfobase.org/josaa/abstract.cfm?URI=josaa-24-12-B70


Sort:  Year  |  Journal  |  Reset  

References

  1. D. D. Dorfman, K. S. Berbaum, and C. E. Metz, "Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method," Invest. Radiol. 27, 723-731 (1992). [CrossRef] [PubMed]
  2. S. V. Beiden, R. F. Wagner, and G. Campbell, "Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis," Acad. Radiol. 7, 341-349 (2000). [CrossRef] [PubMed]
  3. N. A. Obuchowski, S. V. Beiden, K. S. Berbaum, S. L. Hillis, H. Ishwaran, H. H. Song, and R. F. Wagner, "Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods," Acad. Radiol. 11, 980-995 (2004). [CrossRef] [PubMed]
  4. B. D. Gallas, "One-shot estimate of MRMC variance: AUC," Acad. Radiol. 13, 353-362 (2006). [CrossRef] [PubMed]
  5. B. D. Gallas and D. G. Brown, "Reader studies for validation of CAD systems," submitted to Neural Networks.
  6. C. A. Roe and C. E. Metz, "Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation," Acad. Radiol. 4, 298-303 (1997). [CrossRef] [PubMed]
  7. M. Schiffman and M. E. Adrianza, "ASCUS-LSIL triage study: design, methods and characteristics of trial participants," Acta Cytol. 44, 726-742 (2000). [CrossRef] [PubMed]
  8. J. Jeronimo, L. S. Massad, and M. Schiffman, "Visual appearance of the uterine cervix: correlation with human papillomavirus detection and type," Am. J. Obstet. Gynecol. 97, 47.e1-47.e8 (2007). [CrossRef]
  9. S. L. Hillis and K. S. Berbaum, "Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification," Acad. Radiol. 12, 1534-1541 (2005). [CrossRef] [PubMed]
  10. S. L. Hillis, N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum, "A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data," Stat. Med. 24, 1579-1607 (2005). [CrossRef] [PubMed]
  11. X. Song and X.-H. Zhou, "A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data," Biostatistics 6, 303-312 (2005). [CrossRef] [PubMed]
  12. W. A. Yousef, R. F. Wagner, and M. H. Loew, "Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach," IEEE Trans. Pattern Anal. Mach. Intell. 28, 1809-1817 (2006). [CrossRef] [PubMed]
  13. M. S. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction (Oxford U. Press, 2003).
  14. C. A. Roe and C. E. Metz, "Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates," Acad. Radiol. 4, 587-600 (1997). [CrossRef] [PubMed]
  15. H. H. Barrett, M. A. Kupinski, and E. Clarkson, "Probabilistic Foundations of the MRMC Method," Proc. SPIE 5749, 21-31 (2005). [CrossRef]
  16. E. Clarkson, M. A. Kupinski, and H. H. Barrett, "A probabilistic model for the MRMC method. Part 1. theoretical development," Acad. Radiol. 13, 1410-1421 (2006). [CrossRef] [PubMed]

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

Figures

Fig. 1 Fig. 2 Fig. 3
 
Fig. 4 Fig. 5
 

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited