OSA's Digital Library

Virtual Journal for Biomedical Optics

Virtual Journal for Biomedical Optics


  • Editor: Gregory W. Faris
  • Vol. 3, Iss. 1 — Jan. 29, 2008

Multireader multicase variance analysis for binary data

Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers  »View Author Affiliations

JOSA A, Vol. 24, Issue 12, pp. B70-B80 (2007)

View Full Text Article

Enhanced HTML    Acrobat PDF (701 KB) Open Access

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



Multireader multicase (MRMC) variance analysis has become widely utilized to analyze observer studies for which the summary measure is the area under the receiver operating characteristic (ROC) curve. We extend MRMC variance analysis to binary data and also to generic study designs in which every reader may not interpret every case. A subset of the fundamental moments central to MRMC variance analysis of the area under the ROC curve (AUC) is found to be required. Through multiple simulation configurations, we compare our unbiased variance estimates to naïve estimates across a range of study designs, average percent correct, and numbers of readers and cases.

© 2007 Optical Society of America

OCIS Codes
(000.5490) General : Probability theory, stochastic processes, and statistics
(110.3000) Imaging systems : Image quality assessment
(330.5510) Vision, color, and visual optics : Psychophysics

Original Manuscript: April 13, 2007
Revised Manuscript: July 7, 2007
Manuscript Accepted: August 2, 2007
Published: September 28, 2007

Virtual Issues
Vol. 3, Iss. 1 Virtual Journal for Biomedical Optics

Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers, "Multireader multicase variance analysis for binary data," J. Opt. Soc. Am. A 24, B70-B80 (2007)

Sort:  Year  |  Journal  |  Reset  


  1. D. D. Dorfman, K. S. Berbaum, and C. E. Metz, "Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method," Invest. Radiol. 27, 723-731 (1992). [CrossRef] [PubMed]
  2. S. V. Beiden, R. F. Wagner, and G. Campbell, "Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis," Acad. Radiol. 7, 341-349 (2000). [CrossRef] [PubMed]
  3. N. A. Obuchowski, S. V. Beiden, K. S. Berbaum, S. L. Hillis, H. Ishwaran, H. H. Song, and R. F. Wagner, "Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods," Acad. Radiol. 11, 980-995 (2004). [CrossRef] [PubMed]
  4. B. D. Gallas, "One-shot estimate of MRMC variance: AUC," Acad. Radiol. 13, 353-362 (2006). [CrossRef] [PubMed]
  5. B. D. Gallas and D. G. Brown, "Reader studies for validation of CAD systems," submitted to Neural Networks.
  6. C. A. Roe and C. E. Metz, "Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation," Acad. Radiol. 4, 298-303 (1997). [CrossRef] [PubMed]
  7. M. Schiffman and M. E. Adrianza, "ASCUS-LSIL triage study: design, methods and characteristics of trial participants," Acta Cytol. 44, 726-742 (2000). [CrossRef] [PubMed]
  8. J. Jeronimo, L. S. Massad, and M. Schiffman, "Visual appearance of the uterine cervix: correlation with human papillomavirus detection and type," Am. J. Obstet. Gynecol. 97, 47.e1-47.e8 (2007). [CrossRef]
  9. S. L. Hillis and K. S. Berbaum, "Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification," Acad. Radiol. 12, 1534-1541 (2005). [CrossRef] [PubMed]
  10. S. L. Hillis, N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum, "A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data," Stat. Med. 24, 1579-1607 (2005). [CrossRef] [PubMed]
  11. X. Song and X.-H. Zhou, "A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data," Biostatistics 6, 303-312 (2005). [CrossRef] [PubMed]
  12. W. A. Yousef, R. F. Wagner, and M. H. Loew, "Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach," IEEE Trans. Pattern Anal. Mach. Intell. 28, 1809-1817 (2006). [CrossRef] [PubMed]
  13. M. S. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction (Oxford U. Press, 2003).
  14. C. A. Roe and C. E. Metz, "Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates," Acad. Radiol. 4, 587-600 (1997). [CrossRef] [PubMed]
  15. H. H. Barrett, M. A. Kupinski, and E. Clarkson, "Probabilistic Foundations of the MRMC Method," Proc. SPIE 5749, 21-31 (2005). [CrossRef]
  16. E. Clarkson, M. A. Kupinski, and H. H. Barrett, "A probabilistic model for the MRMC method. Part 1. theoretical development," Acad. Radiol. 13, 1410-1421 (2006). [CrossRef] [PubMed]

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.


Fig. 1 Fig. 2 Fig. 3
Fig. 4 Fig. 5

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited