Efficient estimation of ideal-observer performance in classification tasks involving high-dimensional complex backgrounds
JOSA A, Vol. 26, Issue 11, pp. B59-B71 (2009)
http://dx.doi.org/10.1364/JOSAA.26.000B59
Acrobat PDF (701 KB)
Abstract
The Bayesian ideal observer is optimal among all observers and sets an absolute upper bound for the performance of any observer in classification tasks [ Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968). ]. Therefore, the ideal observer should be used for objective image quality assessment whenever possible. However, computation of ideal-observer performance is difficult in practice because this observer requires the full description of unknown, statistical properties of high-dimensional, complex data arising in real life problems. Previously, Markov-chain Monte Carlo (MCMC) methods were developed by Kupinski et al. [ J. Opt. Soc. Am. A 20, 430(2003) ] and by Park et al. [ J. Opt. Soc. Am. A 24, B136 (2007) and IEEE Trans. Med. Imaging 28, 657 (2009) ] to estimate the performance of the ideal observer and the channelized ideal observer (CIO), respectively, in classification tasks involving non-Gaussian random backgrounds. However, both algorithms had the disadvantage of long computation times. We propose a fast MCMC for real-time estimation of the likelihood ratio for the CIO. Our simulation results show that our method has the potential to speed up ideal-observer performance in tasks involving complex data when efficient channels are used for the CIO.
© 2009 Optical Society of America
1. INTRODUCTION
2. M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430–438 (2003). [CrossRef]
3. X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535–1543 (2008). [CrossRef] [PubMed]
4. C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E. A. Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393–400. [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
6. B. D. Gallas and H. H. Barrett, “Validating the use of channels to estimate the ideal linear observer,” J. Opt. Soc. Am. A 20, 1725–1738 (2003). [CrossRef]
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
8. J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009). [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
2. M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430–438 (2003). [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
2. BACKGROUND
2A. Binary Classification and Data Channelization
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
8. J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009). [CrossRef]
2B. Bayesian Ideal and Channelized Ideal Observers
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
8. J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009). [CrossRef]
3. METHODS
3A. Likelihood Ratio for the Channelized Ideal Observer
9. S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008). [CrossRef]
3B. Proposal Density and Acceptance Probability for CIO-MCMC
9. S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008). [CrossRef]
3C. Modeling of Posterior or Prior Densities for CIO-MCMC
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
2. M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430–438 (2003). [CrossRef]
13. J. P. Rolland and H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649–658 (1992). [CrossRef] [PubMed]
3D. Consistency Checks for CIO-MCMC
11. H. H. Barrett, C. K. Abbey, and E. Clarkson, “Objective assessment of image quality III: ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520–1535 (1998). [CrossRef]
12. E. Clarkson, “Bounds on the area under the receiver operating characteristic curve for the ideal observer,” J. Opt. Soc. Am. A 19, 1963–1968 (2001). [CrossRef]
4. SIMULATION STUDY
4A. Models for Imaging System, Object, Noise, and Channels
13. J. P. Rolland and H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649–658 (1992). [CrossRef] [PubMed]
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
4B. Proposal, Posterior, and Prior Densities for CIO-MCMC
4C. Observer Performance, Variance Analysis, and Consistency Checks
14. E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410–1421 (2006). [CrossRef] [PubMed]
15. B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006). [CrossRef] [PubMed]
4D. Analytical Expressions for the CIO Likelihood Ratio
5. RESULTS AND DISCUSSION
5A. Performance of CIO-MCMC Method
5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef]
7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed]
5B. Consistency Checks on the CIO-MCMC Method
5C. Other Choices for Posterior and Prior Densities
6. CONCLUSIONS
7. FUTURE WORK
APPENDIX A: Replacing θ with v b in the Likelihood Ratio
APPENDIX B: “PRIOR” CIO-MCMC as Another Posterior CIO-MCMC
ACKNOWLEDGMENTS
1. | H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968). |
2. | M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430–438 (2003). [CrossRef] |
3. | X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535–1543 (2008). [CrossRef] [PubMed] |
4. | C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E. A. Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393–400. [CrossRef] |
5. | S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre–Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136–B150 (2007). [CrossRef] |
6. | B. D. Gallas and H. H. Barrett, “Validating the use of channels to estimate the ideal linear observer,” J. Opt. Soc. Am. A 20, 1725–1738 (2003). [CrossRef] |
7. | S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). [CrossRef] [PubMed] |
8. | J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009). [CrossRef] |
9. | S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008). [CrossRef] |
10. | C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004). |
11. | H. H. Barrett, C. K. Abbey, and E. Clarkson, “Objective assessment of image quality III: ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520–1535 (1998). [CrossRef] |
12. | E. Clarkson, “Bounds on the area under the receiver operating characteristic curve for the ideal observer,” J. Opt. Soc. Am. A 19, 1963–1968 (2001). [CrossRef] |
13. | J. P. Rolland and H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649–658 (1992). [CrossRef] [PubMed] |
14. | E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410–1421 (2006). [CrossRef] [PubMed] |
15. | B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006). [CrossRef] [PubMed] |
OCIS Codes
(110.3000) Imaging systems : Image quality assessment
(330.1880) Vision, color, and visual optics : Detection
(330.6100) Vision, color, and visual optics : Spatial discrimination
History
Original Manuscript: January 27, 2009
Revised Manuscript: August 1, 2009
Manuscript Accepted: August 17, 2009
Published: October 5, 2009
Virtual Issues
Vol. 4, Iss. 13 Virtual Journal for Biomedical Optics
October 8, 2009 Spotlight on Optics
Citation
Subok Park and Eric Clarkson, "Efficient estimation of ideal-observer performance in classification tasks involving high-dimensional complex backgrounds," J. Opt. Soc. Am. A 26, B59-B71 (2009)
http://www.opticsinfobase.org/josaa/abstract.cfm?URI=josaa-26-11-B59
Sort: Year | Journal | Reset
References
- H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968).
- M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430-438 (2003). [CrossRef]
- X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008). [CrossRef] [PubMed]
- C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E.A.Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393-400. [CrossRef]
- S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre-Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136-B150 (2007). [CrossRef]
- B. D. Gallas and H. H. Barrett, “Validating the use of channels to estimate the ideal linear observer,” J. Opt. Soc. Am. A 20, 1725-1738 (2003). [CrossRef]
- S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009). [CrossRef] [PubMed]
- J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009). [CrossRef]
- S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008). [CrossRef]
- C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004).
- H. H. Barrett, C. K. Abbey, and E. Clarkson, “Objective assessment of image quality III: ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520-1535 (1998). [CrossRef]
- E. Clarkson, “Bounds on the area under the receiver operating characteristic curve for the ideal observer,” J. Opt. Soc. Am. A 19, 1963-1968 (2001). [CrossRef]
- J. P. Rolland and H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649-658 (1992). [CrossRef] [PubMed]
- E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410-1421 (2006). [CrossRef] [PubMed]
- B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353-362 (2006). [CrossRef] [PubMed]
Cited By |
Alert me when this paper is cited |
OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.
Figures
Fig. 1 | Fig. 2 | Fig. 3 |
Fig. 4 | Fig. 5 | Fig. 6 |
Fig. 7 | Fig. 8 | Fig. 9 |
Fig. 10 | Fig. 11 | Fig. 12 |
Fig. 13 | Fig. 14 | Fig. 15 |
Fig. 16 | Fig. 17 | |
« Previous Article | Next Article »
OSA is a member of CrossRef.