## Prediction of human observer performance by numerical observers: an experimental study

JOSA A, Vol. 16, Issue 3, pp. 679-693 (1999)

http://dx.doi.org/10.1364/JOSAA.16.000679

Acrobat PDF (1736 KB)

### Abstract

Numerical observers are investigated for predicting the outcome of a free-response human observer study involving the detection of simulated pulmonary nodules in images reconstructed from low-dose computed tomography projection data by use of several reconstruction algorithms. A new way of calculating the figure of merit of a numerical observer is proposed wherein the detectability of signals in a particular image depends on the noise properties associated with that image and not the other images in the data set. The resulting variants of numerical observers are found to perform better than their traditional counterparts. In particular, the imagewise variant of the region-of-interest observer is found to predict best the rank ordering of algorithms by human observers for the free-response task.

© 1999 Optical Society of America

**OCIS Codes**

(100.0100) Image processing : Image processing

(100.2960) Image processing : Image analysis

(100.3010) Image processing : Image reconstruction techniques

(100.5010) Image processing : Pattern recognition

(110.4280) Imaging systems : Noise in imaging systems

(330.4060) Vision, color, and visual optics : Vision modeling

(330.5020) Vision, color, and visual optics : Perception psychology

(330.5510) Vision, color, and visual optics : Psychophysics

**Citation**

T. K. Narayan and Gabor T. Herman, "Prediction of human observer performance by numerical observers: an experimental study," J. Opt. Soc. Am. A **16**, 679-693 (1999)

http://www.opticsinfobase.org/josaa/abstract.cfm?URI=josaa-16-3-679

Sort: Year | Journal | Reset

### References

- G. T. Herman, Image Reconstruction from Projections: The Fundamentals of Computerized Tomography (Academic, New York, 1980).
- G. T. Herman and D. Odhner, “Performance evaluation of an iterative image reconstruction algorithm for positron emission tomography,” IEEE Trans. Med. Imaging 10, 336–346 (1991).
- H. H. Barrett, “Objective assessment of image quality: effects of quantum noise and object variability,” J. Opt. Soc. Am. A 7, 1266–1278 (1990).
- T. A. Gooley and H. H. Barrett, “Evaluation of statistical methods of image reconstruction through ROC analysis,” IEEE Trans. Med. Imaging 11, 276–283 (1992).
- K. M. Hanson, “Method of evaluating image-recovery algorithms based on task performance,” J. Opt. Soc. Am. A 7, 1294–1304 (1990).
- G. T. Herman and K. T. D. Yeung, “Evaluators of image reconstruction algorithms,” Int. J. Imag. Syst. Technol. 1, 187–195 (1989).
- M. F. Insana and T. J. Hall, “Methods for estimating the efficiency of human and computational observers in ultrasonography,” in Information Processing in Medical Imaging, H. H. Barrett and A. F. Gmitro, eds. (Springer-Verlag, Berlin, 1993), Vol. 687, pp. 542–552.
- J. A. Swets and R. M. Pickett, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory (Academic, New York, 1982).
- D. P. Chakraborty and L. H. L. Winter, “Free-response methodology: alternate analysis and a new observer-performance experiment,” Radiology 174, 873–881 (1990).
- D. M. Green and J. A. Swets, Signal Detection Theory and Psychophysics (Peninsula, Los Altos, Calif., 1988).
- J. Llacer, E. Veklerov, L. R. Baxter, S. T. Grafton, L. K. Griffeth, R. A. Hawkins, C. K. Hoh, J. C. Mazziotta, E. J. Hoffman, and C. E. Metz, “Results of a clinical receiver operating characteristic study comparing filtered backprojection and maximum likelihood estimator images in FDG PET studies,” J. Nucl. Med. 34, 1198–1203 (1993).
- C. E. Metz, “Some practical issues of experimental design and data analysis in radiological ROC studies,” Invest. Radiol. 24, 234–243 (1989).
- R. G. Swensson, “Measuring detection and localization performance,” in Information Processing in Medical Imaging, H. H. Barrett and A. F. Gmitro, eds. (Springer-Verlag, Berlin, 1993), Vol. 687, pp. 525–541.
- J. A. Swets, “ROC analysis applied to the evaluation of medical imaging techniques,” Invest. Radiol. 4, 109–112 (1979).
- R. D. Fiete, H. H. Barrett, W. E. Smith, and K. J. Myers, “Hotelling trace criterion and its correlation with human-observer performance,” J. Opt. Soc. Am. A 4, 945–953 (1987).
- S. S. Furuie, G. T. Herman, T. K. Narayan, P. Kinahan, J. S. Karp, R. M. Lewitt, and S. Matej, “A methodology for testing for statistically significant differences between fully 3-D PET reconstruction algorithms,” Phys. Med. Biol. 39, 341–354 (1994).
- S. Matej, G. T. Herman, T. K. Narayan, S. S. Furuie, R. M. Lewitt, and P. Kinahan, “Evaluation of task-oriented performance of several fully 3-D PET reconstruction algorithms,” Phys. Med. Biol. 39, 355–367 (1994).
- K. J. Myers, H. H. Barrett, M. C. Borgstrom, D. D. Patton, and D. W. Seeley, “Effect of noise correlation on detectability of disk signals in medical imaging,” J. Opt. Soc. Am. A 2, 1752–1759 (1985).
- K. J. Myers and H. H. Barrett, “Addition of a channel mechanism to the ideal-observer model,” J. Opt. Soc. Am. A 4, 2447–2457 (1987).
- W. E. Smith and H. H. Barrett, “Hotelling trace criterion as a figure of merit for the optimization of imaging systems,” J. Opt. Soc. Am. A 3, 717–723 (1986).
- R. F. Wagner, K. J. Myers, D. G. Brown, M. J. Tapiovaara, and A. E. Burgess, “Higher-order tasks: human vs. machine performance,” in Medical Imaging II, R. H. Schneider and S. J. Dwyer III, eds., Proc. SPIE 1090, 183–194 (1989).
- Medical Imaging—The Assessment of Image Quality, ICRU Tech. Rep. 54 (International Commission on Radiation Units and Measurement, Bethesda, Md., 1996).
- A. E. Burgess, “Statistically defined backgrounds: performance of a modified nonprewhitening observer model,” J. Opt. Soc. Am. A 11, 1237–1242 (1994).
- M. Kendall and J. D. Gibbons, Rank Correlation Methods (Oxford U. Press, New York, 1990).
- C. K. Abbey, H. H. Barrett, and D. W. Wilson, “Observer signal-to-noise ratios for the ML-EM algorithm,” in Medical Imaging, 1996: Image Perception, H. L. Kundel, ed., Proc. SPIE 2712, 47–58 (1996).
- D. P. Chakraborty, “Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data,” Med. Phys. 16, 561–568 (1989).
- P. C. Bunch, J. F. Hamilton, G. K. Sanderson, and A. H. Simmons, “A free-response approach to the measurement and characterization of radiographic-observer performance,” J. Appl. Photogr. Eng. 4, 166–171 (1978).
- C. E. Metz, “ROC methodology in radiologic imaging,” Invest. Radiol. 21, 720–733 (1986).
- E. Pernkopf, Atlas of Topographical and Applied Human Anatomy (Saunders, Philadelphia, Pa., 1964), Vol. 2.
- J. A. Browne, G. T. Herman, and D. Odhner, “SNARK93— a programming system for image reconstruction from projections,” Tech. Rep. MIPG198 (Department of Radiology, University of Pennsylvania, Philadelphia, Pa., 1993).
- G. T. Herman, “Algorithms for computed tomography,” in The Digital Signal Processing Handbook, V. K. Madisetti and D. B. Williams, eds. (CRC Press, Boca Raton, Fla., 1998), Chap. 26, pp. 1–9.
- R. M. Lewitt, “Alternatives to voxels for image representation in iterative reconstruction algorithms,” Phys. Med. Biol. 37, 705–716 (1992).
- S. Matej and R. M. Lewitt, “Practical considerations for 3-D image reconstruction using spherically symmetric volume elements,” IEEE Trans. Med. Imaging 15, 68–78 (1996).
- P. P. B. Eggermont, G. T. Herman, and A. Lent, “Iterative algorithms for larger partitioned systems with applications to image reconstruction,” Linear Algebr. Appl. 40, 37–67 (1981).
- L. A. Shepp and Y. Vardi, “Maximum likelihood reconstruction in positron emission tomography,” IEEE Trans. Med. Imaging 1, 113–122 (1982).
- J. Browne and A. R. De Pierro, “A row-action alternative to the EM algorithm for maximizing likelihoods in emission tomography,” IEEE Trans. Med. Imaging 15, 687–699 (1996).
- K. T. D. Yeung and G. T. Herman, “Objective measures to evaluate the performance of image reconstruction algorithms,” in Medical Imaging III: Image Processing, S. J. Dwyer, R. Jost, and R. H. Schneider, eds., Proc. SPIE 1092, 326–335 (1989).
- H. H. Barrett, J. Yao, J. P. Rolland, and K. J. Myers, “Model observers for assessment of image quality,” Proc. Natl. Acad. Sci. USA 90, 9758–9765 (1993).
- J. Yao and H. H. Barrett, “Predicting human performance by a channelized Hotelling observer model,” in Mathematical Methods in Medical Imaging, D. C. Wilson and J. N. Wilson, eds., Proc. SPIE 1768, 161–168 (1992).
- A. Burgess and X. Li, “Experimental evaluation of observer models for detection of signals in statistically defined backgrounds,” in Information Processing in Medical Imaging, Y. Bizais, C. Barillot, and R. Di Paola, eds. (Kluwer Academic, Dordrecht, The Netherlands, 1995), pp. 89–100.
- C. K. Abbey and H. H. Barrett, “Linear iterative reconstruction algorithms: study of observer performance,” in Information Processing in Medical Imaging, Y. Bizais, C. Barillot, and R. Di Paola, eds. (Kluwer Academic, Dordrecht, The Netherlands, 1995), pp. 65–76.
- B. Efron, The Jackknife, the Bootstrap and Other Resampling Plans (Society For Industrial and Applied Mathematics, Philadelphia, Pa., 1982).
- R. F. Mould, Introduction to Medical Statistics, 2nd ed. (Hilger, Bristol, UK, 1989).
- M. A. King, D. J. de Vries, and E. J. Soares, “Comparison of the channelized Hotelling and human observers for lesion detection in hepatic SPECT imaging,” in Medical Imaging 1997: Image Perception, H. L. Kundel, ed., Proc. SPIE 3036, 14–20 (1997).

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.