OSA's Digital Library

Journal of the Optical Society of America A

Journal of the Optical Society of America A


  • Editor: Franco Gori
  • Vol. 27, Iss. 12 — Dec. 1, 2010
  • pp: 2670–2683

Ideal observer analysis for task normalization of pattern classifier performance applied to EEG and fMRI data

Matthew F. Peterson, Koel Das, Jocelyn L. Sy, Sheng Li, Barry Giesbrecht, Zoe Kourtzi, and Miguel P. Eckstein  »View Author Affiliations

JOSA A, Vol. 27, Issue 12, pp. 2670-2683 (2010)

View Full Text Article

Enhanced HTML    Acrobat PDF (696 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



The application of multivariate techniques to neuroimaging and electrophysiological data has greatly enhanced the ability to detect where, when, and how functional neural information is processed during a variety of behavioral tasks. With the extension to single-trial analysis, neuroscientists are able to relate brain states to perceptual, cognitive, and motor processes. Using pattern classification methods, the neuroscientist can extract neural performance measures in a manner analogous to human behavioral performance, allowing for a consistent information content metric across measurement modalities. However, as with behavioral psychophysical performance, pattern classifier performances are a product of both the task-relevant information inherent in the brain and in the task/stimuli. Here, we argue for the use of an ideal observer framework with which the researcher can effectively normalize the observed neural performance given the task’s inherent objective difficulty. We use data from a face versus car discrimination task and compare classifier performance applied to electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data with corresponding human behavior through the absolute and relative efficiency metrics. We show that confounding variables that can lead to erroneous interpretations of information content can be accounted for through comparisons to an ideal observer, allowing for more confident interpretation of the neural mechanisms involved in the task of interest. Finally, we discuss limitations of interpretation due to the transduction of indirect measures of neural activity, underlying assumptions in the optimality of the pattern classifiers, and dependence of efficiency results on signal contrast.

© 2010 Optical Society of America

OCIS Codes
(330.4060) Vision, color, and visual optics : Vision modeling
(330.4300) Vision, color, and visual optics : Vision system - noninvasive assessment
(330.5510) Vision, color, and visual optics : Psychophysics

ToC Category:
Vision, Color, and Visual Optics

Original Manuscript: April 20, 2010
Revised Manuscript: October 8, 2010
Manuscript Accepted: October 19, 2010
Published: November 24, 2010

Matthew F. Peterson, Koel Das, Jocelyn L. Sy, Sheng Li, Barry Giesbrecht, Zoe Kourtzi, and Miguel P. Eckstein, "Ideal observer analysis for task normalization of pattern classifier performance applied to EEG and fMRI data," J. Opt. Soc. Am. A 27, 2670-2683 (2010)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. Further assumptions are well-documented and not the focus of this paper, such as the exact relation between neuronal firing and the observable measurement provided by the specific imaging modality. For our purposes we will assume (somewhat safely, given the large and growing body of evidence) that the imaging observables represent some direct transformation of underlying neural activity (fMRI: below; EEG: below; MEG: below).
  2. J. Haynes and G. Rees, “Predicting the orientation of invisible stimuli from activity in human primary visual cortex,” Nat. Neurosci. 8, 686–691 (2005). [CrossRef] [PubMed]
  3. H. Barlow, “Single units and sensation: A neuron doctrine for perceptual psychology?” Perception 1, 371–394 (1972). [CrossRef] [PubMed]
  4. K. Britten, M. Shadlen, W. Newsome, and J. Movshon, “The analysis of visual motion: a comparison of neuronal and psychophysical performance,” J. Neurosci. 12, 4745–4765 (1992). [PubMed]
  5. J. Haxby, M. Gobbini, M. Furey, A. Ishai, J. Schouten, and P. Pietrini, “Distributed and overlapping representations of faces and objects in ventral temporal cortex,” Science 293, 2425–2430 (2001). [CrossRef] [PubMed]
  6. With spatially coarse measures such as fMRI and ERP the local interactions are pooled together and lost. However, long-range connections can be monitored between the large groups of neurons represented in each voxel or electrode.
  7. R. Duda, P. Hart, and D. Storke, Pattern Classification (Wiley, 2001).
  8. Y. Kamitani and F. Tong, “Decoding the visual and subjective contents of the human brain,” Nat. Neurosci. 8, 679–685 (2005). [CrossRef] [PubMed]
  9. J. Haynes and G. Rees, “Decoding mental states from brain activity in humans,” Nat. Rev. Neurosci. 7, 523–534 (2006). [CrossRef] [PubMed]
  10. D. Ostwald, J. Lam, S. Li, and Z. Kourtzi, “Neural coding of global form in the human visual cortex,” J. Neurophysiol. 99, 2456–2469 (2008). [CrossRef] [PubMed]
  11. T. Preston, S. Li, Z. Kourtzi, and A. Welchman, “Multivoxel pattern selectivity for perceptually relevant binocular disparities in the human brain,” J. Neurosci. 28, 11315–11327 (2008). [CrossRef] [PubMed]
  12. L. Pessoa and S. Padmala, “Quantitative prediction of perceptual decisions during near-threshold fear detection,” Proc. Natl. Acad. Sci. U.S.A. 102, 5612–5617 (2005). [CrossRef] [PubMed]
  13. M. Philiastides and P. Sajda, “Temporal characterization of the neural correlates of perceptual decision making in the human brain,” Cereb. Cortex 16, 509–518 (2006). [CrossRef]
  14. T. Donner, M. Siegel, R. Oostenveld, P. Fries, M. Bauer, and A. Engel, “Population activity in the human dorsal pathway predicts the accuracy of visual motion detection,” J. Neurophysiol. 98, 345–359 (2007). [CrossRef] [PubMed]
  15. R. Ratcliff, M. Philiastides, and P. Sajda, “Quality of evidence for perceptual decision making is indexed by trial-to-trial variability of the EEG,” Proc. Natl. Acad. Sci. U.S.A. 106, 6539–6544 (2009). [CrossRef] [PubMed]
  16. S. Li, S. Mayhew, and Z. Kourtzi, “Learning shapes the representation of behavioral choice in the human brain,” Neuron 62, 441–452 (2009). [CrossRef] [PubMed]
  17. D. Green and J. Swets, Signal Detection Theory and Psychophysics (Wiley, 1966).
  18. J. Solomon and D. Pelli, “The visual filter mediating letter identification,” Nature (London) 369, 395–397 (1994). [CrossRef]
  19. B. Tjan, W. Braje, G. Legge, and D. Kersten, “Human efficiency for recognizing 3-D objects in luminance noise,” Vision Res. 35, 3053–3068 (1995). [CrossRef] [PubMed]
  20. J. Gold, P. Bennett, and A. Sekuler, “Identification of band-pass filtered letters and faces by human and ideal observers,” Vision Res. 39, 3537–3560 (1999). [CrossRef]
  21. W. Geisler, “Ideal observer analysis,” in The Visual Neurosciences, L.Chalupa and J.Werner, eds. (MIT Press, 2003), pp. 825–837.
  22. J. Gold, D. Tadin, S. Cook, and R. Blake, “The efficiency of biological motion perception,” Percept. Psychophys. 70, 88–95 (2008). [CrossRef] [PubMed]
  23. Z. Shang and T. Sim, “When Fisher meets Fukunaga-Koontz: A new look at linear discriminants,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2006), pp. 323–329.
  24. N. Kanwisher, J. McDermott, and M. Chun, “The fusiform face area: a module in human extrastriate cortex specialized for face perception,” J. Neurosci. 17, 4302–4311 (1997). [PubMed]
  25. W. Peterson, T. Birdsall, and W. Fox, “The theory of signal detectability,” IEEE Trans. Inf. Theory 4, 171–212 (1954). [CrossRef]
  26. R. McDonough and A. Whalen, Detection of Signals in Noise (Academic, 1995).
  27. This amounts to a spatial deconvolution, or, equivalently, multiplying by the inverse of the covariance matrix per Eq. .
  28. H. Barlow, “The absolute efficiency of perceptual decisions,” Philos. Trans. R. Soc. London Ser. B 290, 71–91 (1980). [CrossRef]
  29. Cclassifier is equivalent to Chuman as the human’s behavioral response and brain activation are derived from the same displayed stimulus.
  30. M. Eckstein, B. Beutter, and L. Stone, “Quantifying the performance limits of human saccadic targeting during visual search,” Perception 30, 1389–1401 (2001). [CrossRef]
  31. In addition, there were also procedural differences: stimulus presentation time, display luminance, image size on the retina (4.57° vs. 5.13°). However, given that the task is limited by external noise, it is likely that these procedural differences might result in a smaller performance difference than one might expect from noiseless displays.
  32. W. Geisler, “Sequential ideal-observer analysis of visual discriminations,” Psychol. Rev. 96, 267–314 (1989). [CrossRef] [PubMed]
  33. Y. Zhang, B. Pham, and M. Eckstein, “The effect of nonlinear human visual system components on performance of a channelized Hotelling observer in structured backgrounds,” IEEE Trans. Med. Imaging 25, 1348–1362 (2006). [CrossRef] [PubMed]
  34. B. Tjan and A. Nandy, “Classification images with uncertainty,” J. Vision 6, 387–413 (2006). [CrossRef]
  35. Here, the imaging data can be thought of as coming from two classes, face present and car present. On top of the category-specific activity is the imaging noise which has been shown to be highly Gaussian for fMRI data . Thus, the data themselves can be thought of as a Gaussian noise process with a mean displaced by the category-specific activation.
  36. C. Chen, C. Tyler, and H. Baseler, “Statistical properties of BOLD magnetic resonance activity in the human brain,” Neuroimage 20, 1096–1109 (2003). [CrossRef] [PubMed]
  37. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth, “Occam’s Razor,” Inf. Process. Lett. 24, 377–380 (1987). [CrossRef]
  38. T. Cover and J. Thomas, Elements of Information Theory (Wiley, 1991). [CrossRef]
  39. D. Heeger and D. Ress, “What does fMRI tell us about neuronal activity?” Nat. Rev. Neurosci. 3, 142–151 (2002). [CrossRef] [PubMed]
  40. G. Boynton, S. Engel, G. Glover, and D. Heeger, “Linear systems analysis of functional magnetic resonance imaging in Human V1,” J. Neurosci. 16, 4207–4221 (1996). [PubMed]
  41. S. Zhang and T. Sim, “Discriminant subspace analysis: a Fukunaga-Koontz approach,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 1732–1745 (2007). [CrossRef] [PubMed]
  42. V. Vapnik, Statistical Learning Theory (Wiley, 1998).
  43. K. Das and Z. Nenadic, “An efficient discriminant-based solution for small sample size problem,” Pattern Recogn. Lett. 42, 857–866 (2009).
  44. K. Das and Z. Nenadic, “Approximate information discriminant analysis: a computationally simple heteroscedastic feature extraction technique,” Pattern Recogn. Lett. 41, 1565–1574 (2008).
  45. N. Logothetis, J. Pauls, M. Augath, T. Trinath, and A. Oeltermann, “Neurophysiological investigation of the basis of the fMRI signal,” Nature (London) 412, 150–157 (2001). [CrossRef]
  46. P. Nunez, Electrical Fields of the Brain: the Neurophysics of EEG (Oxford University Press, 1981).
  47. Y. Okada, “Neurogenesis of evoked magnetic fields,” in Biomagnetism: an Interdisciplinary Approach, S.Williamson, ed. (Plenum, 1983), pp. 399–408.

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article

OSA is a member of CrossRef.

CrossCheck Deposited