OSA's Digital Library

Journal of the Optical Society of America A

Journal of the Optical Society of America A


  • Vol. 20, Iss. 7 — Jul. 1, 2003
  • pp: 1434–1448

Hierarchical Bayesian inference in the visual cortex

Tai Sing Lee and David Mumford  »View Author Affiliations

JOSA A, Vol. 20, Issue 7, pp. 1434-1448 (2003)

View Full Text Article

Acrobat PDF (1188 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



Traditional views of visual processing suggest that early visual neurons in areas V1 and V2 are static spatiotemporal filters that extract local features from a visual scene. The extracted information is then channeled through a feedforward chain of modules in successively higher visual areas for further analysis. Recent electrophysiological recordings from early visual neurons in awake behaving monkeys reveal that there are many levels of complexity in the information processing of the early visual cortex, as seen in the long-latency responses of its neurons. These new findings suggest that activity in the early visual cortex is tightly coupled and highly interactive with the rest of the visual system. They lead us to propose a new theoretical setting based on the mathematical framework of hierarchical Bayesian inference for reasoning about the visual system. In this framework, the recurrent feedforward/feedback loops in the cortex serve to integrate top-down contextual priors and bottom-up observations so as to implement concurrent probabilistic inference along the visual hierarchy. We suggest that the algorithms of particle filtering and Bayesian-belief propagation might model these interactive cortical computations. We review some recent neurophysiological evidences that support the plausibility of these ideas.

© 2003 Optical Society of America

OCIS Codes
(330.4060) Vision, color, and visual optics : Vision modeling

Tai Sing Lee and David Mumford, "Hierarchical Bayesian inference in the visual cortex," J. Opt. Soc. Am. A 20, 1434-1448 (2003)

Sort:  Author  |  Year  |  Journal  |  Reset


  1. D. Mumford, “On the computational architecture of the neocortex II,” Biol. Cybern. 66, 241–251 (1992).
  2. D. Mumford, “Pattern theory: a unifying perspective,” in Perception as Bayesian Inference, D. C. Knill and W. Richards, ed. (Cambridge U. Press, Cambridge UK, 1996), pp. 25–62.
  3. U. Grenander, General Pattern Theory (Oxford U. Press, Oxford, UK, 1993).
  4. G. Hinton, P. Dayan, B. Frey, and R. Neal, “The wake-sleep algorithm for unsupervised neural networks,” Science 268, 1158–1161 (1995).
  5. P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel, “The Helmholtz machine,” Neural Comput. 7, 889–904 (1995).
  6. M. S. Lewicki and T. J. Sejnowski, “Bayesian unsupervised learning of higher order structure,” in Advances in Neural Information Processing Systems 9, M. Mozer, M. Jordan, and T. Petsche, eds. (MIT Press, Cambridge, Mass., 1997), pp. 529–535.
  7. R. Rao and D. Ballard, “Dynamic model of visual recognition predicts neural response properties in the visual cortex,” Neural Comput. 9, 721–763 (1997).
  8. C. E. Guo, S. C. Zhu, and Y. N. Wu, “Visual learning by integrating descriptive and generative models,” Int. J. Comput. Vision (to be published).
  9. Z. W. Tu and S. C. Zhu, “Image segmentation by data-driven Markov chain Monte Carlo,” IEEE Trans. Pattern Anal. Mach. Intell. 24, 657–673 (2002).
  10. R. Desimone and J. Duncan, “Neural mechanisms of selective visual attention,” Annu. Rev. Neurosci. 18, 193–222 (1995).
  11. M. Usher and E. Niebur, “Modeling the temporal dynamics of IT neurons in visual search: A mechanism for top-down selective attention,” J. Cognit. Neurosci. 8, 311–327 (1996).
  12. G. Deco and T. S. Lee, “A unified model of spatial and object attention based on inter-cortical biased competition,” Neurocomputing 44–46, 769–774 (2002).
  13. D. Marr, Vision (Freeman, San Francisco, Calif., 1983).
  14. R. VanRullen and S. Thorpe, “Is it a bird? Is it a plane? Ultra-rapid visual categorization of natural and artificial objects,” Perception 30, 655–668 (2001).
  15. D. J. Felleman and D. C. Van Essen, “Distributed hierarchical processing in the primate cerebral cortex,” Cereb. Cortex 1, 1–47 (1991).
  16. G. Carpenter and S. Grossberg, “A massively parallel architecture for a self-organizing neural pattern recognition,” Comput. Vision Graphics Image Process. 37, 54–115 (1987).
  17. J. L. McClelland and D. E. Rumelhart, “An interactive activation model of context effects in letter perception. Part I: an account of basic findings,” Psychol. Rev. 88, 375–407 (1981).
  18. J. Pearl, Probabilistic Reasoning in Intelligent Systems (Morgan Kaufmann, San Mateo, Calif., 1988).
  19. J. Yedidia, W. T. Freeman, and Y. Weiss, “Understanding belief propagation and its generalization,” presented at the International Joint conference on Artificial Intelligence (IJCAI 2001), Seattle, Washington, August 4–10, 2001.
  20. E. Sudderth, A. Ihler, W. Freeman, and A. Willsky, “Nonparametric belief propagation,” MIT Artificial Intelligence (AI) Laboratory Memo No. 20 (MIT AI Laboratory, Cambridge, Mass., 2002).
  21. A. Doucet, N. de Freitas, and N. Gordon, eds., Sequential Monte Carlo Methods in Practice (Springer-Verlag, New York, 2001).
  22. M. Isard, “Pampas: real-valued graphical models for computer vision,” Proc. Comput. Vision Pattern Recog. 2003 (to be published).
  23. M. Isard and A. Blake, “ICONDENSATION: Unifying low-level and high-level tracking in a stochastic framework,” in Lecture Notes in Computer Science 1406, H. Burkhardt, B. Neumann, ed. (Springer-Verlag, Berlin, 1998), pp. 893–908.
  24. A. Blake, B. Bascle, M. Isard, and J. MacCormick, “Statis-tical models of visual shape and motion,” Proc. R. Soc. London Ser. A 356, 1283–1302 (1998).
  25. S. Thrun, D. Fox, W. Burgard, and F. Dellaert, “Robust Monte Carlo localization for mobile robots,” Artif. Intell. 101, 99–141 (2001).
  26. D. C. Knill and W. Richards, ed. Perception as Bayesian Inference (Cambridge U. Press, Cambridge, UK, 1996).
  27. T. S. Lee, “A Bayesian framework for understanding texture segmentation in the primary visual cortex,” Vision Res. 35, 2643–2657 (1995).
  28. H. V. Helmholtz, Handbuch der physiologischen Optik (Voss, Leipzig, Germany 1867).
  29. W. S. Geisler, R. L. Diehl, “Bayesian natural selection and the evolution of perceptual systems,” Philos. Trans. R. Soc. London Ser. B 357, 419–448 (2002).
  30. D. Mumford, “On the computational architecture of the neocortex I,” Biol. Cybern. 65, 135–145 (1991).
  31. E. Adelson and A. Pentland, “The perception of shading and reflectance,” in Perception as Bayesian Inference, D. Knill and W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 409–423.
  32. P. Sinha and E. Adelson, “Recovering reflectance in a world of painted polyhedra,” in Proceedings of the 4th International Conference on Computer Vision (IEEE Computer Society Press, Los Alamitos, Calif., 1993), pp. 156–163.
  33. R. Zemel, “Cortical Belief Networks,” in Computational Models for Neuroscience, R. Hecht-Neilsen and T. McKenna, eds. (Springer-Verlag, New York) (to be published).
  34. C. Eliasmith and C. H. Anderson, Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems (MIT Press, Cambridge, Mass., 2002).
  35. D. W. Arathorn, Map-Seeking Circuits in Visual Cognition: a Computational Mechanism for Biological and Machine Vision (Stanford U. Press, Palo Alto, Calif., 2002).
  36. C. M. Gray, “The temporal correlation hypothesis of visual feature integration: still alive and well,” Neuron 24, 31–47 (1999).
  37. E. Bienenstock, S. Geman, and D. Potter, “Compositionality, MDL priors, and object recognition,” in Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, eds. (MIT Press, Cambridge, Mass., 1997), Vol. 9, pp. 838–844.
  38. C. v. D. Malsburg, “The what and why of binding: the modeler’s perspective,” Neuron 24, 95–104 (1999).
  39. P. J. Sjostrom and S. B. Nelson, “Spike timing, calcium signals and synaptic plasticity,” Curr. Opin. Neurobiol. 12, 305–314 (2002).
  40. D. Mumford, “Commentary on banishing the homunculus by H. Barlow,” in Perception as Bayesian Inference, D. C. Knill and W. Richards, eds. (Cambridge U. Press, Cambridge, UK 1996), pp. 501–504.
  41. T. S. Lee, D. Mumford, R. Romero, and V. A. F. Lamme, “The role of the primary visual cortex in higher level vision,” Vision Res. 38, 2429–2454 (1998).
  42. D. H. Hubel and T. N. Wiesel, “Functional architecture of macaque monkey visual cortex,” Proc. R. Soc. London Ser. B 198, 1–59 (1978).
  43. R. Gattass, A. P. Sousa, and C. G. Gross, “Visuotopic organization and extent of V3 and V4 of the macaque,” J. Neurosci. 8, 1831–1845 (1988).
  44. C. G. Gross, “Visual function of inferotemporal cortex,” in Handbook of Sensory Physiology, L. R. Jung, ed. (Springer-Verlag, Berlin, 1973), Vol. VII, Part 3B, pp. 451–482.
  45. R. L. De Valois and K. K. De Valois, Spatial Vision (Oxford U. Press, New York, 1988).
  46. V. A. F. Lamme, “The neurophysiology of figure-ground segregation in primary visual cortex,” J. Neurosci. 15, 1605–1615 (1995).
  47. K. Zipser, V. A. F. Lamme, and P. H. Schiller, “Contextual modulation in primary visual cortex,” J. Neurosci. 16, 7376–7389 (1996).
  48. T. S. Lee and M. Nguyen, “Dynamics of subjective contour formation in the early visual cortex,” Proc. Natl. Acad. Sci. USA 98, 1907–1911 (2001).
  49. T. S. Lee, C. Yang, R. Romero, and D. Mumford, “Neural activity in early visual cortex reflects behavioral experience and higher order perceptual saliency,” Nat. Neurosci. 5, 589–597 (2002).
  50. Y. Kamitani and S. Shimojo, “Manifestation of scotomas by transcranial magnetic stimulation of human visual cortex,” Nat. Neurosci. 2, 767–771 (1999).
  51. J. M. Hupe, A. C. James, B. R. Payne, S. G. Lomber, P. Girard, and J. Bullier, “Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons,” Nature 394, 784–787 (1998).
  52. H. Super, H. Spekreijse, and V. A. F. Lamme, “Two distinct modes of sensory processing observed in monkey primary visual cortex (V1),” Nat. Neurosci. 4, 304–310 (2001).
  53. P. R. Roelfsema, V. A. F. Lamme, and H. Spekreijse, “Object-based attention in the primary visual cortex of the macaque monkey,” Nature 395, 376–381 (1998).
  54. M. Ito and C. D. Gilbert, “Attention modulates contextual influences in the primary visual cortex of alert monkeys,” Neuron 22, 593–604 (1999).
  55. N. P. Bichot and J. D. Schall, “Effects of similarity and history on neural mechanisms of visual selection,” Nat. Neurosci. 2, 549–554 (1999).
  56. Y. Sugase, S. Yamane, S. Ueno, and K. Kawano, “Global and fine information coded by single neurons in the temporal visual cortex,” Nature 400, 869–873 (1999).
  57. S. Kosslyn, W. L. Thompson, I. J. Kim, and N. M. Alpert, “Topographical representations of mental images in primary visual cortex,” Nature 378, 496–498 (1995).
  58. B. C. Motter, “Focal attention produces spatially selective processing in visual cortical areas V1, V2, V4 in the presence of competing stimuli,” J. Neurophysiol. 70, 909–919 (1993).
  59. S. Hochstein and M. Ahissar, “View from the top: hierarchies and reverse hierarchies in the visual system,” Neuron 36, 791–804 (2002).
  60. M. K. Kapadia, G. Westheimer, and C. D. Gilbert, “Spatial distribution of contextual interactions in primary visual cortex and in visual perception,” J. Neurophysiol. 84, 2048–2062 (2000).
  61. J. August and S. W. Zucker, “The curve indicator random field: curve organization via edge correlation,” in Perceptual Organization for Artificial Vision Systems, K. Boyer and S. Sarka, eds. (Kluwer Academic, Boston, Mass., 2000), pp. 265–288.
  62. L. Williams and D. Jacobs, “Stochastic completion fields: a neural model of illusory contour shape and saliency,” Neural Comput. 9, 837–858 (1997).
  63. J. Braun, “On the detection of salient contours,” Spatial Vision 12, 211–225 (1999).
  64. Z. Li, “A neural model of contour integration,” Neural Comput. 10, 903–940 (2001).
  65. R. T. Born and R. B. H. Tootell, “Single-unit and 2-deoxyglucose studies of side inhibition in macaque striate cortex,” Proc. Natl. Acad. Sci. USA 88, 7071–7075 (1991).
  66. R. von der Heydt, E. Peterhans, and G. Baumgarthner, “Illusory contours and cortical neuron responses,” Science 224, 1260–1262 (1984).
  67. V. S. Ramachandran, “Perception of shape from shading,” Nature 331, 163–166 (1988).
  68. J. J. Knierim and D. C. Van Essen, “Neuronal responses to static texture patterns in area V1 of the alert macaque monkey,” J. Neurophysiol. 67, 961–980 (1992).
  69. F. T. Qiu, R. Endo, and R. von der Heydt, “Selectivity for structural depth in neurons of monkey area V2,” presented at the 30th Annual Meeting of the Society of Neuroscience, New Orleans, Louisiana November 4–9, 2000.
  70. S. Ullman, “Visual routines,” Cognition 18, 97–159 (1984).
  71. N. K. Logothetis, “Object vision and visual awareness,” Curr. Opin. Neurobiol. 8, 536–544 (1998).
  72. S. Murray, D. Kersten, B. Olshausen, P. Schrater, and D. Woods, “Shape perception reduces activity in human primary visual cortex,” Proc. Natl. Acad. Sci. USA 99, 15164–19169 (2002).
  73. B. M. Ramsden, C. P. Hung, and A. W. Roe, “Real and illusory contour processing in area V1 of the primate: a cortical balancing act,” Cereb. Cortex 11, 648–665 (2001).
  74. C. Koch and T. Poggio, “Predicting the visual world: silence is golden,” Nat. Neurosci. 2, 9–10 (1999).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article

OSA is a member of CrossRef.

CrossCheck Deposited