OSA's Digital Library

Journal of the Optical Society of America A

Journal of the Optical Society of America A


  • Vol. 20, Iss. 7 — Jul. 1, 2003
  • pp: 1391–1397

Bayesian integration of visual and auditory signals for spatial localization

Peter W. Battaglia, Robert A. Jacobs, and Richard N. Aslin  »View Author Affiliations

JOSA A, Vol. 20, Issue 7, pp. 1391-1397 (2003)

View Full Text Article

Enhanced HTML    Acrobat PDF (151 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



Human observers localize events in the world by using sensory signals from multiple modalities. We evaluated two theories of spatial localization that predict how visual and auditory information are weighted when these signals specify different locations in space. According to one theory (visual capture), the signal that is typically most reliable dominates in a winner-take-all competition, whereas the other theory (maximum-likelihood estimation) proposes that perceptual judgments are based on a weighted average of the sensory signals in proportion to each signal’s relative reliability. Our results indicate that both theories are partially correct, in that relative signal reliability significantly altered judgments of spatial location, but these judgments were also characterized by an overall bias to rely on visual over auditory information. These results have important implications for the development of cue integration and for neural plasticity in the adult brain that enables humans to optimally integrate multimodal information.

© 2003 Optical Society of America

OCIS Codes
(330.0330) Vision, color, and visual optics : Vision, color, and visual optics
(330.1400) Vision, color, and visual optics : Vision - binocular and stereopsis
(330.4060) Vision, color, and visual optics : Vision modeling
(330.7320) Vision, color, and visual optics : Vision adaptation

Original Manuscript: September 11, 2002
Revised Manuscript: January 21, 2003
Manuscript Accepted: January 21, 2003
Published: July 1, 2003

Peter W. Battaglia, Robert A. Jacobs, and Richard N. Aslin, "Bayesian integration of visual and auditory signals for spatial localization," J. Opt. Soc. Am. A 20, 1391-1397 (2003)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. B. E. Stein, M. A. Meredith, The Merging of the Senses (MIT Press, Cambridge, Mass, 1993).
  2. R. Held, “Shifts in binaural localization after prolonged exposure to atypical combinations of stimuli,” Am. J. Psychol. 68, 526–548 (1955). [CrossRef] [PubMed]
  3. H. L. Pick, D. H. Warren, J. C. Hay, “Sensory conflict in judgements of spatial direction,” Percept. Psychophys 6, 203–205 (1969). [CrossRef]
  4. R. B. Welch, D. H. Warren, “Immediate perceptual response to intersensory discrepancy,” Psychol. Bull. 88, 638–667 (1980). [CrossRef] [PubMed]
  5. E. I. Knudsen, M. S. Brainard, “Creating a unified representation of visual and auditory space in the brain,” Annu. Rev. Neurosci. 18, 19–43 (1995). [CrossRef] [PubMed]
  6. M. S. Brainard, E. I. Knudsen, “Sensitive periods for visual calibration of the auditory space map in the barn owl optic tectum,” J. Neurosci. 18, 3929–3942 (1998). [PubMed]
  7. J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Norwell, Mass, 1990).
  8. Z. Ghahramani, Computation and Psychophysics of Sensorimotor Integration, Ph.D. dissertation (Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Mass., 1995).
  9. R. A. Jacobs, “Optimal integration of texture and motion cues to depth,” Vision Res. 39, 3621–3629 (1999). [CrossRef]
  10. M. O. Ernst, M. S. Banks, “Humans integrate visual and haptic information in a statistically optimal fashion,” Nature 415, 429–433 (2002). [CrossRef] [PubMed]
  11. D. C. Knill, J. Saunders, “Humans optimally weight ste-reo and texture cues to estimate surface slant,” J. Vision 2, 400 (2002).
  12. R. E. Kalman, R. S. Bucy, “New results in linear filtering and prediction problems,” J. Basic Eng. Ser. D 83, 95–108 (1961). [CrossRef]
  13. G. C. Goodwin, K. S. Sin, Adaptive Filtering Prediction and Control (Prentice-Hall, Englewood Cliffs, N.J., 1984).
  14. A. L. Yuille, H. H. Bülthoff, “Bayesian decision theory and psychophysics,” in Perception as Bayesian Inference, D. C. Knill, W. Richards, eds. (Cambridge U. Press, Cambridge, UK, 1996), pp. 123–161.
  15. F. Wightman, Department of Psychology, University of Wisconsin, Madison, Wisconsin (personal communication,2000).
  16. J. S. Tittle, V. J. Perotti, J. F. Norman, “Integration of binocular stereopsis and structure from motion in the discrimination of noisy surfaces,” J. Exp. Psychol. Hum. Percept. Perform. 23, 1035–1049 (1997). [CrossRef] [PubMed]
  17. S. Deneve, P. E. Latham, A. Pouget, “Reading population codes: a neural implementation of ideal observers,” Nat. Neurosci. 2, 740–745 (1999). [CrossRef] [PubMed]
  18. A. Pouget, R. S. Zemel, P. Dayan, “Information processing with population codes,” Nature Rev. Neurosci. 1, 125–132 (2000).
  19. S. Deneve, P. E. Latham, A. Pouget, “Efficient computation and cue integration with noisy population codes,” Nature Neurosci. 4, 826–831 (2001). [CrossRef] [PubMed]
  20. S. Wu, S. Amari, “Neural implementation of Bayesian inference in population codes,” in Advances in Neural Information Processing Systems 14, T. G. Dietterich, S. Becker, Z. Ghahramani, eds. (MIT Press, Cambridge, Mass., 2002).
  21. L. Shams, Y. Kamitani, S. Shimojo, “What you see is what you hear,” Nature 408, 788 (2000). [CrossRef]
  22. S. Shimojo, L. Shams, “Sensory modalities are not separate modalities: plasticity and interactions,” Curr. Opin. Neurobiol. 11, 505–509 (2001). [CrossRef] [PubMed]
  23. H. S. Smallman, I. Fine, D. I. A. MacLeod, “Pre- and post-operative characterization of visual function before and after the removal of congenital bilateral cataracts,” Vision Res. 42, 191–210 (2002). [CrossRef]
  24. A. Gelman, J. B. Carlin, H. S. Stern, D. B. Rubin, Bayesian Data Analysis (Chapman & Hall, London, 1995).

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited