OSA's Digital Library

Journal of the Optical Society of America A

Journal of the Optical Society of America A


  • Vol. 16, Iss. 7 — Jul. 1, 1999
  • pp: 1587–1601

Probabilistic framework for the adaptation and comparison of image codes

Michael S. Lewicki and Bruno A. Olshausen  »View Author Affiliations

JOSA A, Vol. 16, Issue 7, pp. 1587-1601 (1999)

View Full Text Article

Enhanced HTML    Acrobat PDF (1072 KB)

Browse Journals / Lookup Meetings

Browse by Journal and Year


Lookup Conference Papers

Close Browse Journals / Lookup Meetings

Article Tools



We apply a Bayesian method for inferring an optimal basis to the problem of finding efficient image codes for natural scenes. The basis functions learned by the algorithm are oriented and localized in both space and frequency, bearing a resemblance to two-dimensional Gabor functions, and increasing the number of basis functions results in a greater sampling density in position, orientation, and scale. These properties also resemble the spatial receptive fields of neurons in the primary visual cortex of mammals, suggesting that the receptive-field structure of these neurons can be accounted for by a general efficient coding principle. The probabilistic framework provides a method for comparing the coding efficiency of different bases objectively by calculating their probability given the observed data or by measuring the entropy of the basis function coefficients. The learned bases are shown to have better coding efficiency than traditional Fourier and wavelet bases. This framework also provides a Bayesian solution to the problems of image denoising and filling in of missing pixels. We demonstrate that the results obtained by applying the learned bases to these problems are improved over those obtained with traditional techniques.

© 1999 Optical Society of America

OCIS Codes
(000.5490) General : Probability theory, stochastic processes, and statistics
(100.2960) Image processing : Image analysis
(100.3010) Image processing : Image reconstruction techniques

Original Manuscript: November 5, 1998
Revised Manuscript: March 22, 1999
Manuscript Accepted: March 22, 1999
Published: July 1, 1999

Michael S. Lewicki and Bruno A. Olshausen, "Probabilistic framework for the adaptation and comparison of image codes," J. Opt. Soc. Am. A 16, 1587-1601 (1999)

Sort:  Author  |  Year  |  Journal  |  Reset  


  1. J. G. Daugman, “Uncertainty relation for resolution in space, spatial-frequency, and orientation optimized by two-dimensional visual cortical filters,” J. Opt. Soc. Am. A 2, 1160–1169 (1985). [CrossRef] [PubMed]
  2. J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural networks for image-analysis and compression,” IEEE Trans. Acoust., Speech, Signal Process. 36, 1169–1179 (1988). [CrossRef]
  3. J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive-fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989). [CrossRef] [PubMed]
  4. D. J. Field, “What is the goal of sensory coding,” Neural Comput. 6, 559–601 (1994). [CrossRef]
  5. T. S. Lee, “Image representation using 2D Gabor wavelets,” IEEE Trans. Pattern. Anal. Mach. Intell. 18, 959–971 (1996). [CrossRef]
  6. M. S. Lewicki, T. J. Sejnowski, “Learning overcomplete representations,” Neural Comput. (to be published).
  7. C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991). [CrossRef]
  8. P. Comon, “Independent component analysis, a new concept,” Signal Process. 36, 287–314 (1994). [CrossRef]
  9. A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995). [CrossRef] [PubMed]
  10. B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997). [CrossRef]
  11. E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992). [CrossRef]
  12. S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).
  13. R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992). [CrossRef]
  14. S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993). [CrossRef]
  15. S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997). [CrossRef]
  16. B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996). [CrossRef]
  17. P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992). [CrossRef]
  18. C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995). [CrossRef]
  19. R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997). [CrossRef] [PubMed]
  20. R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998). [CrossRef]
  21. A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997). [CrossRef]
  22. J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998). [CrossRef]
  23. C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.
  24. D. L. Ruderman, “The statistics of natural images,” Network Comput. Neural Syst. 5, 517–548 (1994). [CrossRef]
  25. H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenbluth, ed. (MIT Press, Cambridge, Mass., 1961), pp. 217–234.
  26. H. B. Barlow, “Unsupervised learning,” Neural Comput. 1, 295–311 (1989). [CrossRef]
  27. W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).
  28. S. Marcelja, “Mathematical description of the responses of simple cortical cells,” J. Opt. Soc. Am. 70, 1297–1300 (1980). [CrossRef] [PubMed]
  29. R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982). [CrossRef] [PubMed]
  30. A. J. Parker, M. J. Hawken, “Two-dimensional spatial structure of receptive fields in monkey striate cortex,” J. Opt. Soc. Am. A 5, 598–605 (1988). [CrossRef] [PubMed]
  31. J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998). [CrossRef]
  32. I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math. XLI, 909–996 (1988). [CrossRef]
  33. R. W. Buccigrossi, E. P. Simoncelli, “Image compression via joint statistical characterization in the wavelet domain,” (University of Pennsylvania, Philadelphia, Penn., May1997).
  34. E. P. Simoncelli, E. H. Adelson, “Noise removal via Bayesian wavelet coring,” in Proceedings of International Conference IEEE on Image Processing, III Lausanne, Switzerland (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.
  35. S. Chen, “Basis pursuit,” Ph.D. dissertation (Stanford University, Stanford, Calif., 1995). Available at http://www-stat.stanford.edu/reports/chen.s
  36. R. Everson, L. Sirovich, “Karhunen–Loève procedure for gappy data,” J. Opt. Soc. Am. A 12, 1657–1664 (1995). [CrossRef]
  37. B. A. Pearlmutter, L. C. Parra, “Maximum likelihood blind source separation: a context-sensitive generalization of ICA,” in Advances in Neural and Information Processing SystemsM. C. Mozer, M. I. Jordan, T. Petsche, eds. (Morgan Kaufmann, Los Altos, Calif., 1997), Vol. 9, pp. 613–619.
  38. H. Attias, “Independent factor analysis,” Neural Comput. 11, 803–851 (1998). [CrossRef]
  39. B. D. Rao, K. Kreutz-Delgado, “An affine scaling methodology for best basis selection,” tech. rep. (Center for Information Engineering, University of California, San Diego, San Diego, Calif., 1997).
  40. R. M. Neal, Bayesian Learning for Neural Networks (Springer-Verlag, New York, 1996).
  41. J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994). [CrossRef]
  42. J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994). [CrossRef]
  43. J-F. Cardoso, “Infomax and maximum likelihood for blind source separation,” IEEE Signal Process. Lett. 4, 109–111 (1997).
  44. G. E. Hinton, T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, eds. (MIT Press, Cambridge, Mass., 1986), Vol. 1, Chap. 7, pp. 282–317.
  45. R. Linsker, “Self-organization in a perceptual network,” Computer 21, 105–117 (1988). [CrossRef]
  46. J. J. Atick, “Could information-theory provide an ecological theory of sensory processing,” Network Comput. Neural Syst. 3, 213–251 (1992). [CrossRef]

Cited By

Alert me when this paper is cited

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article  |  Next Article »

OSA is a member of CrossRef.

CrossCheck Deposited