## Hybrid-resolution multispectral imaging using color filter array |

Optics Express, Vol. 20, Issue 7, pp. 7173-7183 (2012)

http://dx.doi.org/10.1364/OE.20.007173

Acrobat PDF (3868 KB)

### Abstract

Hybrid-resolution multispectral imaging is a framework to acquire multispectral images through a reconstruction procedure using two types of measurement data with different spatial and spectral resolutions. In this paper, we propose a new method for such a framework on the basis of a full-resolution RGB image and the data obtained from an image sensor with a multispectral filter array (MSFA). In the proposed method, a small region of each image band is reconstructed as a linear combination of RGB images, where the weighting coefficients are determined using MSFA data. The effectiveness of the proposed approach is shown by simulations using spectral images of natural scenes.

© 2012 OSA

## 1. Introduction

1. M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. **16**(10), 2352–2362 (1999). [CrossRef]

5. M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. **52**(1), 010201 (2008). [CrossRef]

6. L. Gao, R. T. Kester, N. Hagen, and T. S. Tkaczyk, “Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express **18**(14), 14330–14344 (2010). [CrossRef] [PubMed]

7. R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE **7875**, 787503, 787503-9 (2011). [CrossRef]

15. R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in *Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),* (Institute of Electrical and Electronics Engineers, 2011), 2329–2336.

## 2. Method

*K*-band multispectral image from a set of RGB images and the data of an MSFA with narrow-band filters of

*K*colors (Fig. 1 ). These data can be obtained in one shot, if a camera is configured with four image sensors (three sensors for RGB and one for an MSFA) accompanied with a four-channel separation prism, as shown in Fig. 2 . The data acquisition model and reconstruction method for multispectral images are presented below.

### 2.1 Data acquisition model

**r**

*be an*

_{i}*L*-dimensional column vector representing the spectral reflectance function of the original spectral reflectance image at pixel

*i*, where

*L*is the number of spectral samples and

*N*the number of pixels. The original spectral reflectance image is represented by an

*K*-band multispectral image is defined. The

*K*-band multispectral data corresponding to

**r**

*is represented by a*

_{i}*K*-dimensional column vector,where

**H**

*is an*

_{MS}*k*th

*N*-dimensional column vector of

**F**, another representation of

**F**iswhere

*k*th image band of the multispectral image. Note that the

*K*-band multispectral image defined by Eqs. (2)-(5) is not measurement data of the proposed method, but for an ideal one.

*k*th band color filters iswhere

**c**

*is an*

_{k}*M*-dimensional column vector,

_{k}**S**

*is an*

_{k}*k*th band color filters, and

*M*is the number of pixels assigned to

_{k}*k*th band color filters.

**G**is a

**g**is a one-dimensional representation of a channel of the RGB image, and

**H**

*is an*

_{RGB}**c**

_{1},…,

**c**

*} and*

_{K}**G**, we want to reconstruct the ideal

*K*-band image

**F**.

### 2.2 Localized low-dimensional model for multispectral images

*j*, in which any spectrum can be approximately represented by a linear combination of three

*L*-dimensional spectral basis vectors

*b*is a weighting coefficient. The representation for the whole

_{iq}*j*th region by Eq. (8) is written aswhere []

*is an operator to select the*

_{j}*j*th region from a whole image,

**B**is an

*b*, and

_{ij}**UH**

*is singular,*

_{RGB}**A**

*= (*

_{j}**U**

_{j}**H**

*)*

_{RGB}^{−1}

**U**

_{j}**H**

*. By taking the*

_{MS}*k*th column vectors from [

**F**]

*and*

_{j}**A**

*, we havewhere*

_{j}**a**

*is the*

_{kj}*k*th column vector of

**A**

*.*

_{j}**UH**

*.*

_{RGB}### 2.3 Multispectral image reconstruction method

**a**

*is derived for each region and each band.*

_{kj}*m*is the number of the pixels assigned to the

_{kj}*k*th color filter included in this region, [

**c**

*]*

_{k}*is an m*

_{j}*-dimensional column vector. By substituting Eq. (17) and setting*

_{kj}**c**

*]*

_{k}*and*

_{j}**D**

*as measurement data, the weighting coefficients*

_{kj}**a**

*can be derived by minimizing the error between the sides of Eq. (19). If a simple linear regression analysis is applied, we have*

_{kj}*k*th image band of this region is estimated as

*m*becomes smaller), the regression to derive Eq. (20) becomes unstable. Therefore, these opposite influences should be considered in deciding the size of the regions. Regardless of the size, there are several ways to divide images into regions. A simple method is that an image be divided horizontally and vertically into small blocks. Because neighboring pixels in multispectral images are more likely to have similar spectral characteristics, simple block division can be effective. Another method is that the pixels with similar RGB values be collected into a single region.

_{kj}## 3. Simulation results

16. D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. **21**(3), 331–336 (2004). [CrossRef] [PubMed]

17. W. K. Pratt and C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. **15**(1), 73–75 (1976). [CrossRef] [PubMed]

**a**

*. Except for this kind of error, there is no perceived error in the reconstruction of images by the proposed RGB + 16FA.*

_{jk}**V**= (

**v**

_{1},…,

**v**

_{16}), can be derived as the eigenvectors of

**FF**

*. These are sixteen 16-dimensional vectors ordered by the magnitudes of the corresponding eigenvalues. In a similar way, the principal components of the 16-band images reconstructed by RGB + 16FA, RG1G2B, and 4FA × 4 were derived; i.e.,*

^{T}**V**

_{RGB + 16FA},

**V**

_{RG1G2B}, and

**V**

_{4FA × 4}, respectively. Figure 9 shows

**V**

^{T}**V**

_{RGB + 16FA},

**V**

^{T}**V**

_{RG1G2B}, and

**V**

^{T}**V**

_{4FAx4}in matrix form, where the range of 0–1 is shown in 8-bit grayscale. Because of the orthonormality of principal components, the matrix becomes an identity matrix when the principal components derived from the reconstructed 16-band image are equivalent to

**V**.

## 4. Conclusions

## Acknowledgments

## References and links

1. | M. Hauta-Kasari, K. Miyazawa, S. Toyooka, and J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. |

2. | B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE |

3. | H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. |

4. | J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. |

5. | M. Yamaguchi, H. Hideaki, and N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. |

6. | L. Gao, R. T. Kester, N. Hagen, and T. S. Tkaczyk, “Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express |

7. | R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE |

8. | J. Brauers and T. Aach, “A color filter array based multispectral camera,” presented at the 12. Workshop Farbbildverarbeitung, Ilmenau, Germany, 5–6 Oct. 2006. |

9. | Y. Monno, M. Tanaka, and O. Masatoshi, “Multispectral demosaicking using adaptive kernel upsampling,” in |

10. | F. H. Imai and R. S. Berns, “High-resolution multi-spectral image archives: A hybrid approach,” in |

11. | Y. Murakami, K. Ietomi, M. Yamaguchi, and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. |

12. | Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. |

13. | O. Kohonen, “Multiresolution-based pansharpening in spectral color images,” in |

14. | Y. Murakami, M. Yamaguchi, and N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. |

15. | R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in |

16. | D. H. Foster, S. M. C. Nascimento, and K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. |

17. | W. K. Pratt and C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. |

**OCIS Codes**

(100.3010) Image processing : Image reconstruction techniques

(100.3190) Image processing : Inverse problems

(300.6550) Spectroscopy : Spectroscopy, visible

(110.1758) Imaging systems : Computational imaging

(110.4234) Imaging systems : Multispectral and hyperspectral imaging

**ToC Category:**

Image Processing

**History**

Original Manuscript: January 3, 2012

Revised Manuscript: February 22, 2012

Manuscript Accepted: March 6, 2012

Published: March 14, 2012

**Citation**

Yuri Murakami, Masahiro Yamaguchi, and Nagaaki Ohyama, "Hybrid-resolution multispectral imaging using color filter array," Opt. Express **20**, 7173-7183 (2012)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-20-7-7173

Sort: Year | Journal | Reset

### References

- M. Hauta-Kasari, K. Miyazawa, S. Toyooka, J. Parkkinen, “Spectral vision system for measuring color images,” J. Opt. Soc. Am. 16(10), 2352–2362 (1999). [CrossRef]
- B. Hill, “Color capture, color management and the problem of metamerism,” Proc. SPIE 3963, 3–14 (2000).
- H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39(35), 6621–6632 (2000). [CrossRef] [PubMed]
- J. Y. Hardeberg, F. Schmitt, H. Brettel, “Multispectral color image capture using a liquid crystal tunable filter,” Opt. Eng. 41(10), 2532–2548 (2002). [CrossRef]
- M. Yamaguchi, H. Hideaki, N. Ohyama, “Beyond red-green-blue (RGB): spectrum-based color imaging technology,” J. Imaging Sci. Technol. 52(1), 010201 (2008). [CrossRef]
- L. Gao, R. T. Kester, N. Hagen, T. S. Tkaczyk, “Snapshot image mapping spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express 18(14), 14330–14344 (2010). [CrossRef] [PubMed]
- R. Shrestha, J. Y. Hardeberg, R. Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE 7875, 787503, 787503-9 (2011). [CrossRef]
- J. Brauers and T. Aach, “A color filter array based multispectral camera,” presented at the 12. Workshop Farbbildverarbeitung, Ilmenau, Germany, 5–6 Oct. 2006.
- Y. Monno, M. Tanaka, and O. Masatoshi, “Multispectral demosaicking using adaptive kernel upsampling,” in Proceedings of 18th IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, 2011), 3218–3221.
- F. H. Imai and R. S. Berns, “High-resolution multi-spectral image archives: A hybrid approach,” in Proceedings of 6th Color Imaging Conference, (Society for Imaging Science and Technology, 1998), 224–227.
- Y. Murakami, K. Ietomi, M. Yamaguchi, N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Appl. Opt. 46(28), 7068–7082 (2007). [CrossRef] [PubMed]
- Y. Murakami, M. Yamaguchi, N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Appl. Opt. 48(11), 2188–2202 (2009). [CrossRef] [PubMed]
- O. Kohonen, “Multiresolution-based pansharpening in spectral color images,” in Proceedings of 5th European Conference on Colour in Graphics, Imaging, and Vision, (Society for Imaging Science and Technology, 2010), 535–540.
- Y. Murakami, M. Yamaguchi, N. Ohyama, “Class-based spectral reconstruction based on unmixing of low-resolution spectral information,” J. Opt. Soc. Am. A. 28(7), 1470–1481 (2011). [CrossRef]
- R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Institute of Electrical and Electronics Engineers, 2011), 2329–2336.
- D. H. Foster, S. M. C. Nascimento, K. Amano, “Information limits on neural identification of colored surfaces in natural scenes,” Vis. Neurosci. 21(3), 331–336 (2004). [CrossRef] [PubMed]
- W. K. Pratt, C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. 15(1), 73–75 (1976). [CrossRef] [PubMed]

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.