## Bispectral coding: compressive and high-quality acquisition of fluorescence and reflectance |

Optics Express, Vol. 22, Issue 2, pp. 1697-1712 (2014)

http://dx.doi.org/10.1364/OE.22.001697

Acrobat PDF (1258 KB)

### Abstract

Fluorescence widely coexists with reflectance in the real world, and an accurate representation of these two components in a scene is vitally important. Despite the rich knowledge of fluorescence mechanisms and behaviors, traditional fluorescence imaging approaches are quite limited in efficiency and quality. To address these two shortcomings, we propose a bispectral coding scheme to capture fluorescence and reflectance: multiplexing code is applied to excitation spectrums to raise the signal-to-noise ratio, and compressive sampling code is applied to emission spectrums for high efficiency. For computational reconstruction from the sparse coded measurements, the redundancy in both components promises recovery from sparse measurements, and the difference between their redundancies promises accurate separation. Mathematically, we cast the reconstruction as a joint optimization, whose solution can be derived by the Augmented Lagrange Method. In our experiment, results on both synthetic data and real data captured by our prototype validate the proposed approach, and we also demonstrate its advantages in two computer vision tasks—photorealistic relighting and segmentation.

© 2014 Optical Society of America

## 1. Introduction

1. R. Donaldson, “Spectrophotometry of fluorescent pigments,” Br. J. Appl. Phys. **5**(6), 210–214 (1954). [CrossRef]

2. I. Sato and C. Zhang, “Image-based separation of reflective and fluorescent components using illumination variant and invariant color,” IEEE Trans. Pattern Anal. **35**(12), 2866–2877 (2013). [CrossRef]

4. A. Springsteen, “Introduction to measurement of color of fluorescent materials,” Anal. Chim. Acta **380**(2), 183–192 (1999). [CrossRef]

*l*

_{1}norm of the reflective component. Mathematically, we resort to convex optimization for the solution.

- Explore the redundancy in reflectance and fluorescence and propose an efficient acquisition and computational reconstruction approach for both components.
- Formulate the reconstruction of two components from sparse multiplexed measurements as joint optimization, which is solved as derived in later sections.
- Build a setup for effective capturing of reflectance and fluorescence in real scenes at high spectral resolution.

## 2. Related work

### 2.1. Fluorescence

### 2.2. Compressive spectrum imaging

16. M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual disperser architecture,” Opt. Express **15**(21), 14013–14027 (2007). [CrossRef] [PubMed]

17. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. **47**(10), B44–B51 (2008). [CrossRef] [PubMed]

18. R. Horisaki, X. Xiao, J. Tanida, and B. Javidi, “Feasibility study for compressive multi-dimensional integral imaging,” Opt. Express **21**(4), 4263–4279 (2013). [CrossRef] [PubMed]

19. R. Horisaki and J. Tanida, “Multi-channel data acquisition using multiplexed imaging with spatial encoding,” Opt. Express **18**(22), 23041–23053 (2010). [CrossRef] [PubMed]

16. M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual disperser architecture,” Opt. Express **15**(21), 14013–14027 (2007). [CrossRef] [PubMed]

17. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. **47**(10), B44–B51 (2008). [CrossRef] [PubMed]

20. Y. Wu, I. O. Mirza, G. R. Arce, and D. W. Prather, “Development of a digital-micromirror-device-based multishot snapshot spectral imaging system,” Opt. Lett. **36**(14), 2692–2694 (2011). [CrossRef] [PubMed]

21. Y. August and A. Stern, “Compressive sensing spectrometry based on liquid crystal devices,” Opt. Lett. **38**(23), 4996–4999 (2013). [CrossRef] [PubMed]

### 2.3. Multiplexing capturing

## 3. Formulation

### 3.1. Derivation of the optimization for a single pixel

*m*and

*n*levels, respectively, for each scene point, we can represent its high-spectrum fluorescence as matrix

**F̂**

_{m}_{×}

*and reflectance as*

_{n}**R̂**

_{m}_{×}

*. Suppose we shed illumination combinations and capture accumulated responses*

_{n}**M̂**

_{m}_{×}

*, which are composed of three components, with*

_{n}**N̂**being the noise, which we assume follows Gaussian white noise

*N*(

*μ*,

*σ*

^{2}).

**F̂**is a low-rank coefficient matrix whose rank is 1 if the Kasha–Vavilov rule is strictly followed.

*p*coded illuminations and record

*q*narrowbands at CCD; the illumination code and recording code are respectively

**I**

_{p}_{×}

*and*

_{m}**O**

_{n}_{×}

*. Accordingly, the reconstruction can be performed by the following optimization: Here || · ||*

_{q}_{*}is the nuclear norm for rank minimization; || · ||

_{l1}is the

*ℓ*

_{1}norm, which has been widely used to force the sparsity of

**R̂**;

*α*is a weighting factor introduced to balance energy terms describing two priors;

**Ĉ**(

*i*,

*j*) is the measurement from the

*i*th illumination pattern in

**Î**and the

*j*th recording wavelength in

**Ô**; and

*π*

**: ℝ**

_{Ω}

^{p}^{×}

*→ ℝ*

^{q}

^{p}^{×}

*is a linear operator that subsamples the entries out of all*

^{q}*p*×

*q*possible measurements by performing dot product with a binary matrix

**Ω̂**. As for

**N̂**, the three-sigma rule is used to impose the noise constraints.

### 3.2. From single pixel to image lattice

*w*different pixels horizontally as follows: which is further simplified as Here the size of the matrices

**F**,

**R**,

**N**,

**C**is

*m*× (

*n*×

*w*) with

*w*being the number of pixels,

**I**is equivalent to

**Î**, and the coding matrix

**O**turns into a diagonal replication of the

*n*×

*q*coding matrix

**Ô**in Eq. (2).

*ℛ*to normalize the difference and make the batch processing feasible, as shown in Fig. 3(b). Let

**a**= [

*a*

_{1},

*a*

_{2}, ···,

*a*],

_{m}**b**= [

*b*

_{1},

*b*

_{2}, ···,

*b*],

_{m}**f**and

**g**denote four row vectors and [

*a*

_{1}

**f**;

*a*

_{2}

**f**; ··· ;

*a*

_{m}**f**] and [

*b*

_{1}

**g**;

*b*

_{2}

**g**; ··· ;

*b*

_{m}**g**] denote low-rank fluorescence matrices at two pixels. We normalize them by factor matrix

*ℛ*= [

**a′**,

**a′**,···,

**a′**,

**b′**,

**b′**,···,

**b′**] and concatenate them to get a low-rank matrix [

**f g;f g**;··· ;

**f g**]. Here

*ℛ*needs to be estimated automatically. In addition, concatenating multiple image pixels will not change the sparsity of the reflective component. Then the optimization for reconstructing fluorescent and reflective components of a whole image can be rewritten as where ⊙ is the component-wise product that for any two matrices

**A**and

**B**, (

**A**⊙

**B**)

*=*

_{ij}**A**

_{ij}**B**

*. Note that the subsampling matrix*

_{ij}**Ω**is the horizontal replication of

**Ω̂**in Eq. (2).

## 4. Optimization

27. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learning **3**(1), 741–755 (2010). [CrossRef]

**(∀**

*ε**ij*,

*ε*≥ 0) to convert the inequality |

_{ij}**N**−

*μ*| < 3

*σ*into an equality constraint (

**N**−

*μ*) ⊙ (

**N**−

*μ*) − 9

*σ*

^{2}+

*ε*

^{2}= 0. In addition,

**F**⊙

*ℛ*and

**R**are replaced with

*S*

_{1}and

*S*

_{2}, respectively, to get closed-form solutions to

**F**and

**R**. So the objective turns into

29. Y. Deng, Q. Dai, and Z. Zhang, “An overview of computational sparse models and their applications in artificial intelligence,” Artif. Intell. Evol. Comput. Metaheuristics **427**, 345–369 (2012). [CrossRef]

27. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learning **3**(1), 741–755 (2010). [CrossRef]

**F**,

**R**and

**N**are optimization variables; matrices

**Y**

_{0∼3}define the Lagrangian multipliers; and the other matrices, e.g.,

**I**,

**O**, and

**N**, are all known. The above objective is analytically tractable by Distributed Optimization as used in [28]. To solve Eq. (7), we need to derive the update rules for all the unknowns. In the following derivations of updating rules, we omit the superscript (

*k*) or (

*k*+ 1) on the right-hand side of the derivation.

**S**

_{1}, the Lagrangian equation can be rewritten as where

*C*is irrelevant to

**S**

_{1}. According to [30], the update rule of such a nuclear norm optimization can be written as where

*US*is the Singular Value Decomposition of (

_{temp}V^{T}*ℛ*⊙

**F**−

*β*

^{−1}

**Y**

_{1}) and

**S**

_{2}as Referring to the solution to

*ℓ*

_{1}problem in [30],

**S**

_{2}can be updated as

**E**and

*ε*, we set the Lagrangian equation’s partial derivative to be zero and thus obtain the updated value. The Lagrangian equation’s partial derivative to

**E**is and the update rule is Similarly, we can get

*ε*’s update rule,

**F**,

**R**and

**N**, it is difficult to obtain the closed-form solution to the three equations; we use gradient descent method [29

29. Y. Deng, Q. Dai, and Z. Zhang, “An overview of computational sparse models and their applications in artificial intelligence,” Artif. Intell. Evol. Comput. Metaheuristics **427**, 345–369 (2012). [CrossRef]

*γ*

_{1∼3}represents the step size parameters. The corresponding partial derivatives are respectively

**Y**

_{0∼3}can be derived in closed form as listed in Algorithm 1, and the other variables are kept constant during the optimization:

*ρ*= 1.05,

*β*= 1

*e*− 2 and

*β*= 1

_{max}*e*6.

## 5. Experiments

### 5.1. Synthetic data

24. Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur, “Multiplexing for optimal lighting,” IEEE Pattern Anal. Mach. Intell. **29**(8), 1339–1354 (2007). [CrossRef]

24. Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur, “Multiplexing for optimal lighting,” IEEE Pattern Anal. Mach. Intell. **29**(8), 1339–1354 (2007). [CrossRef]

*m*+ 1)/4 is an integer, we use full-rank random multiplexing codes in this paper without loss of generality. The optimum rate for coded illumination is around 50% according to [24

24. Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur, “Multiplexing for optimal lighting,” IEEE Pattern Anal. Mach. Intell. **29**(8), 1339–1354 (2007). [CrossRef]

### 5.2. Real data

### 5.3. Advantages in other applications

#### 5.3.1. Relighting

5. G. M. Johnson and M. D. Fairchild, “Full-spectral color calculations in realistic image synthesis,” IEEE Comput. Graphics Appl. **19**(4), 47–53 (1999). [CrossRef]

#### 5.3.2. Segmentation

## 6. Conclusions and discussions

### 6.1. Conclusions

### 6.2. Limitations and potential extensions

## Acknowledgments

## References and links

1. | R. Donaldson, “Spectrophotometry of fluorescent pigments,” Br. J. Appl. Phys. |

2. | I. Sato and C. Zhang, “Image-based separation of reflective and fluorescent components using illumination variant and invariant color,” IEEE Trans. Pattern Anal. |

3. | A. D. McNaught and A. Wilkinson, |

4. | A. Springsteen, “Introduction to measurement of color of fluorescent materials,” Anal. Chim. Acta |

5. | G. M. Johnson and M. D. Fairchild, “Full-spectral color calculations in realistic image synthesis,” IEEE Comput. Graphics Appl. |

6. | M. B. Hullin, J. Hanika, B. Ajdin, H.-P. Seidel, J. Kautz, and H. P. A. Lensch, “Acquisition and analysis of bis-pectral bidirectional reflectance and reradiation distribution functions,” ACM Trans. Graphics |

7. | M. Soriano, W. Oblefias, and C. Saloma, “Fluorescence spectrum estimation using multiple color images and minimum negativity constraint,” Opt. Express |

8. | Q. Liu, K. Chen, M. Martin, A. Wintenberg, R. Lenarduzzi, M. Panjehpour, B. F. Overholt, and T. Vo-Dinh, “Development of a synchronous fluorescence imaging system and data analysis methods,” Opt. Express |

9. | T. Vo-Dinh, “Principle of synchronous luminescence (SL) technique for biomedical diagnostics,” Proc. SPIE |

10. | I. Sato, T. Okabe, and Y. Sato, “Bispectral photometric stereo based on fluorescence,” in |

11. | S. Han, Y. Matsushita, I. Sato, T. Okabe, and Y. Sato, “Camera spectral sensitivity estimation from a single image under unknown illumination by using fluorescence,” in |

12. | C. Chi, H. Yoo, and M. Ben-Ezra, “Multi-spectral imaging by optimized wide band illumination,” Int. J. Comput. Vision |

13. | M. Alterman, Y. Schechner, and A. Weiss, “Multiplexed fluorescence unmixing,” in |

14. | J. Park, M. Lee, M. D. Grossberg, and S. K. Nayar, “Multispectral imaging using multiplexed illumination,” in |

15. | S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Separating the fluorescence and reflectance components of coral spectra,” Appl. Opt. |

16. | M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual disperser architecture,” Opt. Express |

17. | A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. |

18. | R. Horisaki, X. Xiao, J. Tanida, and B. Javidi, “Feasibility study for compressive multi-dimensional integral imaging,” Opt. Express |

19. | R. Horisaki and J. Tanida, “Multi-channel data acquisition using multiplexed imaging with spatial encoding,” Opt. Express |

20. | Y. Wu, I. O. Mirza, G. R. Arce, and D. W. Prather, “Development of a digital-micromirror-device-based multishot snapshot spectral imaging system,” Opt. Lett. |

21. | Y. August and A. Stern, “Compressive sensing spectrometry based on liquid crystal devices,” Opt. Lett. |

22. | G. Wetzstein, I. Ihrke, and W. Heidrich, “On plenoptic multiplexing and reconstruction,” Int. J. Comput. Vision |

23. | N. Ratner and Y. Y. Schechner, “Illumination multiplexing within fundamental limits,” in |

24. | Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur, “Multiplexing for optimal lighting,” IEEE Pattern Anal. Mach. Intell. |

25. | C. Chen, D. Vaquero, and M. Turk, “Illumination demultiplexing from a single image,” in |

26. | F. Moreno-Noguer, S. Nayar, and P. Belhumeur, “Optimal illumination for image and video relighting,” in Proceedings of IEE European Conference on Visual Media Production (IEE, 2005), pp. 201–210. |

27. | S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learning |

28. | A. Yang, S. Sastry, A. Ganesh, and Y. Ma, “Fast-minimization algorithms and an application in robust face recognition: A review,” in |

29. | Y. Deng, Q. Dai, and Z. Zhang, “An overview of computational sparse models and their applications in artificial intelligence,” Artif. Intell. Evol. Comput. Metaheuristics |

30. | Z. Lin, M. Chen, L. Wu, and Y. Ma, “The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices,” in Technical Report UILU-ENG-09-2215 (UIUC, 2009). |

31. | M. Harwit and N. J. A. Sloane, |

32. | A. Lam and I. Sato, “Spectral modeling and relighting of reflective-fluorescent scenes,” in |

**OCIS Codes**

(300.6280) Spectroscopy : Spectroscopy, fluorescence and luminescence

(110.1758) Imaging systems : Computational imaging

(110.4234) Imaging systems : Multispectral and hyperspectral imaging

**ToC Category:**

Imaging Systems

**History**

Original Manuscript: September 12, 2013

Revised Manuscript: December 19, 2013

Manuscript Accepted: January 5, 2014

Published: January 17, 2014

**Virtual Issues**

Vol. 9, Iss. 3 *Virtual Journal for Biomedical Optics*

**Citation**

Jinli Suo, Liheng Bian, Feng Chen, and Qionghai Dai, "Bispectral coding: compressive and high-quality acquisition of fluorescence and reflectance," Opt. Express **22**, 1697-1712 (2014)

http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-22-2-1697

Sort: Year | Journal | Reset

### References

- R. Donaldson, “Spectrophotometry of fluorescent pigments,” Br. J. Appl. Phys. 5(6), 210–214 (1954). [CrossRef]
- I. Sato, C. Zhang, “Image-based separation of reflective and fluorescent components using illumination variant and invariant color,” IEEE Trans. Pattern Anal. 35(12), 2866–2877 (2013). [CrossRef]
- A. D. McNaught, A. Wilkinson, Compendium of Chemical Terminology (Blackwell Science, 1997).
- A. Springsteen, “Introduction to measurement of color of fluorescent materials,” Anal. Chim. Acta 380(2), 183–192 (1999). [CrossRef]
- G. M. Johnson, M. D. Fairchild, “Full-spectral color calculations in realistic image synthesis,” IEEE Comput. Graphics Appl. 19(4), 47–53 (1999). [CrossRef]
- M. B. Hullin, J. Hanika, B. Ajdin, H.-P. Seidel, J. Kautz, H. P. A. Lensch, “Acquisition and analysis of bis-pectral bidirectional reflectance and reradiation distribution functions,” ACM Trans. Graphics 29(4), 1–7 (2010). [CrossRef]
- M. Soriano, W. Oblefias, C. Saloma, “Fluorescence spectrum estimation using multiple color images and minimum negativity constraint,” Opt. Express 10(25), 1458–1464 (2002). [CrossRef] [PubMed]
- Q. Liu, K. Chen, M. Martin, A. Wintenberg, R. Lenarduzzi, M. Panjehpour, B. F. Overholt, T. Vo-Dinh, “Development of a synchronous fluorescence imaging system and data analysis methods,” Opt. Express 15(20), 12583–12594 (2007). [CrossRef] [PubMed]
- T. Vo-Dinh, “Principle of synchronous luminescence (SL) technique for biomedical diagnostics,” Proc. SPIE 3911, 42–49 (2000). [CrossRef]
- I. Sato, T. Okabe, Y. Sato, “Bispectral photometric stereo based on fluorescence,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 270–277.
- S. Han, Y. Matsushita, I. Sato, T. Okabe, Y. Sato, “Camera spectral sensitivity estimation from a single image under unknown illumination by using fluorescence,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2012), pp. 805–812.
- C. Chi, H. Yoo, M. Ben-Ezra, “Multi-spectral imaging by optimized wide band illumination,” Int. J. Comput. Vision 86(2–3), 140–151 (2010). [CrossRef]
- M. Alterman, Y. Schechner, A. Weiss, “Multiplexed fluorescence unmixing,” in Proceedings of IEEE International Conference on Computational Photography (IEEE, 2010), pp. 1–8.
- J. Park, M. Lee, M. D. Grossberg, S. K. Nayar, “Multispectral imaging using multiplexed illumination,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2007), pp. 1–8.
- S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, “Separating the fluorescence and reflectance components of coral spectra,” Appl. Opt. 40(21), 3614–3621 (2001). [CrossRef]
- M. E. Gehm, R. John, D. J. Brady, R. M. Willett, T. J. Schulz, “Single-shot compressive spectral imaging with a dual disperser architecture,” Opt. Express 15(21), 14013–14027 (2007). [CrossRef] [PubMed]
- A. Wagadarikar, R. John, R. Willett, D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. 47(10), B44–B51 (2008). [CrossRef] [PubMed]
- R. Horisaki, X. Xiao, J. Tanida, B. Javidi, “Feasibility study for compressive multi-dimensional integral imaging,” Opt. Express 21(4), 4263–4279 (2013). [CrossRef] [PubMed]
- R. Horisaki, J. Tanida, “Multi-channel data acquisition using multiplexed imaging with spatial encoding,” Opt. Express 18(22), 23041–23053 (2010). [CrossRef] [PubMed]
- Y. Wu, I. O. Mirza, G. R. Arce, D. W. Prather, “Development of a digital-micromirror-device-based multishot snapshot spectral imaging system,” Opt. Lett. 36(14), 2692–2694 (2011). [CrossRef] [PubMed]
- Y. August, A. Stern, “Compressive sensing spectrometry based on liquid crystal devices,” Opt. Lett. 38(23), 4996–4999 (2013). [CrossRef] [PubMed]
- G. Wetzstein, I. Ihrke, W. Heidrich, “On plenoptic multiplexing and reconstruction,” Int. J. Comput. Vision 101(2), 384–400 (2013). [CrossRef]
- N. Ratner, Y. Y. Schechner, “Illumination multiplexing within fundamental limits,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 1–8.
- Y. Y. Schechner, S. K. Nayar, P. N. Belhumeur, “Multiplexing for optimal lighting,” IEEE Pattern Anal. Mach. Intell. 29(8), 1339–1354 (2007). [CrossRef]
- C. Chen, D. Vaquero, M. Turk, “Illumination demultiplexing from a single image,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2011), pp. 17–24.
- F. Moreno-Noguer, S. Nayar, P. Belhumeur, “Optimal illumination for image and video relighting,” in Proceedings of IEE European Conference on Visual Media Production (IEE, 2005), pp. 201–210.
- S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learning 3(1), 741–755 (2010). [CrossRef]
- A. Yang, S. Sastry, A. Ganesh, Y. Ma, “Fast-minimization algorithms and an application in robust face recognition: A review,” in Proceedings of IEEE Conference on Image Processing (IEEE, 2010), pp. 1849–1852.
- Y. Deng, Q. Dai, Z. Zhang, “An overview of computational sparse models and their applications in artificial intelligence,” Artif. Intell. Evol. Comput. Metaheuristics 427, 345–369 (2012). [CrossRef]
- Z. Lin, M. Chen, L. Wu, Y. Ma, “The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices,” in Technical Report UILU-ENG-09-2215 (UIUC, 2009).
- M. Harwit, N. J. A. Sloane, Hadamard Transform Optics (Academic, 1979).
- A. Lam, I. Sato, “Spectral modeling and relighting of reflective-fluorescent scenes,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2013), pp. 1452–1459.

## Cited By |
Alert me when this paper is cited |

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.

« Previous Article | Next Article »

OSA is a member of CrossRef.