In traditional photography, it is often necessary to modify a picture to enhance its visual quality, or to create special effects that are absent in the image acquisition process. Amongst different techniques, relighting is perhaps one of the most popularly practiced methods as it allows the optical properties (such as illumination), which may be difficult to control in the real world environment, to be synthesized. Nowadays, relighting can now be conducted by both professional and novice users through a comprehensive choice of commodity photo-editing software. In brief, a picture is relit by modifying the value of each pixel according to a given criteria, an operation commonly referred to as the 'point' process. For example, the effect of a spotlight can be simulated by modulating the luminance of each pixel with the spatial distribution of the illumination. Apparently, it will be desirable if the relighting mechanism can be applied to digital holograms to enhance their impact to the observers. The problem is, rendering a digital hologram with the point process is erroneous, as each pixel is representing the holistic information from the entire object scene. Until now, research on hologram relighting has not been investigated. A straightforward solution is to render the original object scene, if it is still available, whenever a relighting task is required, and then regenerate the hologram afterwards. Despite the effectiveness and simplicity of this method, the process is time-consuming as the numerical generation of a digital hologram involves enormous amount of arithmetic operations. Although there are quite a number of fast algorithms, such as [1

T. Shimobaba, H. Nakayama, N. Masuda, and T. Ito, “Rapid calculation algorithm of Fresnel computer-generated-hologram using look-up table and wavefront-recording plane methods for three-dimensional display,” Opt. Express 18(19), 19504–19509 (2010). [CrossRef] [PubMed]

–5

T. Yamaguchi, G. Okabe, and H. Yoshikawa, “Real-time image plane full-color and full-parallax holographic video display system,” Opt. Eng. 46(12), 125801 (2007). [CrossRef]

] which attempt to alleviate the problem, they are not capable of generating holograms in real time (e.g., at the video frame rate) if there are large number of object points. Besides, if the digital hologram is captured with optical means, the original scene may not be available afterwards. If that is the case, theoretically we can apply some sort of inverse mapping [6

P. W. M. Tsang, K. W. K. Cheung, T. Kim, Y. S. Kim, and T.-C. Poon, “Fast reconstruction of sectional images in digital holography,” Opt. Lett. 36(14), 2650–2652 (2011). [CrossRef] [PubMed]

–8

L. Denis, D. Lorenz, E. Thiébaut, C. Fournier, and D. Trede, “Inline hologram reconstruction with sparsity constraints,” Opt. Lett. 34(22), 3475–3477 (2009). [CrossRef] [PubMed]

] to reconstruct, and then relight the scene image. Subsequently, the rendered scene image can be converted into a hologram. However, as reported in some literatures, the inverse process itself is complicated. Besides, so far only the reconstruction of holograms representing sparse images (i.e., images that contain few number of object points) had been successfully demonstrated. In this paper, we propose a fast method for relighting digital hologram without the presence, or the reconstruction of the original object scene. Our scheme is based on the wavefront recording plane (WRP) which is originally employed in fast generation of digital holograms [1

T. Shimobaba, H. Nakayama, N. Masuda, and T. Ito, “Rapid calculation algorithm of Fresnel computer-generated-hologram using look-up table and wavefront-recording plane methods for three-dimensional display,” Opt. Express 18(19), 19504–19509 (2010). [CrossRef] [PubMed]

] [9

P. Tsang, W.-K. Cheung, T.-C. Poon, and C. Zhou, “Holographic video at 40 frames per second for 4-million object points,” Opt. Express 19(16), 15205–15211 (2011). [CrossRef] [PubMed]

]. To our knowledge, it is the first time we have applied the WRP concept to achieve real-time relighting of holograms. In our proposed method, a digital hologram is projected onto a virtual wavefront recording plane (WRP) which is placed sufficiently near to the object points in the scene. At close proximity, each object will only cast its optical wave on a small region on the WRP. Hence, relighting the intensity of a pixel in the WRP is equivalent to modifying the intensities of a small cluster of object points that is contributing to the pixel of interest. On this basis, we apply relighting to the WRP, and subsequently expand it to a full hologram. The entire process mainly involves 4 Fast Fourier Transform (FFT) operations which can be realized with the graphical processing unit (GPU) in less than 20ms for a hologram comprising of 2048x2048 pixels. Experimental results demonstrated that the target relighting effects are correctly synthesized in the reconstructed images of holograms that are relit with our proposed method.

The concept of the proposed hologram relighting can be illustrated with Fig. 1 . To start with, we insert a hypothetical diffraction plane, known as the wavefront recording plane (WRP), between the digital hologram and the scene. Given an arbitrary object point, its optical wave will propagate by diffraction to the entire hologram. Other object points in the scene are contributed to the hologram in a similar manner. Hence, modifying a hologram pixel will affect the diffracted waves contributed by the entire scene image, instead of localizing in the region around the pixel of interest. However, as shown in the diagram, an object point will only cover a small area on the WRP (the dotted region). The closer the distance between the object point and the WRP, the smaller will be the coverage (a.k.a. the support) of the diffraction pattern on the latter. As such, relighting a pixel in the WRP will only affect the diffraction pattern of a small cluster of object points that share the same support. Our proposed relighting method is realized in 3 stages. First, we derive the WRP based on the mathematical framework in [1

T. Shimobaba, H. Nakayama, N. Masuda, and T. Ito, “Rapid calculation algorithm of Fresnel computer-generated-hologram using look-up table and wavefront-recording plane methods for three-dimensional display,” Opt. Express 18(19), 19504–19509 (2010). [CrossRef] [PubMed]

] that described the relationship between the object points in a 3D scene, the field distribution on WRP uw ( x,y), and the hologram u ( x,y). These three entities are assumed to have the same horizontal, as well as vertical extents of X and Y units.-The complex wavefront contributed by the object points on the WRP is given by (Eq. (1).where 0< xj<X and 0< yj<Y are the horizontal and vertical positions of the jth object point. ajand R wj= ( x− xj)2+ ( y− yj)2+ dj2 are the amplitude of the 'jth' object point and its distance from the WRP, respectively. djis the perpendicular distance from the jth object point to the WRP and λis the wavelength of the reference light. As the object scene is very close to the WRP, the diffracted beam of each object point only covers a small square window of size W×W (the dotted window in Fig. 1). As such, Eq. (1) can be rewritten as (Eq. (2).where Fj= { aj R wjexp ( i 2πλ R wj) if | x− xj| and | y− yj|< 12W 0 otherwise.

Next, the WRP is expanded to a hologram u ( x,y) given by (Eq. (3).where F [⋅] and F -1 [⋅] denote the forward and inverse fast Fourier transform (FFT), respectively. K=− i λ zwexp ( i 2πzλ) is a constant and h ( x,y)=exp ( i π λ zw ( x2+ y2)) is a fixed impulse function for a given separation zw between the WRP and the hologram. From Eq. (3), the inverse process projecting the hologram to the WRP can be determined as (Eq. (4).

In the second stage, the WRP obtained in Eq. (4) is modulated with the relighting image (RI) G ( x,y) that simulates a given relighting condition. For example, the RI in Fig. 2 emphasizes the intensity within a circular region around the center of the scene, creating the effect of a spotlight. The relit WRP is given by (Eq. (5).

Subsequently, uwL ( x,y) is expanded to a hologram as (Eq. (6)

The complete relighting process of our proposed method involves 4 FFT operations (2 in Eq. (4) and 2 in Eq. (6)), which constitutes to the majority of arithmetic operations. The pair of terms, 1 F [ h ( x,y)]and F [ h ( x,y)] can be pre-calculated in advance and stored in a look up table (LUT), and hence not count towards the computation load. With a typical PC and a GPU, the 4 FFTs can be executed in less than 20ms. The computation time on the rest of the process, comprising of multiplication between pairs of 2D arrays (e.g. Equation (5)), is negligible.

Our proposed hologram relighting method is demonstrated with the double depth image shown in Fig. 3(a) . The image is evenly partitioned into a left side and a right side, located at 0.55m and 0.6m from the hologram, respectively. A digital Fresnel hologram is generated with the 'point-light' method described in [10

T.-C. Poon, ed., Digital holography and three-dimensional display: Principles and Applications (Springer, 2006).

]. The wavelength of the optical beam and the pixel size of the hologram are 650nm and 7um, respectively. A relighting image G ( x,y) shown in Fig. 3(b) is employed to simulate the directional illumination emerging from the upper right corner. G ( x,y) is divided into an illuminated (the white area) and a shadow (the grey area) regions that are separated by a sharp boundary. When relighting is not applied, the numerical reconstructed images at the two depth planes are shown in Figs. 3(c) and 3(d). When either side of the reconstructed image is in focus, it is a good recovery of the original content.

Next we relight the hologram directly with G ( x,y) by multiplying the two images on a pixel by pixel basis. A hologram u ( x,y), after direct relighting with an image G ( x,y), is given by (Eq. (7)

The reconstructed images of uDL ( x,y) at the two depth plane are shown in Fig. 4(a) and 4(b). We observe that the relighting effect is not totally in line with the relighting image. Notably, the boundary between the illuminated and the shadow regions are fuzzy, and the area around it is heavily contaminated with slanting bars. The defects exhibited in Figs. 4(a) and 4(b) are expected, as direct modification of a small part of the hologram will change the diffraction waves contributed by the entire object scene instead of localizing in the neighborhood of the modified area. To overcome this problem, we applied our proposed method to relight the digital hologram. The latter is first converted to the WRP uw ( x,y) based on Eq. (4), and multiplied with the relighting imageG ( x,y). Subsequently, the result is expanded into a relit hologram based on Eq. (6). The reconstructed images of the relit hologram at the two depth planes are shown in Figs. 5(a) and 5(b). We observe that the relighting effect is in good agreement with the relighting image, with a clear boundary between the illuminated and the shadow regions.

To further illustrate our proposed method, we generated the digital hologram u ( x,y) of a hemisphere which is rendered with the texture of the earth image as shown in Fig. 6(a) . The hemisphere has a radius of 0.005m, with the tip located at 0.001m and 0.3m from the WRP and the hologram, respectively. A real, off-axis hologram H ( x,y) is generated by adding a planar reference wave R (y) (illuminating at an inclined angle 1.2o on the hologram) to u ( x,y), and taking the real part of the result to give (Eq. (8)where RE [⋅] denotes the real part of a complex variable. The real, off-axis hologram is displayed on a liquid crystal on silicon (LCOS) modified from the Sony VPL-HW15 Bravia projector having a horizontal and vertical resolution of 1920 and 1080, respectively, and a dot-pitch of 7μm. The optical reconstructed image displayed on the LCOS device is shown in Fig. 6(b). Subsequently, we applied our proposed method to relight the hologram u ( x,y) with the image in Fig. 2. The latter is translated horizontally in a back and forth manner to generate the effect of a panning spotlight. Each relit hologram (corresponding to a particular spotlight position) is then converted into a real hologram based on Eq. (8), and reconstructed on the LCOS display. A single frame excerpt of the optical reconstructed animation clip (Media 1) is shown in Fig. 6(c). The clip is showing 25 frames per second. It can be seen from the excerpt, as well as in the animation clip, that the effect of the panning spotlight is correctly generated in the sequence of reconstructed images.

This paper reports a fast method for relighting a digital hologram without the need of re-generating the latter from the original object scene. We note that traditional photographic relighting techniques based on the 'point' process is not applicable on a hologram. To overcome this problem, we project the digital hologram to a hypothetic WRP that is inserted near to the object scene. Due to the close proximity, the wavefront of each object point is occupying a small neighborhood on the WRP. As such, relighting can be conducted by modulating the WRP with a relighting function, and expanded the modified WRP to a full hologram. Our proposed method is realized with the GPU, and the time taken to relight a 2048x2048 hologram is less than 20ms, equivalent to a rate of over 50 frames per second. Experimental results reveal that the reconstructed images of holograms that are processed with our proposed method are correctly relit with the target optical effects. In our evaluation, we have only adopted some simple relighting effects for synthesizing different kinds of illumination, mainly to demonstrate the effectiveness of our approach. However, it is evident that the current framework can be easily applied to handle more sophisticated effects, such as the image-based and the geometry-based relighting.