The factors that drive the design of head-worn display systems (HWD) are cost effectiveness, portability, and light-weight packaging with an ergonomic form factor, as well as at least a 20 degree field-of-view (FOV.) A recent review of HWDs was reported in [1
1. O. Cakmakci and J. Rolland, “Head-worn displays,” J. Display Technol. 3, 199–216 (2006). [CrossRef]
]. Among HWDs, the head-worn projection display (HWPD) has attracted much interest because of its wide FOV (i.e., greater than 40 degrees) distortion-free images. Fischer, the initial developer of HWPD systems, employed a combination of projection optics and a retro-reflective screen placed in the environment to develop a projection-based HWD assisted by a crane [2
2. R. E. Fischer, “Optics for head-mounted displays,” Information Disp. 10, (1994).
]. The first prototype was developed with commercial off-the-shelf (COTS) projection optics and a COTS retro-reflective screen. The major benefit of using a retro-reflective screen instead of various other projection screens is the small scattering angle of the screen that maximizes the brightness of the return image. Since 1998, Rolland and her team have developed prototype HWPDs utilizing custom designed projection optics integrating a combination of glass and plastic components, thus reducing the overall weight of the system (e.g.., as low as 6 grams per eye.). The prototypes have included aspheric surfaces and diffractive optical elements (DOEs) to experiment with the tradeoffs of reduced weight and overall image quality. [3
3. J.P. Rolland, F. Biocca, F. Hamza-Lup, Y. Ha, and R. Martins “Development of Head-Mounted Projection Displays for Distributed, Collaborative Augmented Reality Applications, Presence: SI Immersive Projection Technology, 5, (2005).
Until recently, a limiting factor that constrained the use of HWPD technology was the requirement to place the retro-reflective screen in the environment, thus restricting its use only to indoor areas. While there are many indoor applications, there are also many outdoorr applications of interest. Developing a HWPD systems that is functional outdoors is the driving force behind the research reported in this paper. Our work is focused on conceiving and developing an HWPD with a retro-reflective screen integrated within the system itself, thereby providing full mobility. The resulting HWPD system will be referred to as a mobile HWPD (M-HWPD.) Although this integration presents several challenges, the positive results obtained with the conceived system provide the impetus for the continuation of our research to move the M-HWPD technology described here to a full-scale commercial solution.
In this paper, we will first review the principle of a binocular M-HWPD that integrates projection optics, an imaging lens, and a retro-reflective screen, and show the newly developed assembly of the first M-HWPD prototype. Next, we will demonstrate the feasibility of replacing eyepiece-based (also called direct-view) HWDs with moderate FOVs (>20 degrees), extensively used since the 1960s for various indoor and outdoor applications, with projection based M-HWPD. Furthermore, we will establish the requirements for a custom-designed, retro-reflective screen for imaging applications and we will demonstrate a typical augmented reality (AR) image captured outdoors using the M-HWPD described in this paper. Finally, we conclude with an overall assessment of the M-HWPD technology and a discussion of follow-up research planned by our group to advance this emerging technology.
Previously, as a proof of concept, we assembled a monocular bench setup consisting of a retro-reflective screen and an imaging lens (L1
) along an alternate optical path (Path 2) provided by a beam splitter, as shown in Fig. 1(a)
4. R. Martins, V. Shaoulov, Y. Ha, and J. P. Rolland, “Projection based head-mounted displays for wearable computers” Proc. SPIE 5442, (2004).
]. The bench setup was made up of a 100 mm diameter lens L1
with a xx mm focal length integrated with a 100-by-100 mm section of a COTS retro-reflective screen. The observed results illustrated the feasibility of integrating a retro-reflective screen with an HWPD. As a result, the research progressed into an actual M-HMPD, which is first demonstrated in this paper and is shown in Fig. 1(b–c)
. The driving criterion for the M-HWPD prototype design was compactness while using a COTS lens and retro-reflective screen. In addition to the demonstration of the M-HWPD concept in an actual HWPD system, a new design to mount the display is presented that incorporates a flexible hat for mounting optical components and distributing the weight uniformly on the user’s head. This that the headset can be worn for extended periods of time [5
5. A. Oranchak and J. Rolland, “Elastic head mounting platform cap,” US Patent Filed July 2006.
Fig. 1. (a) First order layout for one eye of the see-through M-HWPD with retro-reflective screen placed along Path 2. (b) Assembly of a binocular see-through M-HWPD with robust titanium mounting structures and the integrated retro-reflective screen. (c) User wearing the binocular M-HWPD
2. M-HWPD System
The M-HWPD, similar to the HWPD, utilizes a micro-display, which then motivates the choice of FOV and visual performance. A typical layout is shown in Fig. 1 (a)
. The essence of the M-HWPD innovation is in transferring the physical retro-reflective screen from the environment to the headset, thus integrating it within the headset at the conjugate image plane location of the projection optics. This is achieved by imaging the retro-reflective screen in the M-HWPD using an imaging lens L1
. Without such imaging lens, the integration would lead to a non-resolvable virtual image, since the retro-reflective screen would not lie in the image plane. Quantitatively, in the configuration without L1
, the amount of blurring produced would result in an unfocused image with a resolution of ≥400 arcmin, rendering the image undistinguishable. An imaging lens L1
is thus placed between the projection optics and the retro-reflective screen, allowing the physical screen to be optically imaged in front of the user. The imaging distance can be varied by adjusting the imaging conjugates of the projection optics and the location of the retro-reflective screen with respect to the focal point of L1
. For example, if it is desirable to work with a collimated image, the projection optics is optimized to form an image at optical infinity, and the image of the retro-reflective screen will be set to infinity by placing it at the focal point of L1
. However, in many instances, the optical image will reside at a finite distance from the user, and thus the physical screen will be placed inside the focal point of L1
In the current design, which uses a COTS lens L1, the entire optical system has an overall length of approximately 120 mm measured from the micro-display to the retro-reflective screen. In practice, one could design the lens L1 to be telecentric, however this would sacrifice one of our primary design criterion for the most compact solution. The configuration is based around a 15 mm eye relief, which was also selected in order to maximize the compactness of the system. An eye relief of 25 mm has been recommended with a 95th percentile human head circumference (MIL-STD-1472D), with a minimum of 15 mm needed for eyeglasses to be comfortably worn with the HWPD. Although the 15 mm eye relief does not provide adequate distance for all users with eyeglasses, we can custom fit for near- and far-sightedness by refocusing the projection optics.
Theoretically, the most compact overall length is achieved with the shortest focal length possible for L1, but the latter is limited by the diameter or equivalent F-number of L1. Because of the retro-reflective screen properties (i.e. the light falling on the retro-reflective screen is reflected back on itself, creating a condition similar to phase conjugation), in principle the imaging lens L1 can satisfy the first-order imaging properties (i.e. the optical path difference, or the OPD, will be zero if all the light is perfectly retro-reflected back on itself). Thus, from a geometrical optics perspective, a compact Fresnel lens can serve as the imaging optics in place of L1. However, as the F-number decreases, the grooves of the Fresnel lens get deeper, affecting the light transmission properties of the Fresnel lens and creating an image with significantly reduced sharpness. Therefore, in practice, an F-number of 0.7 may be considered to minimize transmission losses, which is close to the limits of state-of-the-art fabrication techniques. Also, there is a trade-off between increasing the accuracy of the Fresnel lens phase profile and minimizing the diffraction by the Fresnel grooves; as the number of grooves increases, light throughput through the Fresnel lens decreases. The quantification of these parameters is beyond the scope of this paper and will be reported in a focused investigation of a custom Fresnel lens as one solution to creating a compact module for the M-HWPD
2.1 Microdisplay Device
The M-HWPD prototype reported in this paper is based on a COTS organic, light-emitting micro-display (OLED), with a 0.6 inch diagonal, composed of 800-by-600 pixels. This is the display source for the projection optics. The major benefit of selecting the OLED micro-display compared to other COTS micro-displays is that its composition (as a series of thin-film, organic substrates sandwiched between two conductors producing a self-emitting display source on a chip) reduces the bulkiness of the electronic components as well as removes the requirement for an external light source [5
5. A. Oranchak and J. Rolland, “Elastic head mounting platform cap,” US Patent Filed July 2006.
]. Such a micro-display facilitates the design of a compact and light-weight optical assembly and minimizes the complexity of the optomechanical assembly as well. Our requirement for the most compact display narrowed our choice to an OLED SVGA micro-display for the design. The tradeoff in OLEDs is reduced brightness compared to custom designed LED-based illumination schemes that are commonly used in LCOS, LCD, and DLP micro-displays [6
6. L. Bogaert, Y. Meuret, B.V. Giel, and H. Thienpont, “LED based full color stereoscopic projection system,” Proc. SPIE 6489, (2007). [CrossRef]
]. Because one of the HWD main research goals is establishing the most compact solutions, the geometry of the M-HWPD presented in this paper will provide a path to viable compact commercial solutions as OLED micro-displays, or equivalent self-emitting technologies emerge. Approaches based on external illumination that are highly relevant for today’s product development will become less relevant for the long-term advancement of HWD technology [7
7. A. Rapaport, J. Milliez, A. Cassanho, H. Jenssen, and M. Bass, “Review of the properties of Up-Conversion Phosphors for new Emissive Displays,” J. Display Technol. 2, 68 (2006). [CrossRef]
2.2 Projection Optics
Various applications may require different operating distances, but state-of-the-art HWDs typically offer only one optimal viewing distance. Here, the projection optics were optimized for multiple (3 in this case) viewing distances simultaneously, 1.5m, 3.5m and infinity. The effectiveness of this optimization technique, which could be applied to any HWD, has been validated in human perception studies [8
8. C. Fidopiastis. User-centered virtual environment assessment and design for cognitive rehabilitation applications. Ph.D. Dissertation, University of Central Florida (2006).
]. For example, if the tasks to be performed are solely in the far field, it is optimal to set the distance of the optical image for each eye to be located beyond 6 m (i.e., at optical infinity). If the desired application also involves the manipulation of objects in the near field, the optical system has been designed to also form a sharp optical image at 1.5m. While the optimization for multiple conjugates reduces slightly the performance at all viewing distances, the loss in resolution is small compared to the gain in functionality. In this system, the same projection optics module, as shown in Fig. 2
, can be effectively used for all three viewing distances by adjusting the back focal distance,, which is accomplished by a slight rotation of the optics barrel.
Fig. 2. Monocular lens-mount assembly.
The M-HWPD was designed for a 12 mm exit pupil diameter, which comfortably allows for natural eye movements within the 42 degrees FOV without vignetting. It is important to note that the pupil of the optical system is located within the optics in a HWPD, which together with the integration of the beam splitter oriented at 90 degrees, yield a projection optics pupil that is the optical conjugate of the eye’s pupil. Because the pupil is internal to the projection optics, the M-HWPD can be more easily corrected for optical distortion. In contrast, conventional eyepiece-based HWDs provide an external pupil making it not only challenging to control distortion but also to minimize the weight of the eyepiece optics, which increases as the cube of the FOV.
The optical performance was characterized by evaluating the polychromatic modulation transfer function (MTF) for the full 12 mm pupil. The MTF plots predict the contrast as a function of spatial frequency for three optical images distances (1.5m, 3.5m, and infinity). The maximum spatial frequency of interest is set by the 15 μm pixel size of the miniature display. The Nyquist frequency was computed using the pixel diagonal size, which is approximately 24 cycles/mm. The lens was designed to support a minimum criterion of 20% modulation across all FOVs for a 12mm effective eye pupil at 24 cycles/mm. If this performance metric is satisfied across the full pupil, it will be satisfied for the 3 mm effective pupil as well as for all of its decentered values. The MTF curves across the full 12 mm pupil are shown in Fig. 3
. Results show that the design exceeds the design specifications required to produce a well-balanced image quality across the entire FOV. The fact that the MTFs across the three different image depths are quite similar in value indicates that our projection optics have been corrected effectively for all three image distances.
Fig. 3. The MTF plots for a projected scene located at (a) 1.5 m, (b) 3.5 m, and (c) infinity based on a 12 mm pupil diameter for the projection optics.
Another key benefit to a projection-based HWD versus an eyepiece (direct-view) HWD is the low percent distortion across the image. In the current projection system, we were able to limit distortion to a maximum of 1% across the field at all the three image distances, as shown in Fig. 4
. By utilizing a projection system, the inherent properties of a lens system that is symmetrical about the stop are used to substantially reduce the odd aberrations such as distortion, coma, and lateral chromatic aberrations. In contrast, in an eyepiece system that has an external stop, symmetry is not possible. An eyepiece system is commonly accepted as well corrected for distortion if distortion is limited to 3–5%, although distortions in the range of 8–12% are common with a FOV of 60 degrees.
Fig. 4. Distortion plots for a projected scene located at (a) 1.5 m, (b) 3.5 m, and (c) infinity across the full 12 mm of the projection optics.
2.3 Retro-reflective screen
The M-HWPD system is currently hindered by the COTS retro-reflective screen. Various COTS retro-reflective screens were investigated and found to be fabricated with corner-cube microstructures of approximately 150 μm. By integrating such retro-reflective screens with imaging optics, the microstructures are magnified by L1 and become visible on the image plane, which reduces the perceived computer-generated image resolution. In addition, the large corner-cube microstructures yield an additional degradation in image sharpness caused by the retro-reflected rays departing a maximum of 150 μm from their incident location on the Fresnel lens, causing residual optical aberrations. To eliminate the loss of resolution caused by the retro-reflective screen and thus improve visual performance, we have established design requirements for a custom-designed, retro-reflective screen that should result in a miniaturization of the microstructures, whose perceived sizes would be below that of human visual acuity. Throughout our further discussion, we will consider only corner-cube-based retro-reflective screens.
There are two key aspects of imaging with an integrated retro-reflective screen that affect the selection of first order parameters. The first is the construction of the corner-cube microstructure in terms of retro-reflected angle, and the second is the magnification produced by L1
. Provided that the retro-reflective screen is manufactured with three perfectly orthogonal surfaces, the rectilinear propagation from a point entering the surface lens L1
will also exit at approximately the same position after retro-reflection. By satisfying this condition, the incident and retro-reflected angles will be equal for a corner-cube design with orthogonal surfaces. We can also conclude that the properties of the retro-reflective screen are of greater importance than those of L1
, since the light entering the lens exits the lens at approximately the same location after being retro-reflected. Thus L1
, regardless of its physical properties except those affecting its transmission, will yield a zero optical path difference between the incident and transmitted light, canceling the optical aberrations. The second requirement of the retro-reflective screen is the required aperture size and depth of the trihedral corner-cube after the magnification produced by the lens L1
. It should be noted that if we implement a shorter focal length, the magnification of the microstructures will increase and the pixel width of the image at the screen will decrease, making it even more difficult to fabricate a miniaturized microstructure. If we consider the first order layout, as shown in Fig. 5
, the height of the virtual image hprojection
, given by the projection optics module, is perceived at a distance zprojection
and will subtend a FOV with a half angle θhalf-Fov
, as given by
Therefore, the image seen through L1 located at a distance zimage will yield a slightly magnified image with respect to the OLED size HOLED and distance ZOLED given by:
Fig. 5. Monocular Lens-Mount Assembly.
When an image is formed on the retro-reflective screen with the appropriate magnification defined by Eq. (2
), we can separate the image into individual pixels and compare the pixel area versus the trihedral aperture area. In the condition where a single pixel area is smaller than the area of a single trihedral aperture, multiple pixels, each with their own corresponding color, will be inverted with respect to their neighboring pixels. This occurrence is caused by the corner-cube construction, which will invert the incoming pixel information. Consequently, a local inversion will occur within a finite area reducing the resolution. In our case, the COTS screen has a ratio of pixel area to trihedral aperture area of approximately 25/1. Therefore, we expect that the image will clearly show artifacts, such as an array of magnified corner-cube structures, as well as an AR image with a loss in resolution compared with the initial 800-by-600 OLED resolution. In contrast, with the area of a single pixel being greater than the
aperture area of the trihedral corner-cube, the local inversion will only occur on each emitted pixel color. Inverting the color of any individual pixels will not directly affect the overall AR image. Therefore, it is desirable that the custom-designed, retro-reflective screen have a smaller trihedral aperture area than pixel area.
In this research, we demonstrated a fully integrated, see-through, wearable M-HWPD as a novel method of utilizing HWPD technology for mobile outdoors applications. Currently, the integration yields optical elements in close proximity to the user’s mouth that could present condensation and fogging with cool outdoor temperatures. An immediate solution is to embed a dense fabric cover to shield the retro-reflective screen and the Fresnel lens from unwanted condensation and potential other environment effects. With the addition of light control devices, for example, photo or electrochromic windows to attenuate the external light, and a custom designed retro-reflective screen, the M-HWPD design can ultimately provide SVGA quality computer generated images superimposed on top of the natural environment at various levels of illumination. Future research will focus on the development of custom designed, nano-fabricated, retro-reflective microstructures, as well as novel micro-optics designs to replace the Fresnel lens and retro-reflective screens for more compact solutions. Finally, the development of an electrochromic window for the M-HWPD can provide a feasible solution for adjustment of the ambient light, thus achieving optimized imaging in outdoor environments.