In 2009 augmented reality technology (AR) became mainstream. Though it has been under development for over four decades, in the past year it was prominently featured in major ad campaigns and was on the cover of Esquire. Concurrently, Layar, Wikitude, and a number of AR applications were released for mobile phones.  The future potential of AR has now captured the imagination of both the public and the press. The hype surrounding this technology is similar to the excitement over virtual reality during the 1990’s and 3D online communities, namely Second Life, during this past decade. Unfortunately, in the mind of consumers, neither of these technologies lived up to the hype. Due to a lack of understanding, virtual reality and 3D online communities were unfairly and prematurely dismissed as failures by many. AR is in danger of suffering the same fate. Geoff Northcott described the situation well in his post Augmented Reality, Second Life, and the Trough of Disillusionment.

In an effort to help manage expectations regarding AR technology, I will briefly describe what works today while clarifying what we can expect in the future.

Augmented Reality Glasses are not Viable in the Near Term

When covering AR, a number of technology pundits have assumed that within the next few years, we can expect head mounted displays or augmented reality “glasses” to become the best display for AR applications. Without a doubt, it would be groundbreaking if a high quality AR display could be built into the form factor of sunglasses. Unfortunately, a lightweight, wide field of view, daylight readable, head mounted display (HMD) at mass market prices is not something we can expect to see in the next five years. I have either bought or used most head mounted displays sold commercially since 1992, and I have seen great strides over the years in HMD resolution, brightness, and power usage.  To illustrate where the technology now stands, here are the best that I have used.

-Fakespace Labs “Wide 5” : Easily, the most immersive head mounted display I have tried, it can be modified for video see through augmented reality applications.

-The “Advanced Helmet Mounted Display”:  Optical Research Associates, a firm that helped fix the Hubble telescope, designed the optics. It has an amazing field of view, it’s optical see through, and has great resolution. It is also very large and is intended for helmet mounting.

-eMagin Z800 3Dvisor: It has a relatively small form factor, it’s very affordable, uses OLED’s, and it can be adapted for video see through augmented reality applications. The field of view is limited.

No one has yet delivered a wide field of view display in a small package that approaches the footprint of sunglasses as the public expects. With significant engineering investments, innovative approaches from companies such as Lumus, ORA, and Digilens may show promise in solving this problem. In addition to optics issues, eye strain and other head mounted display ergonomics problems must be dealt with.  For a detailed overview of the current and future state of head mounted display technology, I suggest the following:

-The Coming Generation of Head-worn Displays (HWDs): Will the Future come to us through new Eyes? Kevin and Jannick Rolland-Thompson, James P. McGuire, and Ozan Cakmakci, Optical Society of America 2009 Annual Meeting, San Jose, CA Download PDF

-The Past, Present, and Future of Head Mounted Display Designs” by Jannick Rolland and Ozan Cakmakci (College of Optics and Photonics: CREOL & FPCE, University of Central Florida) Download PDF

Ultimately, there will be no single “ultimate” AR display. Instead, the type of display used for AR will be determined by the needs of the application. Current and future developers will have a range of options including smart phone screens, handheld tablets, desktop monitors, micro projectors, and AR glasses.

GPS and Compass is not Enough

There are a number of mobile augmented reality applications, namely Layar, Wikitude, and the Yelp Monocle, which use GPS and compass data to overlay graphic information on a live video view of the real world. Though these applications are novel and interesting, the data provided by a mobile device’s GPS and compass is simply not precise enough to deliver a quality AR user experience. Information overlays usually appear to wobble or bounce around the video view. Consumers will rarely use these applications after the novelty factor wears off. At best, these applications provide an alternative viewing mode for data that should be first presented on a 2D map or in a list. Here’s a well written piece from New Scientist commenting on the current state of mobile augmented reality applications: Augmented Reality Gets off to a Wobbly Start

Sensor Fusion is the Answer

The best approach to AR tracking and registration involves hybrid tracking and sensor fusion techniques which use computer vision technology in conjunction with GPS and compass data. This paper provides deeper insight.

-Azuma, Ronald T., Bruce R. Hoff, Howard E. Neely III, Ronald Sarfaty, Michael J. Daily, Gary Bishop, Vern Chi, Greg Welch, Ulrich Neumann, Suya You, Rich Nichols, and Jim Cannon. Making Augmented Reality Work Outdoors Requires Hybrid Tracking. Proceedings of the First International Workshop on Augmented Reality, (San Francisco, CA, 1 November 1998), 219-224. Download PDF

Mobile augmented reality applications using a variety of sensor fusion techniques have been prototyped over the years, and we will hopefully see applications leveraging them on the market soon.

AR Hardware Platforms for Today

Here are five AR hardware platforms that developers can utilize today for creating augmented reality applications.

-Webcams

In the past year, there has been an explosion of ad campaigns which deliver webcam based AR experiences through an internet site or a downloadable application. Most of these utilized the FLARToolkit. I’ve used it to develop a number of AR web demos, available via the links below.

-Augmented Reality Interactive Album Cover

-Augmented Reality Concert

-Augmented Reality Fashion Ad

-Augmented Reality Business Card

-Kiosks, Digital Signage, Window Displays

Webcam AR applications can be packaged as interactive kiosks, digital signs, or window displays installed in retail locations as seen in this excellent example from Lego and this one promoting the movie Coraline.

-3G Smartphones

3G smartphones have enormous potential for AR applications. Phones from manufacturers including Apple, HTC, and Nokia are equipped with a camera, GPS, compass, accelerometers, and 3D graphics capabilities. These devices have enormous potential for AR applications. Unfortunately developers are currently more limited by the restrictions of manufacturer API’s than by the hardware itself. For example, developers cannot currently release iPhone applications which directly perform vision based tracking on the camera’s videostream. Computer vision based AR applications that are currently in the app store rely on analyzing still frames which limits the AR experience. In 2010, hopefully these frustrating API barriers will be overcome, and we can see vision based AR applications proliferate on mobile devices.

-Game Consoles

Augmented reality first came to game consoles when Dr. Richard Marks invented the EyeToy for the Playstation 2. More recently, we have seen the release of AR applications for the Playstation 3 (PS3) with the Eye of Judgment and the Eyepet.  In 2010, Sony is expected to debut their motion controller which will further extend the PS3’s AR capabilites.  Here’s an article where Richard Marks answers questions about the motion controller along with a video showing a precursor running on the Playstation 2.

Microsoft’s heavily publicized Project Natal should provide a set of tools for authoring AR experiences for the Xbox 360.  Hopefully developers will be able to embrace AR and ingeniously incorporate it into their game designs.

-Tablets

Researchers have extensively used tablets to prototype AR applications proving the viability of the tablet as an AR platform.  Most of these demos were created using the last generation of tablets from HP and Toshiba.  Those these devices provided great functionality, they were not widely accepted by consumers.  Here’s a 2004 AR paper from George Klein et. al. which utilized the HP TC1000 tablet.  With its sleek design and form factor, Apple’s forthcoming tablet may have greater success. Apple’s tablet will reportedly be equipped with a camera which could make it another excellent AR platform. The process of porting an iPhone application to the Apple tablet is expected to be a straightforward process which could encourage the release of AR applications with versions for both devices.

Focus on Utility and Fun, not Novelty

After the “wow factor” wears off, there’s a risk that consumers may begin to dismiss AR as a gimmick rather than a technology that provides real value. Developers can prevent this situation from happening by using AR to design applications that focus on utility and fun rather than the novelty of seeing an object pop out of a marker.

A great example of a useful AR tool is the US Postal Service’s box measurement application.

Try it here.

This application turns a cereal box into an interactive game.  It’s a fun gaming experience which simultaneously promotes a new film.

Both of these applications provide unique value helping consumers understand that AR is a technology that is useful beyond just being an attention grabbing feature.

The Road Ahead

For those who have been a part of the AR community since its infancy and those who are just entering the field, these are very exciting times. After decades of research, mass market AR applications are finally viable and can be delivered on a variety of platforms. If developers, investors, analysts, and consumers can develop a real understanding of what AR can and cannot do, the future of AR technology is bright, and I look forward to its evolution as an innovative and inspiring medium for creativity, communication, and commerce.

Further background reading suggestions:

-Thoughts on AR, an excellent November 2009 post on the state of AR from  Dr. Blair MacIntyre

-The Ultimate Display: The seminal 1965 essay envisioning AR from Ivan Sutherland, the creator of Sketchpad, and  the father of computer graphics.

-Sutherland, I. E. 1968. A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I (San Francisco, California, December 09 – 11, 1968): Sutherland’s 1968 paper describing the first AR display.    Download PDF