In 1997, I attended a brilliant presentation by wearable computing pioneer Dr. Thad Starner that made me aware of how this vision would be realized.  At the time, Thad wore a PC/104 based computer equipped with a “Private Eye” head worn display, a twiddler chorded keyboard, and a CDPD wireless internet connection. With a series of demonstrations, he illustrated the concept of contextually aware computing in which knowledge of location, time, and past user behaviors can be leveraged to better assist a person in completing their tasks. The idea is that through contextual information and a growing body of knowledge of a user’s habits, a computer interface can evolve to fit the user as opposed to the user having to adapt to a static interface. Over time, he described how such an interface could learn enough about an individual to become a “digital doppelganger” which could independently handle a number of one’s routine responsibilities. As an example, he described a scenario in which the time of year is December, and your wearable computer uses its knowledge of your gift buying habits to act on your behalf to complete all of your Christmas shopping for you.

Millions of us now carry contextually aware computing devices in the form of smartphones. Consequently, these devices are a mass market platform for advancing the notion of the digital doppelganger, now commonly referred to as a virtual personal assistant (VPA). Last fall, Clark Dodsworth, pointed me to a new mobile application called Siri which I see as the next step in VPA development. Siri is notable in that it uses a conversational interface that accepts either typed or spoken word input. It goes beyond traditional speech to text systems in that it can utilize natural language processing techniques to derive a person’s intent. Context from location, time, and user history is used to select from a variety of web based services capable of carrying out the requested task. For instance, if I say “I am hungry,” the system first determines that I am looking for restaurants. Next, using the phone’s geo-location capabilities, it determines that I am in Santa Monica, California, and then lists restaurants within walking distance along with details provided from a variety of web services.

This short video of demonstrates Siri’s current features.

Two decades ago, Apple produced a now famous video called “Knowledge Navigator” which depicted a future virtual personal assistant system. In this keynote from the Semantic Technology Conference, Tom Gruber introduces Siri and compares its features to the ideas presented in the Knowledge Navigator video.

It should be noted that beyond making existing information services easier to use, virtual personal assistants like Siri will also play an important role in winning over late adopters who have not been fully utilizing online search and recommendation tools.

Siri is without a doubt a major step in the fulfillment of a vision for computing set forth decades ago, and I look forward to seeing additional innovative VPA systems in the coming years.

In an effort to help manage expectations regarding AR technology, I will briefly describe what works today while clarifying what we can expect in the future.

Augmented Reality Glasses are not Viable in the Near Term

When covering AR, a number of technology pundits have assumed that within the next few years, we can expect head mounted displays or augmented reality “glasses” to become the best display for AR applications. Without a doubt, it would be groundbreaking if a high quality AR display could be built into the form factor of sunglasses. Unfortunately, a lightweight, wide field of view, daylight readable, head mounted display (HMD) at mass market prices is not something we can expect to see in the next five years. I have either bought or used most head mounted displays sold commercially since 1992, and I have seen great strides over the years in HMD resolution, brightness, and power usage.  To illustrate where the technology now stands, here are the best that I have used.

-Fakespace Labs “Wide 5” : Easily, the most immersive head mounted display I have tried, it can be modified for video see through augmented reality applications.

-The “Advanced Helmet Mounted Display”:  Optical Research Associates, a firm that helped fix the Hubble telescope, designed the optics. It has an amazing field of view, it’s optical see through, and has great resolution. It is also very large and is intended for helmet mounting.

-eMagin Z800 3Dvisor: It has a relatively small form factor, it’s very affordable, uses OLED’s, and it can be adapted for video see through augmented reality applications. The field of view is limited.

No one has yet delivered a wide field of view display in a small package that approaches the footprint of sunglasses as the public expects. With significant engineering investments, innovative approaches from companies such as Lumus, ORA, and Digilens may show promise in solving this problem. In addition to optics issues, eye strain and other head mounted display ergonomics problems must be dealt with.  For a detailed overview of the current and future state of head mounted display technology, I suggest the following:

-The Coming Generation of Head-worn Displays (HWDs): Will the Future come to us through new Eyes? Kevin and Jannick Rolland-Thompson, James P. McGuire, and Ozan Cakmakci, Optical Society of America 2009 Annual Meeting, San Jose, CA Download PDF

-The Past, Present, and Future of Head Mounted Display Designs” by Jannick Rolland and Ozan Cakmakci (College of Optics and Photonics: CREOL & FPCE, University of Central Florida) Download PDF

Ultimately, there will be no single “ultimate” AR display. Instead, the type of display used for AR will be determined by the needs of the application. Current and future developers will have a range of options including smart phone screens, handheld tablets, desktop monitors, micro projectors, and AR glasses.

GPS and Compass is not Enough

There are a number of mobile augmented reality applications, namely Layar, Wikitude, and the Yelp Monocle, which use GPS and compass data to overlay graphic information on a live video view of the real world. Though these applications are novel and interesting, the data provided by a mobile device’s GPS and compass is simply not precise enough to deliver a quality AR user experience. Information overlays usually appear to wobble or bounce around the video view. Consumers will rarely use these applications after the novelty factor wears off. At best, these applications provide an alternative viewing mode for data that should be first presented on a 2D map or in a list. Here’s a well written piece from New Scientist commenting on the current state of mobile augmented reality applications: Augmented Reality Gets off to a Wobbly Start

Sensor Fusion is the Answer

The best approach to AR tracking and registration involves hybrid tracking and sensor fusion techniques which use computer vision technology in conjunction with GPS and compass data. This paper provides deeper insight.

-Azuma, Ronald T., Bruce R. Hoff, Howard E. Neely III, Ronald Sarfaty, Michael J. Daily, Gary Bishop, Vern Chi, Greg Welch, Ulrich Neumann, Suya You, Rich Nichols, and Jim Cannon. Making Augmented Reality Work Outdoors Requires Hybrid Tracking. Proceedings of the First International Workshop on Augmented Reality, (San Francisco, CA, 1 November 1998), 219-224. Download PDF

Mobile augmented reality applications using a variety of sensor fusion techniques have been prototyped over the years, and we will hopefully see applications leveraging them on the market soon.

AR Hardware Platforms for Today

Here are five AR hardware platforms that developers can utilize today for creating augmented reality applications.

-Webcams

In the past year, there has been an explosion of ad campaigns which deliver webcam based AR experiences through an internet site or a downloadable application. Most of these utilized the FLARToolkit. I’ve used it to develop a number of AR web demos, available via the links below.

-Augmented Reality Interactive Album Cover

-Augmented Reality Concert

-Augmented Reality Fashion Ad

-Augmented Reality Business Card

-Kiosks, Digital Signage, Window Displays

Webcam AR applications can be packaged as interactive kiosks, digital signs, or window displays installed in retail locations as seen in this excellent example from Lego and this one promoting the movie Coraline.

-3G Smartphones

3G smartphones have enormous potential for AR applications. Phones from manufacturers including Apple, HTC, and Nokia are equipped with a camera, GPS, compass, accelerometers, and 3D graphics capabilities. These devices have enormous potential for AR applications. Unfortunately developers are currently more limited by the restrictions of manufacturer API’s than by the hardware itself. For example, developers cannot currently release iPhone applications which directly perform vision based tracking on the camera’s videostream. Computer vision based AR applications that are currently in the app store rely on analyzing still frames which limits the AR experience. In 2010, hopefully these frustrating API barriers will be overcome, and we can see vision based AR applications proliferate on mobile devices.

-Game Consoles

Augmented reality first came to game consoles when Dr. Richard Marks invented the EyeToy for the Playstation 2. More recently, we have seen the release of AR applications for the Playstation 3 (PS3) with the Eye of Judgment and the Eyepet.  In 2010, Sony is expected to debut their motion controller which will further extend the PS3’s AR capabilites.  Here’s an article where Richard Marks answers questions about the motion controller along with a video showing a precursor running on the Playstation 2.

Microsoft’s heavily publicized Project Natal should provide a set of tools for authoring AR experiences for the Xbox 360.  Hopefully developers will be able to embrace AR and ingeniously incorporate it into their game designs.

-Tablets

Researchers have extensively used tablets to prototype AR applications proving the viability of the tablet as an AR platform.  Most of these demos were created using the last generation of tablets from HP and Toshiba.  Those these devices provided great functionality, they were not widely accepted by consumers.  Here’s a 2004 AR paper from George Klein et. al. which utilized the HP TC1000 tablet.  With its sleek design and form factor, Apple’s forthcoming tablet may have greater success. Apple’s tablet will reportedly be equipped with a camera which could make it another excellent AR platform. The process of porting an iPhone application to the Apple tablet is expected to be a straightforward process which could encourage the release of AR applications with versions for both devices.

Focus on Utility and Fun, not Novelty

After the “wow factor” wears off, there’s a risk that consumers may begin to dismiss AR as a gimmick rather than a technology that provides real value. Developers can prevent this situation from happening by using AR to design applications that focus on utility and fun rather than the novelty of seeing an object pop out of a marker.

A great example of a useful AR tool is the US Postal Service’s box measurement application.

Try it here.

This application turns a cereal box into an interactive game.  It’s a fun gaming experience which simultaneously promotes a new film.

Both of these applications provide unique value helping consumers understand that AR is a technology that is useful beyond just being an attention grabbing feature.

The Road Ahead

For those who have been a part of the AR community since its infancy and those who are just entering the field, these are very exciting times. After decades of research, mass market AR applications are finally viable and can be delivered on a variety of platforms. If developers, investors, analysts, and consumers can develop a real understanding of what AR can and cannot do, the future of AR technology is bright, and I look forward to its evolution as an innovative and inspiring medium for creativity, communication, and commerce.

Further background reading suggestions:

-Thoughts on AR, an excellent November 2009 post on the state of AR from  Dr. Blair MacIntyre

-The Ultimate Display: The seminal 1965 essay envisioning AR from Ivan Sutherland, the creator of Sketchpad, and  the father of computer graphics.

-Sutherland, I. E. 1968. A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I (San Francisco, California, December 09 – 11, 1968): Sutherland’s 1968 paper describing the first AR display.    Download PDF

Researchers at the University of California San Diego Computer Audition Lab have developed a system that could address this problem by allowing people to find music using descriptive words rather than artist names and song titles.  For instance, a person could enter the words “high energy guitars” or “romantic vocals” and then receive a list of tracks that match that description.

The USCD system is capable of ingesting songs and automatically tagging them with annotation data without human intervention. To provide accurate results, the system must first be taught to hear music and describe it using natural language.  The training process uses digital signal processing and machine learning algorithms to expose the system to a broad array of music along with the words people use to describe it.  For example, to be able to accurately identify music that is referred to as “driving rock”, the system must analyze a large number of driving rock songs and then identify signal patterns that make that particular style of song unique.

The researchers have been gathering training data through crowdsourcing using an innovative Facebook game called “Herd-It“.  In this game, users are played a song snippet and asked to associate descriptive words and phrases with it.  Users earn points based on how well their answers match those of previous players.  Here’s a video describing the game.

The research group’s latest work in improving automatic music analysis was recently presented at the 2009 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) in the paper “Dynamic Texture Models of Music,” by Luke Barrington, Antoni Chan, and Gert Lanckriet.

With the continuing decline of the radio DJ as taste maker, web based music search and discovery tools will become increasingly important. With further development, machine learning driven music search systems such as this one could provide an intuitive and compelling method for listeners to find music they will enjoy.

Here are video captures of the performances.

This promotion is an excellent example of using innovative technology and engaging content to capture audience attention. Simultaneously, it provides a unique avenue for artists to promote their music and live shows.

Augmented reality (AR) is a technology that has been around for quite a while, primarily in the academic and research domains. Recently, its mainstream presence has increased due to the development of the Flash version of the Augmented Reality Toolkit (FLARToolkit). The ARToolkit was developed by Dr. Hirokazu Kato with Dr. Mark Billinghurst at the University of Washington’s Human Interface Technology Lab (HITLAB) over ten years ago.

In 2000-2001, I led a team which modified the original C based ARToolkit to work on 3D accelerated desktop Windows PC’s. We used the toolkit to develop interactive augmented reality projections for the band Duran Duran’s live concert tour. Here’s a video of ARToolkit effects that we used in the live shows.

The project was documented in this presentation at the 2002 Augmented Reality Toolkit Workshop:

Jarrell Pair, Jeff Wilson, Jeff Chastine, Maribeth Gandy. “The Duran Duran Project: The Augmented Reality Toolkit in Live Performance”. The First IEEE International Augmented Reality Toolkit Workshop, 2002. Download PDF

Now that it’s possible, what will be the breakthrough applications for mobile live broadcasting?   I think the answer may lie in looking at the trend of celebrities using Twitter.  Twitter is popular with stars because it is simple and easy to maintain.  It can be almost spontaneously updated unlike a traditional blog or personal website.  Celebrities can easily enhance this fan communication channel using Ustream.tv, Qik, Kyte, or Flixwagon.  In particular, mobile broadcasting could be appealing to touring musicians who rarely have an opportunity to sit at a computer and send a well composed personal blog post. Rapper Soulja Boy has been an early adopter in this area by using Kyte’s mobile platform to keep his fans in the loop.  In a similar fashion, Lil Wayne is set to begin using Ustream.tv’s mobile application.  Mobile broadcasting is clearly a concept to keep an eye on over the next year.

Here’s Ustream.tv’s mobile demo video.

Microsoft Tag has recently become publicly available at http://tag.microsoft.com/.  Visit http://gettag.mobi/ to obtain a reader for most camera equipped smart phones.  Potential applications for media distribution, promotion, and advertising are vast.  For example, a band could enhance their show flyers with tags linking to online ticket offers and guest lists.  A couple of years ago I experimented with Semacode, a similar technology, to develop demos showing how tags could be used to distribute music videos and film trailers.

Demo 1: Music Video Distribution With Visual Tags

Demo 2: Film Trailer Distribution With Visual Tags

The second is Zunavision, a spin off from the Stanford Artificial Intelligence Lab.  Zunavision’s technology allows advertisters to place ads onto the sides of buildings, walls, and other surfaces.  I was very impressed by its ease of use.

Zunavision Demo Video

We are now faced with new challenges.  An astounding set of powerful media devices, platforms, and services have become available which present new possibilities for how we interact with media.  At the same time, questions regarding monetization and consumer adoption of these new capabilities remain largely unanswered.

Laboratory4 will discuss and track interactive media technology trends as we travel the road to define the next stage of digital media convergence, one will that will likely focus less on the delivery of media, but more upon how we interact with it.

Topics will span a variety of areas related to advanced interactive media technologies including:

Mobile Platforms and Services: There are well over 3 billion mobile phones available for use in the world today.  Many of them are powerful, internet connected multimedia computers. How can these nearly ubiquitous computing devices be leveraged to create new ways of interacting with and experiencing media?

The New Music Industry: Physical music sales are nearly dead.  An enormous amount of music content can be found free online.  Blogs and social networks are replacing the radio DJ as tastemaker.  What technologies and services will further evolve how music is experienced and created?

The Transformation of Film and Television: The movie and tv industry is going through the same painful process of change that the music industry was thrust into a decade ago.  What role will new technologies play as this business struggles to adapt?

Emerging Technologies: Augmented reality, online virtual worlds, 3D displays, micro projectors, perceptual user interfaces, and other advanced technologies have enormous potential.  What role will they play?

Business Models: How can new and existing technologies and services be packaged to provide sustainable revenue streams?

The journey ahead is complex and risky.  For those of us inspired to embark upon it, opportunities for true innovation await.

Onward,

Jarrell Pair

CTO
LP33.tv