Conference highlights include keynotes by Bruce Sterling and Will Wright among others.  On Thursday I will be giving a talk titled “Augmented Reality in Music Entertainment: Then and Now.”

You can receive the reduced $195 registration price – $200 off the $395 normal price – by using Discount Code: E195 during registration. This fee includes both conference days, lunch, reception and more.

To get a sense of the richness and depth of content, take a look at the ARE2010 Schedule – it will be THE must-attend AR event of the year: 2 days, 3 tracks, 40 AR companies; with keynotes:

- Bruce Sterling – (The AR Prophet!)
- Will Wright – (Spore and the Sims)
- Jesse Schell – (The Gamepocalypse)
- Blaise Aguera y Arcas – (Microsoft, BIng Maps & AR)
….and 80 other speakers! AR fans from around the world are registering and will be there, will you?

This event will be a unique opportunity to see the “state of play” of the emerging augmented reality industry.  Special events include:

- A press conference for new product announcements by leading AR companies
- Launch pad for new Augmented Reality start ups – a judging panel with VCs and entrepreneurs
- Over 80+ AR speakers over two intense days
- “The Auggies” – competition for the best augmented reality demo
- Art and magic live performances by AR artists

All details, speakers, and registration at www.augmentedrealityevent.com.

Using mobile apps like those from Ustream and Qik, a few are even live streaming the show. Taking note of this behavior, a compelling opportunity exists to use these devices to create opportunities for fans to interact with the on stage artist and become participants in the performance.

One of the most ambitious efforts in this area is being led by techno DJ and producer Richie Hawtin. An established music technology innovator, Hawtin gained attention in recent years for tweeting his DJ sets. As part of his current tour performing as his alter ego, Plastikman, he and his collaborators have released the “SYNK” application for the iPhone and the iPod Touch.

During performances audience members can use the app to see real time data from percussion gear, live video from an on stage perspective, and the application will synchronize phone visuals with activity on the stage LED wall.  More details on the app can be found here.

Here’s video of the “Synkotik” mode of the app.

In this LA Weekly Coachella interview, Richie Hawtin describes the motivation for integrating SYNK into his performance.

“I think a great concert experience is when unexpected things happen and when people come together to experience something that goes beyond music. The idea was to throw something into the mix that would heighten that experience. With the app we’re able to send people messages, we’re able to actually synchronize things that are happening on stage with the iPhone apps, we’re even actually allowing people to actually have access to my set-up, you know, triggering sounds. So everyone talks about technology democratizing everything. With our app we will actually blur the lines between the performer and the audience, actually forcing questions of, who is the performer, who is being in control, It is an experiment.”

In addition to the work of Hawtin and his collaborators, it should be noted that there are other efforts in this area, such as Eyebeam’s Open Video Sync project.  It aims to create an open source codebase which will allow multiple iPhones to synchronously playback video using Bluetooth or Wifi transmitted timecode.

Using smartphones apps for interactive concerts is a concept that is still in its infancy.  It is clearly a powerful avenue for tech savvy artists to push the boundaries of live performance.  Furthermore, by creating a way for audiences to participate in the performance, fans are given an additional incentive to buy tickets, an opportunity which cannot be overlooked in today’s music industry where artists are increasingly dependent upon live show revenues.

In 1997, I attended a brilliant presentation by wearable computing pioneer Dr. Thad Starner that made me aware of how this vision would be realized.  At the time, Thad wore a PC/104 based computer equipped with a “Private Eye” head worn display, a twiddler chorded keyboard, and a CDPD wireless internet connection. With a series of demonstrations, he illustrated the concept of contextually aware computing in which knowledge of location, time, and past user behaviors can be leveraged to better assist a person in completing their tasks. The idea is that through contextual information and a growing body of knowledge of a user’s habits, a computer interface can evolve to fit the user as opposed to the user having to adapt to a static interface. Over time, he described how such an interface could learn enough about an individual to become a “digital doppelganger” which could independently handle a number of one’s routine responsibilities. As an example, he described a scenario in which the time of year is December, and your wearable computer uses its knowledge of your gift buying habits to act on your behalf to complete all of your Christmas shopping for you.

Millions of us now carry contextually aware computing devices in the form of smartphones. Consequently, these devices are a mass market platform for advancing the notion of the digital doppelganger, now commonly referred to as a virtual personal assistant (VPA). Last fall, Clark Dodsworth, pointed me to a new mobile application called Siri which I see as the next step in VPA development. Siri is notable in that it uses a conversational interface that accepts either typed or spoken word input. It goes beyond traditional speech to text systems in that it can utilize natural language processing techniques to derive a person’s intent. Context from location, time, and user history is used to select from a variety of web based services capable of carrying out the requested task. For instance, if I say “I am hungry,” the system first determines that I am looking for restaurants. Next, using the phone’s geo-location capabilities, it determines that I am in Santa Monica, California, and then lists restaurants within walking distance along with details provided from a variety of web services.

This short video of demonstrates Siri’s current features.

Two decades ago, Apple produced a now famous video called “Knowledge Navigator” which depicted a future virtual personal assistant system. In this keynote from the Semantic Technology Conference, Tom Gruber introduces Siri and compares its features to the ideas presented in the Knowledge Navigator video.

It should be noted that beyond making existing information services easier to use, virtual personal assistants like Siri will also play an important role in winning over late adopters who have not been fully utilizing online search and recommendation tools.

Siri is without a doubt a major step in the fulfillment of a vision for computing set forth decades ago, and I look forward to seeing additional innovative VPA systems in the coming years.

In an effort to help manage expectations regarding AR technology, I will briefly describe what works today while clarifying what we can expect in the future.

Augmented Reality Glasses are not Viable in the Near Term

When covering AR, a number of technology pundits have assumed that within the next few years, we can expect head mounted displays or augmented reality “glasses” to become the best display for AR applications. Without a doubt, it would be groundbreaking if a high quality AR display could be built into the form factor of sunglasses. Unfortunately, a lightweight, wide field of view, daylight readable, head mounted display (HMD) at mass market prices is not something we can expect to see in the next five years. I have either bought or used most head mounted displays sold commercially since 1992, and I have seen great strides over the years in HMD resolution, brightness, and power usage.  To illustrate where the technology now stands, here are the best that I have used.

-Fakespace Labs “Wide 5” : Easily, the most immersive head mounted display I have tried, it can be modified for video see through augmented reality applications.

-The “Advanced Helmet Mounted Display”:  Optical Research Associates, a firm that helped fix the Hubble telescope, designed the optics. It has an amazing field of view, it’s optical see through, and has great resolution. It is also very large and is intended for helmet mounting.

-eMagin Z800 3Dvisor: It has a relatively small form factor, it’s very affordable, uses OLED’s, and it can be adapted for video see through augmented reality applications. The field of view is limited.

No one has yet delivered a wide field of view display in a small package that approaches the footprint of sunglasses as the public expects. With significant engineering investments, innovative approaches from companies such as Lumus, ORA, and Digilens may show promise in solving this problem. In addition to optics issues, eye strain and other head mounted display ergonomics problems must be dealt with.  For a detailed overview of the current and future state of head mounted display technology, I suggest the following:

-The Coming Generation of Head-worn Displays (HWDs): Will the Future come to us through new Eyes? Kevin and Jannick Rolland-Thompson, James P. McGuire, and Ozan Cakmakci, Optical Society of America 2009 Annual Meeting, San Jose, CA Download PDF

-The Past, Present, and Future of Head Mounted Display Designs” by Jannick Rolland and Ozan Cakmakci (College of Optics and Photonics: CREOL & FPCE, University of Central Florida) Download PDF

Ultimately, there will be no single “ultimate” AR display. Instead, the type of display used for AR will be determined by the needs of the application. Current and future developers will have a range of options including smart phone screens, handheld tablets, desktop monitors, micro projectors, and AR glasses.

GPS and Compass is not Enough

There are a number of mobile augmented reality applications, namely Layar, Wikitude, and the Yelp Monocle, which use GPS and compass data to overlay graphic information on a live video view of the real world. Though these applications are novel and interesting, the data provided by a mobile device’s GPS and compass is simply not precise enough to deliver a quality AR user experience. Information overlays usually appear to wobble or bounce around the video view. Consumers will rarely use these applications after the novelty factor wears off. At best, these applications provide an alternative viewing mode for data that should be first presented on a 2D map or in a list. Here’s a well written piece from New Scientist commenting on the current state of mobile augmented reality applications: Augmented Reality Gets off to a Wobbly Start

Sensor Fusion is the Answer

The best approach to AR tracking and registration involves hybrid tracking and sensor fusion techniques which use computer vision technology in conjunction with GPS and compass data. This paper provides deeper insight.

-Azuma, Ronald T., Bruce R. Hoff, Howard E. Neely III, Ronald Sarfaty, Michael J. Daily, Gary Bishop, Vern Chi, Greg Welch, Ulrich Neumann, Suya You, Rich Nichols, and Jim Cannon. Making Augmented Reality Work Outdoors Requires Hybrid Tracking. Proceedings of the First International Workshop on Augmented Reality, (San Francisco, CA, 1 November 1998), 219-224. Download PDF

Mobile augmented reality applications using a variety of sensor fusion techniques have been prototyped over the years, and we will hopefully see applications leveraging them on the market soon.

AR Hardware Platforms for Today

Here are five AR hardware platforms that developers can utilize today for creating augmented reality applications.

-Webcams

In the past year, there has been an explosion of ad campaigns which deliver webcam based AR experiences through an internet site or a downloadable application. Most of these utilized the FLARToolkit. I’ve used it to develop a number of AR web demos, available via the links below.

-Augmented Reality Interactive Album Cover

-Augmented Reality Concert

-Augmented Reality Fashion Ad

-Augmented Reality Business Card

-Kiosks, Digital Signage, Window Displays

Webcam AR applications can be packaged as interactive kiosks, digital signs, or window displays installed in retail locations as seen in this excellent example from Lego and this one promoting the movie Coraline.

-3G Smartphones

3G smartphones have enormous potential for AR applications. Phones from manufacturers including Apple, HTC, and Nokia are equipped with a camera, GPS, compass, accelerometers, and 3D graphics capabilities. These devices have enormous potential for AR applications. Unfortunately developers are currently more limited by the restrictions of manufacturer API’s than by the hardware itself. For example, developers cannot currently release iPhone applications which directly perform vision based tracking on the camera’s videostream. Computer vision based AR applications that are currently in the app store rely on analyzing still frames which limits the AR experience. In 2010, hopefully these frustrating API barriers will be overcome, and we can see vision based AR applications proliferate on mobile devices.

-Game Consoles

Augmented reality first came to game consoles when Dr. Richard Marks invented the EyeToy for the Playstation 2. More recently, we have seen the release of AR applications for the Playstation 3 (PS3) with the Eye of Judgment and the Eyepet.  In 2010, Sony is expected to debut their motion controller which will further extend the PS3′s AR capabilites.  Here’s an article where Richard Marks answers questions about the motion controller along with a video showing a precursor running on the Playstation 2.

Microsoft’s heavily publicized Project Natal should provide a set of tools for authoring AR experiences for the Xbox 360.  Hopefully developers will be able to embrace AR and ingeniously incorporate it into their game designs.

-Tablets

Researchers have extensively used tablets to prototype AR applications proving the viability of the tablet as an AR platform.  Most of these demos were created using the last generation of tablets from HP and Toshiba.  Those these devices provided great functionality, they were not widely accepted by consumers.  Here’s a 2004 AR paper from George Klein et. al. which utilized the HP TC1000 tablet.  With its sleek design and form factor, Apple’s forthcoming tablet may have greater success. Apple’s tablet will reportedly be equipped with a camera which could make it another excellent AR platform. The process of porting an iPhone application to the Apple tablet is expected to be a straightforward process which could encourage the release of AR applications with versions for both devices.

Focus on Utility and Fun, not Novelty

After the “wow factor” wears off, there’s a risk that consumers may begin to dismiss AR as a gimmick rather than a technology that provides real value. Developers can prevent this situation from happening by using AR to design applications that focus on utility and fun rather than the novelty of seeing an object pop out of a marker.

A great example of a useful AR tool is the US Postal Service’s box measurement application.

Try it here.

This application turns a cereal box into an interactive game.  It’s a fun gaming experience which simultaneously promotes a new film.

Both of these applications provide unique value helping consumers understand that AR is a technology that is useful beyond just being an attention grabbing feature.

The Road Ahead

For those who have been a part of the AR community since its infancy and those who are just entering the field, these are very exciting times. After decades of research, mass market AR applications are finally viable and can be delivered on a variety of platforms. If developers, investors, analysts, and consumers can develop a real understanding of what AR can and cannot do, the future of AR technology is bright, and I look forward to its evolution as an innovative and inspiring medium for creativity, communication, and commerce.

Further background reading suggestions:

-Thoughts on AR, an excellent November 2009 post on the state of AR from  Dr. Blair MacIntyre

-The Ultimate Display: The seminal 1965 essay envisioning AR from Ivan Sutherland, the creator of Sketchpad, and  the father of computer graphics.

-Sutherland, I. E. 1968. A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I (San Francisco, California, December 09 – 11, 1968): Sutherland’s 1968 paper describing the first AR display.    Download PDF

Now that it’s possible, what will be the breakthrough applications for mobile live broadcasting?   I think the answer may lie in looking at the trend of celebrities using Twitter.  Twitter is popular with stars because it is simple and easy to maintain.  It can be almost spontaneously updated unlike a traditional blog or personal website.  Celebrities can easily enhance this fan communication channel using Ustream.tv, Qik, Kyte, or Flixwagon.  In particular, mobile broadcasting could be appealing to touring musicians who rarely have an opportunity to sit at a computer and send a well composed personal blog post. Rapper Soulja Boy has been an early adopter in this area by using Kyte’s mobile platform to keep his fans in the loop.  In a similar fashion, Lil Wayne is set to begin using Ustream.tv’s mobile application.  Mobile broadcasting is clearly a concept to keep an eye on over the next year.

Here’s Ustream.tv’s mobile demo video.

Microsoft Tag has recently become publicly available at http://tag.microsoft.com/.  Visit http://gettag.mobi/ to obtain a reader for most camera equipped smart phones.  Potential applications for media distribution, promotion, and advertising are vast.  For example, a band could enhance their show flyers with tags linking to online ticket offers and guest lists.  A couple of years ago I experimented with Semacode, a similar technology, to develop demos showing how tags could be used to distribute music videos and film trailers.

Demo 1: Music Video Distribution With Visual Tags

Demo 2: Film Trailer Distribution With Visual Tags