Wearable SDKs Overview at AWE 2014

At this year’s Augmented World Expo, I gave a talk on the currently available wearables and wearable SDKs. This talk expands upon my talk from last year on Mobile AR SDK’s.

This year was a truly exciting one for the augmented world, as we finally saw the release of several wearables in the digital eyewear / smart glasses space, including stereo see-through glasses such as the Moverio BT-200 from Epson. The sessions this year were top-notch, the expo hall was brimming with awesome new hardware and software, and it was a real pleasure to be there presenting again alongside all of the other excellent speakers.

In the talk, I outlined what capabilities each device has, what the vendor SDK provides, and what 3rd-party SDKs are available for each. I also tried to point out the current limitations of each of the devices or the wearable SDKs available to develop for them.

Wearables Covered

Wearable SDKs Covered




Mobile AR SDK Tutorial at AWE 2013

I gave a talk this year at the Augmented World Expo, comparing and contrasting the current best-in-breed Mobile AR SDK’s, and providing a behind-the-curtains look at how to use them. The show was truly awe-some, and it was a privilege to be given the opportunity to speak. Here are the slides and video from the talk:


Patched Reality interviewed on AR Dirt Podcast Episode 16

If you haven’t had the pleasure of listening to Joseph Rampola’s AR Dirt Podcast, you’re really missing out. Coming from a cyber crimes law enforcement background, Joseph brings a unique perspective to augmented reality’s potential dark side, but he is also a tireless advocate for the positive promise of AR. The podcast always features the latest AR news, discussions on what the future holds, and interviews with leaders in the augmented reality space.

You can listen to me (Patrick O’Shaughnessey) interviewed here: AR Dirt Episode 16.

Pepsi Max Your World

Mobile Augmented Reality

Augmented reality delivered through a smartphone or tablet is the most accessible form of AR available today. Patched Reality has delivered AR experiences and games to iPhone, iPad, and Android phones and tablets. To experience AR on their mobile device, a user downloads and installs an existing AR browser such as junaio, or a custom developed augmented reality app. Existing apps can also be enhanced with AR features.

A mobile AR app presents a view of the world through the device’s video camera, and draws pertinent overlays over top. These overlays can be positioned based upon GPS location, over printed materials such as magazines, catalogs, or product labels, or even over the user’s face. Newer technologies such as SLAM (Simultaneous Localization and Mapping) can even place augmented content out in the world without requiring any printed material at all. The augmented content presented is limited only by the imagination, and can be in the form of text, images, video, sounds, and/or 3D animation. Rich interactive experiences such as games can also be produced to further engage the user. A key component of many AR apps lets users capture their AR experience (e.g. a screenshot) to share with friends through social media sites like Facebook and Twitter.

Screenshot of The Getty's Augsburg Cabinet Augmented Reality Experience.

Webcam Augmented Reality

Webcam AR, also known as Magic Mirror AR, is ideal for experiences that take place at a home computer, an in-store kiosk, or a trade show booth. Patched Reality has developed a large number of webcam AR experiences and games. These applications make use of a traditional webcam or a 3D-sensing cam such as a Kinect Camera  or Creative Interactive Gesture Camera, attached to a desktop computer. The experience can be delivered either through a web page using Adobe Flash, or through a native PC or Mac application. As with mobile AR, the types of augmentable objects and content displayed to the user are quite flexible. With webcam AR, one can also augment the viewer, such as their face/head, hands, or whole body.

Augmented Space Exploration

Augmented Space Exploration Exhibit Opens November 19th, 2011

If you’re going to be in New York between November 19th and August 12th, download the Beyond Planet Earth Augmented Reality app developed by Patched Reality onto your iPhone, iPod Touch, or iPad 2, and stop by a certain museum that specializes in natural history (Map). The exhibit itself is stunning – Gizmodo calls it a “Can’t miss”. The curators and museum staff have really outdone themselves.

Beyond Planet Earth AR on iTunes
YouTube video preview

If you can’t make it to the museum, you can still have fun at home with the web version.

In all, there are 11 augmentations that you can collect. Taking photos is encouraged – so go ahead and take a picture of yourself with the Curiosity Mars rover or a lunar space elevator, and share it with your friends on Facebook or Twitter. If you stop by, I hope you enjoy the show and learn a lot in the process!

Ben and Jerry's Moo Vision

Ben and Jerry’s Moo Vision Markerless Augmented Reality for iPhone

Ben and Jerry's "Moo Vision" Augmented Reality for iPhone.

Chunk Generosity Diorama revealed with Moo Vision

Ben & Jerry’s “Moo Vision” has now been available for two weeks (via iTunes), and its been great to see the reaction.  Here are some quotes, movie poster style:

“a must try for ice cream fans” – Augmented Planet

“Ben & Jerry’s iPhone App Gets Even More Psychedelic with Augmented Reality” – Mashable

“Ben & Jerry’s is now the first major brand to sport natural feature tracking on the iPhone” – ReadWriteWeb

I was thrilled to be involved in developing the first 3rd-party iPhone app to use matio’s Unifeye Mobile SDK, and working with Circ.us, Edelman, and metaio was fantastic.  But even better is knowing that we’ve only scratched the surface of what’s possible, and I can’t wait to see what else comes along and to get involved in making it happen.

The project had some interesting challenges.  The Unifeye SDK performs natural feature tracking (NFT), which means that it is able to track 2D images that aren’t limited to black & white blocky markers.  In the ideal case when using NFT, you’ll have control over the target image to be recognized and tracked, so that you can optimize the image for track-ability by the vision system.  For this app, the target images were Ben & Jerry’s pint lids, so we very well couldn’t ask them to reprint them!  Also, pint lids are circular and AR libraries generally tend to deal with rectangular images. Therefore, the tracking configuration went through several iterations to get a result that tracked well under various lighting conditions and camera angles. Major kudos to Stefan Misslinger at metaio and his engineering team in Germany for their fantastic support and tweaks to the SDK.

Here’s a video of the app in action (with editing by the multi-talented Paul Jannicola):

Will Wright tries out the Five Live of Criss Angel Game developed by Patched Reality.

Five Lives of Criss Angel Augmented Reality Experience Launched

This was a fun project to put together. It’s somewhat unique in the increasingly crowded field of FLARToolkit-based marketing applications out there, as it makes use of the pan, tilt, and roll of the marker as inputs to a game.  For this project, I decided to give the FLARManager a try, mostly for its pretty decent smoothing algorithm.  The game wouldn’t be much fun to play if the playing field was jumping around sporatically.

The game takes a bit of concentration to play, since the game piece always levitates upward (like Criss Angel, get it?), instead of what one’s sense of gravity would expect.  But once you get the hang of it, some of the trickier levels towards the end are a lot of fun to play (my favorite is level four).

I programmed the gameplay, 3D rendering, and Facebook/high score integration, and Matt Dominianni provided the level design and programmed many of the UI screens.  The application was produced by Circ.us for the A&E Television Network.

You can play the game at the official Criss Angel site here: Five Lives of Criss Angel Augmented Reality Experience.

Patched Reality FLAR Framework

Patched Reality FLAR Framework v0.1 release

Today I’m announcing the first public availability of an application framework I’ve been developing that complements FLARToolkit and FLARManager, and makes it possible to rapidly produce Flash augmented reality applications.

This past year I spent many many hours with FLARToolkit and  Eric Socolofsky’s excellent FLARManager.  FLARManager makes the overall FLARToolkit configuration much simpler, especially if you want to mirror the camera or use multiple markers, and has some nice additional features like marker smoothing.

However, after two or three projects I found myself spending a lot of time writing very similar 3D asset loading and manipulation code, which FLARManager wasn’t really created to address.  This is where the framework I’m announcing today comes in.  A simple “show an animated 3D model on a marker” application can be written in about 5 minutes with about 10 lines of mxml code.  Of course, much more complex applications are possible, such as the first application to use it, which lets you explore a 17th century “Cabinet of Curiosities”  at The Getty Museum web site: http://bit.ly/GettyAugsburgAR.

Alternative content 

The key features of the framework are:

  • Clean separation between tracking libraries and rendering engines, allowing for easy mix and match
  • Configure 3D scenes with a simple XML file and specify scale, position, and rotation
  • Group 3D models and assign them to different AR markers
  • Configure Collada animation clips (especially useful for Maya-exported models)
  • Easily specify marker detection/removal behavior – fade out, fade in out, stay visible, or disappear immediately.
  • Pluggable and extensible scene renderers – currenlty Papervision3D, Away3D, and Away3DLite.
  • Integration with FLARManager and flare*NFT (commercial only)
  • Sound manager for configuring and playing sounds
  • Extend all XML configuration items with application-specific data

The framework is dual licensed in the same manner as FLARToolkit – it is licensed under GPL for non-commercial projects, or projects where you are willing to share your source code.  For use in projects where you do not want to share your source code, a commercial license is available. Send commercial inquiries to:


The framework doesn’t have an official name yet, but I wanted to be able to share it with fellow ARE2010 attendees.  You can check out the framework here:

svn co http://projects.patchedreality.com/FLARFramework/releases/v0.1 PatchedRealityFLARFramework

Or browse the source at:


There is a simple example application in the “examples/Simple” directory.  More information and documentation to come.

If you’re going to be at the ARE2010conference, look me up and I’d be happy to give you a demonstration. I hope you’ll download it and give it a try, and enjoy spending more of your time on application/game logic than boilerplate code.  And if you do, please share your experiences.

Parrot unveils AR.Drone iPhone-controlled quadricopter

Okay.  Here is something I didn’t predict would come out in 2010 – a carbon fiber quadricopter AR input device and gaming platform for the iPhone.  Why wait for Apple to open up their camera API, when you can slap one onto a self-stabilizing 4-propeller geeks’ dream toy?  The clever folks at Parrot and int13 have developed the “AR.Drone”, a remote-controlled copter with two on-board cameras (one for flight stabilization, and one for web cam broadcast and AR gaming).  The AR software is capable of detecting other drones in flight or markers on the ground, making it possible to play both solo games and multi-player games against freinds’ AR.Drones.  Parrot has also opened up their SDK to developers to create their own games for the AR.Drone.   Developer prototype kits can be had for $1200, and start shipping in February.

IMHO, this is the most exciting development in AR games to come along to-date, and I can’t wait to get my hands on one (or two).  It combines nearly every one of my childhood and adult toy loves – remote control, robotics, flying, mobile gaming, and augmented reality – into a single package.  Kudos to Parrot and int13 for a truly innovative product.

If the rest of 2010 has more surprises in store of the likes of this, it is truly shaping up to be an amazing year for augmented reality.  What do you think?  Am I overstating the awesomeness of this new development, or do you share my breathless enthusiasm?