Was going to call this Augmented Reality Story Environments but…:)

It is fascinating to see how quickly Augmented Reality (AR) is permeating our lives and the blogosphere. But what will the mass adoption of mobile devices that allow you to layer ‘virtual story worlds’ over the real world mean for new forms of entertainment & marketing? Also what will it mean when celebrities and audience/users, begin to merge – avatars appearing in broadcast TV and film/gamestars composited into our homes?


I have posted about the cross-reality evolution over the last 3 years on this blog under a general mixed-reality umbrella. Now we have every blogger & journalist talking about their AR engaged iPhone, DSi, PSP or smart mobile as if they have discovered some advanced alien technology. But is it really is a game changer, a new playground for storytellers? A window to another world at one end through to a simple layered utility at the other. Actors and fantasy characters deliver lines, embedded in real world scenes, you find the hidden virtual treasure, the historical or future backstories and clues, video, sound, images – even fellow ‘players’ morph into strange aliens or dissapear, you leave red herrings or leave help for other players the possiblities, endless.

Most of the applications reported on at the moment are of the ‘make-money’ utility or contextual geo-advertising variety eg: Mashables Top 6 AR Mobile Apps or Mobile phones get cyborg vision from BBC but I am more interested in how storytellers (writers, filmmakers, tv producers etc) can use AR to deliver new kinds of transmedia entertainment – something new we really need at the moment.

transocialmediaplay1In my day job at LAMP there have been several ‘inclusions’ of AR in quite a few projects. Over the past year I have also been in an alternative Experience Design Lecturer role at AFTRS and at foundation & graduate diploma, many ‘film’ students have realised the story potential of AR and come up with interesting, narrative driven applications.I have also created a few Alternate Reality ‘team bonding’ Games that have involved game and real world space such as the Marysville ‘Old Forest’ one last year and wonderful to see companies like GoGame offering the commercial creation of these types of experiences for corporates.

Sadly though there are not many good examples yet in more popularist commercial media of real time, augmented reality storytelling. Admittedly there are more filmmakers using game engines and combining them with live action actors to do well…the same stuff, films. Where are the original new forms?

I repeat my simple transocialmedia triangle here to draw attention to the bottom left corner. The place where game/virtual worlds meet the real world creating either a) real world being ‘composited’ into game space or b) game and virtual elements being layered (using an intermediary screen & software) over the real world. Popular fiction has drawn on the possibilities of this blended reality in many a film. Perhaps the most memorable in terms of its marketing potential but also privacy infringing hell is what happens when the ‘street’ can recognise us and send ‘personalized’ messages back. A short clip from minority report film.

I recall watching that a few years back thinking well another few decades and we will be there but we are now months away from seeing similar installations in various malls around US and Europe! Another aspect which will have many running for the hills over the coming months will be facial recognition and social geolinking – consider the last 15 seconds of this already infamous video

About to be released will be applications that cross-link your face (identified through live scanning in the street) with your many online profiles/data and something much simpler find your location based on your real time conversation online as in the next clip. The opportunity for story creation, to have ‘placed’ actors in urban areas, who are meant to be scanned so associated ‘story’ info can be collected, will be fascinating as will tracking down those live ‘tweeting’ – everyone becomes the forensic, PI.

But back to story and narrative. A couple of years ago I talked about the possibilities once we can get audience driven game-like avatars appearing in live TV shows – a match made in heaven. They are both in real time and can easily integrate the chat or back channel combined with slightly anonymous identity and a real connection for those lucky to be in the show. BOXi below was an early pioneer here.

BOXi is an interactive quiz show with impressive 3D animations. It is primarily targeted to teenagers, and it includes studio teams from different schools. Viewers and studio teams answer general knowledge quiz questions, and viewers can win prizes in each episode. The show is hosted by a live presenter. The teams compete in a live studio environment, while home contestants send answers as SMS messages. The game of home players is set in a 3D animated virtual auditorium, in which each player is represented by a virtual avatar.

BOXi certainly fired a few peoples imaginations and perhaps the producers of Sleuths, which is taking a similar idea to a mainstream US audience, may have drawn on BOXi? Sleuths may open the ‘new TV form’ floodgates and here is more information on the show from their press release

Sleuths will be the first television show in the United States to let audiences become part of the storyline. Sleuths is a half-hour animated episodic series featuring four kids solving a mystery every week. In the show, the audiences register and customize their own avatar which will appear on the screen representing them while the show airs live on national television. Three times per episode, the avatars of the registered audience members will appear in the show for voting sessions. A question will be asked in each session and they will have to text in their vote within a limited time frame. Those who get it wrong will be eliminated from the screen; those who get the question right will be congratulated on the correct answer and stay for the next question. At the end of each episode, the top five avatars who answered all of the questions correctly will appear on screen one final time standing with the main characters. All audience members will have a shot as seeing their avatar standing next to the show’s stars.

Along with the engaging and entertaining show comes a robust online community and additional revenue from premium SMS text messages that bring the audiences’ avatars live on screen. The show was created by Max Benator, Senior Vice President of Multi- Platform Entertainment for RDF USA. RDF will produce the series. “It’s both a television show and an interactive experience — it’s revolutionary and we couldn’t be more excited to introduce this to the States,” said Max Benator, Senior Vice President of Multi-Platform Entertainment for RDF USA.

Chris Harnick from mobile marketer has a good walk-through of the format of the Sleuths format and interestingly talks about scale – 100 000 audience members live in the studio?!.

While in the studio, a real-time joystick control allows a studio operator to create a camera path flying through the 3D scene of all participating viewers’ avatars. MoPA-TV can animate over 100,000 participants in real time.

Of course good old fashioned augmented reality is often seen as best delivered to the personal vs social screen. The many apps about to hit the small screen as I have mentioned will be utility focused, find a location, get more info on a person or place, or play a simple first person shooter. A few though start to take us down a different road – if you will forgive the pun. Ghostwire (similar to a LAMP project called Portable Ghosts) is in prototype and this video gives a real sense of how story could be delivered via AR and told compellingly in location. We can now see what types of games may be appearing in XMas stockings this year! www.ghostwiregame.com

Ghostwire is a collection- and adventure game, where you use your handheld device to communicate with ghosts. Your portable device is used as a portal to the astral plane, and helps you find and collect ghosts that exist all around you. In your quest you use real tools, such as the built-in camera and microphone, as well as abstract ones like an “EMF tuner” for tracking down the ghosts. Once you find the spirits, you document them and find out why they haunt our world. They also give you riddles to solve. Ultimately, you will help them find peace.

There are thousands of Augmented reality story games that Ghostwire and others of its ilk will inspire in the next 18 months or so, others focused on deeper story and physical exploration and of course a killer app for the tourist industry who can ‘direct’ new visitors around locations using narrative vs just ‘tourist spots’ as in this application, the Layar Reality Browser.

Invizimals hosted on another handheld games console the PSP, takes the social play angle a bit further and gets AR players fighting each other around hotspots and cleverly creating environmental effects with Wii-like gestures (see second video after the Joystik one). Interesting to see famed ‘serious’ thesbian Brian Blessed part of the ‘story world’ in this teaser.

Kotaku also has a nice spin on the significance of these AR games

Invizimals adds to the mini-monster collect-and-evolve genre with some technically impressive augmented reality tricks, something along the lines of Sony’s PlayStation Eye powered Eye of Judgment. Invizimals turns the PSP and camera attachment into something like a Geiger counter, or for the sake of our analogy, the PKE meter used in Ghostbusters. Players will need to discover Invizimals in the real world, using the camera to root out “creatures of light,” according to Novarama’s Daniel Sanchez-Crespo. The game’s fiction says that the PSP Camera has been discovered by a Sony engineer in Tokyo to be more sensitive than the human eye, able to detect invisible creatures lurking on real-world objects. Using the PSP and a specially designed disk, Invizimal hunters will have to locate and trap 100-plus hidden creatures, adding them to their collection. Creatures will be spawned based on environmental conditions, with a bit of randomness, to keep things interesting. The trapping of those creatures is vaguely Ghostbusters-esque: find the Invizimal with the on-screen sensor, throw down the trap, then slap the beast to stun it. Very tactile, very cool. And potentially very dangerous, should a stray Invizimal spawn on some poor grandmother’s china cabinet.

For me the new EyePet for PS3 really starts to take AR story into a very commercial domain and again the story potential of having story book codes linked to AR apps on large TV screens in this context is potentially a completely new medium – especially for the kids who can interact with the 3D virtual characters.

Environments can be enhanced with rich characters too, that as above perform set pieces on triggers or in loops. A nice taste of a UGAR (User Generated Augmented Reality) future is demonstrated by the Living Sasquatch service. This allows you to script sequences together and create mini narratives of basic AR – a unique shape generates a single virtual element on the screen with you. The ability to choose actions and sequence them in the simple assemble editor should provide endless ideas for those AR storytellers who want to make the whole thing far more participative and grow the AR worlds by incorporating potentially hundreds of alternate characters & events. Heres a screenshot of the simple Community Created AR machinima (?) tool.


For the younger kids who are perhaps too young to be running around the house (or indeed the street) with mobile screens we have KWeekies.  Created by int13 one of many companies popping up creating fun kids based augmented reality story games, this introduces kids to battling and quests by assigning various skills & powers to the creatures – not unlike Invisimals above. It doesn’t take any stretch of the imagination to see what bedtime stories will be like in a year or so!

To bring this post to a conclusion, The Mixed Reality Labs in Singapore ask “What are we going to do for fun?” when AR is ubiquitous?  The video below called ‘Future Fun-Mixed Reality Lab features Adrian David Cheok in true doco style (shown on Discovery) showing some more possibilities for story and AR combinations – new game settings, the street.

OK units like Blast Theory with creations such as Can You See Me Now and several other pervasive game design/production outfits have been creating urban, real world game play for years – but we are now at the threshold of something pretty special. The uninhibited world of virtual design layered over real space, in real time and most importantly, connected. Social Augmented (or Mixed) Reality gaming will be big, very big. It won’t all be run about, shooting or questing either the potential here is to have meaningful and emotional journeys into the past, other peoples lives and become immersed in layer upon layer of story. The first incarnations are and will be pretty crude but with the potential for AR ‘players’ to also leave their mark, their response to stories in the geo-tagged, carefully positioned cloud, it will bring new meaning to parallel reality. The young kids getting their first AR presents or experiences this year will  be growing up in a world where alternate parallel reality and augmented worlds are pretty normal. In the very near future everywhere may not be what it seems, as AR creators leave their invisible marks – story threads & adventures for those with the right ‘filters’ to discover.