Jan 012012

Originally published Oct 2011 in Wired Magazine ‘Change Accelerators‘ by Gary Hayes 4 of 5

Image by Gary Hayes

When planning your next holiday to London with the fam, don’t forget to sync up your iGlasses and load up the London experience packs. On arrival, slip on your augmented reality sunglasses and take a look around: Roman-era London appears before your eyes. Slaves and gladiators walk through the streets and chariots rush past. You can add your own comments leaving virtual “We Were Here” graffiti for all time. The experience is part documentary, part user-generated narrative, and entirely pervasive. In other words, augmented reality meets living history.

In our everyday lives, we engage with stories in many ways, whether it’s eye-to-eye contact with a stranger that sparks an instant connection or a well-crafted movie or TV show. But what if we started experiencing those stories in the outernet’s layers?

While online networks are evolving traditional entertainment, such as TV and web series, we are also witnessing the rise of a new form of media called “augmented reality storytelling.” I’ve dubbed this new form of diversion ’ntertainment, as a shorthand for immersive augmented reality entertainment.

At its broadest level, augmented reality is about enhancing the physical world through digital elements, such as images, sound, and information. Now technology is enabling us to further situate and layer our digital stories in places where other narratives can’t reach. Right now, we see this happening when someone holds up a camera on an iPhone or tablet and shares objects or stories from the real world.

The opening Roman London example is based on an existing service called Londinium, which is a collaboration between the History Channel and the Museum of London using augmented reality video layered over real-world streets to re-create an alternate history. Coincidentally, London is also used as a site in the globe-spanning Ghost Tours 2.0. Haunted London encourages visitors to explore the city’s eerie side using locative AR (augmented reality). Likewise, another situated project is Witness, which draws participants into the dramatic and seedy underbelly of criminal Berlin. In this case, players are the hero: They watch graphic video scenes at different city locations and are then sent detective challenges to uncover the truth. But here’s the twist: The story might just bite you back! Augmented reality games and stories can even get physical, like the recent example of Chelsea FC playing the world’s largest Space Invaders game in a stadium using projection AR.

Gaming is leading the way. New consoles, like Vita, allow users to literally take game characters orreality fighters into the streets. Other gaming advances like AR games on Nintendo’s 3DS start to recognize place markers placed around a player’s city, transforming screen-based MMORPG(massively multiplayer online role-playing games) into an LMMOG (location-based massively multiplayer online games).

Augmented reality storytelling is starting to appear across our smart GPS mobile devices. Several marketing campaigns are taking the initiative by spearheading real-time AR challenges, such as Vodafone’s Buffer Monsters, which challenged German smartphone users to download a mobile app to capture virtual creatures and win a lifetime plan. This is only one example, other AR advergames encourage users to competitively run around cities on scavenger hunts for real-world prizes, such as the Droid Bionic AR Game. Similarly, this October, Gundam, the Japanese anime giant, release an iPhone/iPad app called Gundam Area Wars. The game uses the devices’ camera and gyroscopic sensors to show life-size 3D models situated in the player’s real-world landscape.

Given these above examples, I return to my earlier travel scenario and I wonder how commonplace it will become for people arriving in a new location to start experiencing it through augmented reality storytelling and gameplay? The traditional guidebook has already morphed into digital form. The Lonely Planet is already a downloadable app. Is it a big jump to imagine AR and location-based storytelling won’t soon allow travelers to engage history on a whole new level? One might even argue a deeper and more meaningful one than just the 2D sightseeing experience of looking at crumbling ruins. So many guidebooks have been written on the principle of making history come to life—AR actually makes it possible.

One could even take this one step further and question, why do we need to travel at all when we have our own personal Holodecks at our fingertips? Fast Company recently reported on Tour Wrist, a virtual tour that lets iPad users move around a global location with unlimited zoom and freedom. “Travelers” are virtually transported to that place and able to immerse themselves in it becoming the hero in a remotely situated, digital storyworld.

Finally, in the near future, we might all have the capability to create duplicates of our surroundings in 3D for others. This Microsoft R&D initiative to map the world uses the fastest selling piece of tech on the planet, the Xbox Kinect. This would allow everyday people to create unlimited user-generated 3D AR—foreseeable as easily as snapping a digital picture. In addition to this, there is a saturation of location-stamped social stories inside services, such as Google Earth, TagWhat, HistoryPin, Facebook Places, CheckIn+, Foursquare, and Gowalla, among others. What will result from all these stories becoming interconnected and navigable using AR devices?

From that point on, we will be co-creating an augmented entertainment eternity. Together. Will you be a part of it?

Mar 212009


There are a multitude of sites out there asking the ‘audience’ (grrr)  to submit films, music, scripts, stories, bits of their personal life and anything the brand or property feel will draw them into their branded world. Many go about it in a really poor way, providing virtually no incentive, a pretty small audience (as regards the actual community that will likely see their work) and often give little or no encouragement to learn and improve on their original submission. Many even resort to seeding the ‘competition’ with faux community created videos (made by pros deliberately shaking the camera!) – to pretend to kick-start it – they have no idea how easy it is for the community to sniff that out.

An easier and more enpowering way in for the co-creative community is to give them ‘great’ assets to create great content – it gives them a big kick-start. Giving them the same stuff that the pros get generates the real big incentive – it lets them show how ‘they’ are as good, if not better, than the pros! It also shows a willingness on the part of the usually out of reach ‘creative production’ to expose some of the real behind the scenes’ness and draw communities into the brand firstly from a ‘trust’ perspective. Secondly as the ‘creative audience’ members themselves will be spending many hours with the assets and this creates very strong brand/story loyalty. Often these creative types are pretty active influencers on the web anyhow and will draw their own communities into the brand. The list of benefits goes on.

Below are relatively recent examples of professional film and music folk throwing top draw fragments of content (rushes, isolated tracks etc) over the wall for the wannabee’s to work their own magic.


I have written about this subject a few times in the past but Bronwen Clune on Twitter pointed out her brainchild initiative, a Disney / AFTRS Film  “Two Fists One Heart” making available some rushes for the community to re-edit.  Cutscene site has all the information on the project and a mention should go out to my colleague at AFTRS, Bill Russo who has the enviable task (yes enviable as it is a joy to see community wisdom like this) of viewing potentially hundreds (see examples below) of alternately edited scenes.

We are giving you the chance to download and edit rushes from the international film TWO FISTS ONE HEART. This is the footage that editors work with and it is free for you to use under creative commons as long as you acknowledge the source and link back to the official movie page. We€™ve tried to give you a mix of scenes so that you can even download all of them and put together a short film of your own. When we told the folks at AFTRS and Disney what we were doing they were so exited they wanted a way to recognise some of the best videos that are created. SO €¦

The best 5 scenes will be posted on Disney€™s promotional TWO FISTS ONE HEART site. This is a great opportunity for some exposure to high-profile people in the film industry. The 5 best scene cuts will be selected by Bill Russo head of Editing at the AFTRS and the creative team from TWO FISTS ONE HEART. TWO FISTS ONE HEART Director Shawn Seet, Editor Milena Romanin and Cinematogropher Hugh Miller are all graduates from AFTRS, Australia€™s premiere Film and Television school.

The WINNERS will be personally contacted by Bill Russo who will give them editing advice and help with their editing careers. All you have to do to enter is post your video on YouTube, link to the Two Fists One Heart site for the movie in the info section and tag it TFOH, then email the link of your entry to competition@cutyourownscene.com



Of course this is not a new idea and I recall at least five major projects at the BBC I was involved in from 96-03 that did a call out based on downloading and then re-editing, ‘professional’ footage (and at least 20 other ‘mash-ups’ using web interfaces). A recent non-BBC, feature example (Jan 08) that springs to mind is Tracey Fragments – a sort of timecode’esque Canadian film that gave away the ‘whole’ film to re-edit and again used YouTube to show the re-fragmentation. These are still available on the site – click refragmented. Here’s one example of a re-edit that have around 10 000 views each on YouTube.

Tracey: Re-Fragmented made available all the footage from the shoot of the film for users to download and re-edit their own replated projects including music videos, new trailers or to re-redit the entire movie themselves. A contest for best use of the footage has just closed at the end of January and judging will commence soon. The re-reditng initiative also has a competititve elements with Bruce McDonald and his editors selecting the best from the pojects from Canadaa to win an Apple Final Cut Pro prize pack. The winning project will also appear in the bonus features on the DVD release of The Tracey Fragments.


As mentioned the BBC has a long track record of opening up its rushes to the public (well it would do because the ‘internal’ BBC av assets are effectively owned by the taxpayer!) as well as providing community filming skills to a vast audience with 15 year old initiative such as Video Nation. Another recent example (2006), was when the BBC Commercial Archive opened up some of the rushes from its natural history section and asked the audience to re-cut or rather creatively make a brand new trailer for Planet Earth. Here is one example entry to the BBC Planet Earth video editing competition

To highlight the creative potential of the Open Earth Archive the BBC is also running a competition to make a short ‘advert’ for Planet Earth. Novice editors can enter the competition through the Easy Edit Suite, an exclusive application available free on the Open Earth Archive site allowing users to create a short video with a sound track using some of the best bits of the archive. The competition closes on 30 April 2006 and winners will spend time in an edit suite with experts seeing how the professionals edit for BBC Television. The Open Earth Archive is made freely available for the UK public to use under the terms of the Creative Archive Licence. The Creative Archive Licence allows people within the UK to watch, download and edit material released for non-commercial purposes, using it to inspire and create their own creative endeavours.


insectmen02At LAMP there have been at least 20 projects based on the call for creative contribution. This includes the iEmmy nominee Scorched as well as a Gruen Transfer’esque ‘Sold in 60 Seconds‘ and most of our ARG variants. One really cool ‘video fragment’ example was on our very first residential workshop lab in late 2005,  Insect Men. Insect Men was a video fragment scavenger hunt. The mind of a character (of course represented by a linear film sequence) was shattered in a freak lab accident and his mind spread all over the web, outdoors and on linear channels (these things happen!). The audience had to find, collect them and put it back together in a meaningful way. This predated Eternal Sunshine of the Spotless Mind and other similar recent play’s on fragmented memory but this is less about story and more about bringing a game-play element into the re-cut too. The team were clear that there was no right or wrong way to put this unfortunate characters mind back together but rather courted community compare contrast – which is the best way forward.


What to do when all this great stuff comes back?! Looking at exactly the same methodology of – here are some professionally created assets by your heros for you to rebuild in ‘your’ wayImogen Heap (who I keep going on about?!)  in the past month gave away a song in the form of lead and backing vocals for the co-creative fans to provide their own music. This takes it a step further as the backing to the vocal tracks truly represents the genre and style of the contributors vs the more subtle ‘persona’ that comes through a pure film edit. Imogen and team got back nearly 400 completely rebuilt unique 3 minute songs and you can listen to 219 of them here on the Twestival site. They were so overawed by them they are going to release a special EP with the best ones on. Shame that wasn’t part of the original incentive – but it probably didn’t matter in the case of a loyal fan base already!


OK there are folk out there who use ProTools, Final Cut, Logic, Cubase etc etc: There are even more who are now familiar with iMovie, MovieMaker, Garageband etc: But although these will produce much better and original results it requires a significant amount of extra effort than a browser based tool.  I was a little derogatory about web based editing or mash-up tools earlier but there have been a few good examples over the past 12 years on the web we know today – yes folks they have been around for that long! Even I did some in 1995 when I put the first audio and video on the main BBC sites in the UK – simple quicktime based mixers which worked even in the days of 28k modems!

A recent one that works because it is so integrated with the TV component is ABC Australia’s Gruen Transfer. The thing that really works for me (above and beyond the AdMixer interface and usuabilty – yes they are getting better) is the fact the call out from the show gives a very specific task – this week/series for example the presenter said – go and create using a bunch of pre-built clips/audio and text, a promotional advert to sell Australia. Simple, a bit limited tool and clip wise, but really easy for an audience to quickly produce something meaningful with very little effort.



As regards remixing, mashable content there are many others worth investigating if your into the subject and a quick look around will reveal the likes of

But a last point I shall leave you with is make it accessible. If only 1-2 % of the audience will get off their a$&#s and make something and send it in then make it easier for them – grow that to 10% or more. I covered the sort of splits of the co-creative audience on my post Web 2.0 and the Myth of Non-Participation. Allow them to search and embed other peoples work, allow them to just vote or rate (obvious and used a lot of course) or give them some really good material so they can create something ‘they’ will be proud of – like most of the examples above here.

OK there are lots more examples of this including community driven film sites like my fav triggerstreet and I have probably missed the ‘big’ ones (so tell me in comments!) but the last message –  Throw good stuff at your collaborative, co-creative audience and they will reward you many times over!

Sep 282008

Have we reached a tipping point – with many more user hours spent with games than films are they now more culturally relevant (as in our cultures are saturated with them)? With most films having ‘game-like’ story arcs and, at the last count, nearly 80 films with stories based on game titles in production I am starting to think so.

Game culture and their inherent stories are now absolutely mass media. In a low risk, and dwindling film business, creating stories around experiences that people have already spent 20-40 hours immersed in the story world, is a no brainer, so what we are seeing is a threshold now of game-like films but more importantly films based on games. Anyway more after the ten minute video – stick with it.

“Playing With Stories” THE CINEMATIC GAME. A Film by GARY HAYES

I am designing curriculum for cinematic games and virtual worlds at AFTRS but also doing another report on the market potential of this cross-media, gilm (game/film) landscape. In the process again I threw together a compilation video of notable examples (I know there are at least ten times this btw!) interspersed with tasty quotations. “The Cinematic Game” was initially designed to be a look at the cross-promotion and story development potential of this most powerful mixed-media marketing machine. But, during the process though I was staggered to see the number of major feature films in production based on new and existing game universes (listed in this post below and scrolling at the end of the video) – suggesting to me a tipping point.

Game story starts to lead film development?

TV and Cinema has already become much more of a background or escapist medium for larger numbers of media consumers. In homes around the world we are spending more time in online pursuits than glued to the content breaks, in-between the advertising slots of traditional TV. We are also immersing ourselves in the social and story ‘exploration’ of the current generation of PC and console games. So how will TV and Film survive in a world where social gaming and associated peer appraisal online is far more compelling? Also given the choice will we continue to passively watch the protagonist or ‘be/live’ the hero? It is interesting to see 8000 employee EA Games now developing major strategies whereby games are made to be easily adapted to comics, books, TV and Film. In the business week article “Morphing Video Games into Movies” they note how EA are trying to emulate small non-game companies have built mini empires on their ‘story IP”

The idea is to repeat the success of companies such as Marvel Entertainment (MVL) and Hasbro (HAS), which used their base of fans to transform from marginal companies into Hollywood players. After licensing Spider-Man to Sony Pictures for a string of hit movies, Marvel has created its own studio, with Iron Man and other films set for release this summer. The Hasbro-backed Transformers movie grossed more than $400 million in 2007 global box-office sales, which in turn boosted company sales of movie-related toys and games.

It is interesting to note that the music industry is also starting to ride the coat tails of the games world. Kotaku reported on a ‘run-in’ between Warner Bros. and Activision about Guitar Hero. Suggesting the music publishers should get more royalties from games that use music, Activision’s boss Bobby Kotick hit back at Warner’s and said the following (which implied as the Kotaku item said ‘Perhaps the record companies should pay us‘)

We’re going to favour those publishers that recognise and appreciate how much we can add value to their artists… in the case of those kinds of products, you should be paying any money at all and whether it should be the reverse.

Back to the main thread of the post, it does make you wonder how many screenplay writers are sitting in front of their XBoxPS3Wii’s looking for inspiration nowadays? Variety suggests that in fact ‘all’ games could be made into movies but I will be really interested in what kind of film comes from The Sims and already know the likely story arc of MassEffect having run through it a couple of times but many others on the list below will be of interest, especially World of Warcraft which has around 4000 story threads/quests – so which story will we be ‘offered’?Films of games have had a shaky past with only a few critical successes such as Tomb Raider, Silent Hill, Resident Evil (there are several on slide 75 of my game/story presentation below, that I did several months ago) but given the serious money and credible directors such as Landau, Lucas, Speilberg, Cameron, Jackson etc: plus a deep desire to properly reflect the integrity of the ‘interactive’ experience, the tide is turning. Being an avid machinima maker I know at first hand what it means to capture the ‘essence’ of game playing, adapt it, reflect it and, if you understand the culture of the game, interpret it – the good thing is A list filmmakers (as you can hear Peter Jackson say at the end of my video) understand it too.

I must again finish with a plug for a couple of unique courses at AFTRS and related to this post an article in one of its blogs RedSet called “What’a film, TV & Radio school doing offering courses in game design and virtual worlds?” makes the relevant point:

“AFTRS new game design and virtual world graduate diplomas will push students to go beyond the generation of clichéd actions and stereotypical characters, students of these new courses will be encouraged to step up and learn how to create meaningful interactive experiences for a variety of platforms informed by the expertise offered in all of the other creative disciplines taught at AFTRS such as directing, screen composition, screenwriting, sound design, production design and more. The field of game design and interactive experiences is equally as collaborative as the world of filmmaking, drawing together diverse specialists who together create the whole – writers, screen composers, programmers, animators, art directors – at AFTRS all of these disciplines are already housed under one roof – with a track record of cross disciplinary interaction and a staggering successful graduates.”

More about my video

A non-exhaustive compilation of story rich games or gamic films including in order of appearance: Contact, Indiana Jones, Heavy Rain, The Game, Burning Crusade, Max Payne, The Matrix, Heavenly Sword, Final Fantasy, Lord of the Rings, Ironman, Call of Duty 4, Simone, Rage, Tron, Bicentennial Man, SpiderMan 3, War of the Worlds, Tomb Raider, I am Alive, WoW Lich King, Indigo Prophesy, Jumanji, Desperate Housewives, Da Vinci Code, The Beach, Assassins Creed, Thomas Crown Affair, CSI, Halo, Resident Evil, James Bond, Sleuth, Afrika, The Godfather, The Cube, Narnia, Time Bandits, The Golden Compass, Half Life, Never Winter Nights, Silent Hill, Hellgate, Beowulf and interviews with George Lucas and Peter Jackson plus quotes from many film directors and games designers

My film contains some of the better hybrids, either films inspired by games, games inspired by films or just very rich cinematic, story or character rich games. I make no excuses that I have used a mixture of cut scenes as well as ‘real’ game play in the video – that is really to show where we are heading as game graphics continues to hurtle towards the real time equivalent of the likes of Beowulf and other ‘trickle’ rendered CG features. After the quotes and textual references from the compilation below, are more elements on this very exiting hybrid cross-story, cross-IP, cross-reality world.

I want gamers to be surprised by their own creativity. I want players to feel not like Luke Skywalker, but George Lucas Will Wright (Sims, Spore)

We’re way beyond the notion of game-as-brand-extending afterthought. Let the virtual world–the vibrant, living world that people inhabit–let that influence the movie. Let it feed back into the process and provide unparalleled riches and depth to what we’re doing
John Landau (Titanic)

Games are already good at creating fear, suspense, excitement, shocked surprise, and laughter. Much rarer are games that create genuine sadness … I have never cried during a videogame
Marc Laidlaw (Half-Life)

I think the real indicator will be when somebody confesses that they cried at level 17
Steven Spielberg

When I found out one of my guildmates had died, someone with whom I had fought monsters, explored exotic lands, shared moments of jubilation and defeat, I wept. In spite of having never met him, the knowledge that we would not continue the story together, brought me great grief.
Laurel Papworth

We had a notion to take the stars of the movies and have them play supportive roles in the video game and tell a story that is a companion story to the movies
Joel Silver (Matrix)

If done well, I don’t believe a videogame itself can detract from a film experience. Ideally, it would be a complement to the film and a way for fans to further involve themselves in a world once they leave the cinema
Peter Jackson, (King Kong, Lord of the Rings)

There are scenes that start in the video game and will complete the movie – ¦and fell like it’s a part and experience of the movie
Joel Silver (Matrix)

Games and MMOs in particular are providing such a sustaining experience that challenges us to make the theatrical experience better
John Landau (Titanic)

The next big emotional breakthrough in gaming is being able to tell a story that is consistent throughout the narrative. If the game is 15 levels, it’s just like 15 chapters in a story
Steven Spielberg

We’re trying to understand the language of the film, but diverge in ways that are right for the game medium.
Neil Young’ EA VP (Lord of the Rings)

Games sometimes can reveal things. To watch someone in movement, unconscious movement, can be very stimulating and revealing, whether they win or not.
John Turturro (actor)

People wonder why games don’t have the same emotional palette as movies. But that’s the wrong way to look at it. It’s like saying, ‘Why isn’t radio like reading a book?’ Games, inherently, have a different emotional palette, which is their strength
Will Wright (Sims, Spore)

Scroll at the end of the compilation:

FILMS BASED ON GAMES IN Development or Production 2008/9 (Wikipedia and IGN source)

Alone in the Dark 2
American McGee’s Alice
Area 51
Battle Royale
Biohazard: Degeneration
BloodRayne III: Warhammer
Broken Sword: Shadow of the Templars
Citizen Siege
City of Heroes
Clock Tower
Cold Fear
Crazy Taxi
Deus Ex
Devil May Cry
Doom 2
Dragon’s Lair
Duke Nukem
Earthworm Jim
Eternal Darkness
Eternity’s Child
Far Cry
Fatal Frame
Fear Effect
Gears of War
God of War
Hunter: The Reckoning
Jagged Alliance
Kane & Lynch
Legend: Hand of God
Lost Planet
Mass Effect
Metal Gear Solid
Mortal Kombat: Devastation
The Neverhood
Nightmare Creatures
Ninja Gold
Paul Blart: Mall Cop
Rainbow Six
Resident Evil IV
Return to Castle Wolfenstein
Sabotage 1943
Silent Hill 2
The Sims
Soul Calibur
Sonic the Hedgehog
Spy Hunter
Street Fighter:
The Legend of Spyro
The Sims
The Suffering
The Unforgettable
Tom Clancy’s Splinter Cell
Tomb Raider III
Warcraft (based on World of Warcraft)
Zombie Massacre

Mixed Reality Futures

As a lead into a post about to be published I have been talking for a couple of years now (The Mixed Reality Perfect Storm ) about the fantastic potential of the live and by implication shared TV experience to be enhanced by extending the world into online games. It is exciting to think where we will be in a few years once the ‘broadcasters & studios’ realise that keeping an audience involved in the ‘IP’/programme in-between airings or sequels is a good thing. Good for the story creators, the latent creative audience and of course advertisers who need eyeballs/hands/ears/minds and hearts.

A further afterthought there are several companies around the world developing Cross-Reality forms, one that I am heavily involved with‚ The Format Factory, are pioneering formats that bridge the space between compelling participatory TV/Film and online game worlds. They have a promotional video that metaphorically demonstrates some of the ’embedded’ world-within-worlds. A trailer video teaser for the mixed reality, inhabited TV formats being pioneered and piloted by The Format Factory.

Other posts on this topic:

Sep 232008

(short LAMP watercooler cross post)

…and we are all encouraged to be a part of the demise of mankind for a few months starting on October 6th. I suggest you join the facebook group which will keep you up to date with events. Feed your experiences into your blog/s, find wiki’s tracking events then contribute and of course post flickr and YouTube response videos. Also subscribe to the SuperStruct Global Extinction Awareness System YouTube channel.

We are all in this together!

To start the event off lets fast forward to 2019 and see what is happening through the eyes of GEAS (Global Extinction Awareness System), then immerse yourself in the fiction, the game, the social media entertainment event of the year…I hope!

Superstruct Superthreat: Power Struggle

Superstruct Superthreat: Generation Exile

Superstruct Superthreat: Outlaw Planet

Superstruct Superthreat: Quarantine

Superstruct Superthreat: Ravenous


Q: What is Superstruct?

A: Superstruct is the world’s first massively multiplayer forecasting game. By playing the game, you’ll help us chronicle the world of 2019–and imagine how we might solve the problems we’ll face. Because this is about more than just envisioning the future. It’s about making the future, inventing new ways to organize the human race and augment our collective human potential.

Q: What does “superstruct” mean?

Su`per`struct´ v. t. 1.To build over or upon another structure; to erect upon a foundation.

Superstructing is what humans do. We build new structures on old structures. We build media on top of language and communication networks. We build communities on top of family structures. We build corporations on top of platforms for manufacturing, marketing, and distribution. Superstructing has allowed us to survive in the past and it will help us survive the super-threats.

Q: How do I play Superstruct?

A: Superstruct is played on forums, blogs, videos, wikis, and other familiar online spaces. We show you the world as it might look in 2019. You show us what it’s like to live there. Bring what you know and who you know, and we’ll all figure out how to make 2019 a world we want to live in.

Visit http://www.superstructgame.org/ to sign up for news and updates!