Mar 092011

I was interviewed by Andrew Collins in December’s Hyper Magazine about Social and Augmented Reality gaming. Hyper magazine is a great game monthly and a regular buy for me with its pretty solid reviews and impartial editorial for the game world as well as some forward looking features. The gaming industry is close to a precipice as games spill out into the real world (as I have blogged about many times before!) so I thought I would publish the article (and my interview on which much of the article is based) this week as the race for the augmented reality, locative game space trophy truly begins and the contestants line up on the starting grid…

  1. Sony with it’s NGP virtual treasure hunts
  2. Nintendo’s 3DS games in the real world AR launch
  3. a multitude of Android AR game apps in development/release and in case you missed it
  4. about to be released the iPad 2 with it’s dual camera support for Augmented Reality locative games and all the iPhone AR apps that will flood across
  5. and of course

Mid to late 2011 is going to be significant – a fun, social, locative augmented reality game nirvana. Perhaps the real battle though is going to be between locked down, TV room, single player console gaming vs open, social, locative casual AR gaming? Interview after the cover…



Traditional game developers are extending the gaming experience beyond what appears on the retail disc and into the social realm, rewarding players for exploring media outside of the console and the PC.

Andrew Collins takes a look at what’s on offer

Casual social networking games have exploded in popularity recently, with a bunch of casual game developers popping out of the woodwork producing low-tech but addictive games. Now traditional game developers and publishers have joined the party, seeking to adapt the trend to their own needs, and their own games.

This bleed of PC and console games out into social networking services has immense potential. There’s a whole world of cool stuff going on right now, and even greater stuff just around the corner – that has the potential to change the way we game completely.

You probably already know the most basic form of this blend of traditional and social gaming: the automatic status update. Many games now will notify your Facebook or Twitter contacts when you accomplish a goal in-game.

It’s unfortunate that this is the most recognisable example of this trend; at best, it’s annoying, and at worst, it’s annoying as hell. Do you really care that your flatmate’s cousin’s boyfriend just unlocked an achievement in FIFA 11? How do you feel when he unlocks 10 in the space of half an hour, flooding your social networking news feed?

Fortunately, developers have realised this and have moved on to integrating gaming and social networking in more interesting ways that suit us all.


Before we look at these developments, it’s worth looking at why the games industry is embracing social networking.

As we found out in issue 204, the market for casual social networking games is booming, generating ridiculous amounts of revenue for those lucky or smart enough to have a finger in this lucrative social pie.

But the learned readers of Hyper are not the only ones who have cottoned on to this fact. Traditional games developers and publishers have seen the sheer number of people drawn into this social gaming trend, and have realised that it could work for them – not as a direct source of revenue, but rather as a form of marketing.

Put simply, every time you tell your 600 Facebook friends what game you’re playing, you’re giving the publisher 600 free ads for their game, and giving the game your own personal stamp of approval. Congratulations! You are advertising space.

Gary Hayes is an expert on the relationship between games and social networking. He has a terribly long bio – far too long to reproduce in full here – with experience in TV, music, virtual worlds, game production, lecturing, and many, many other things. He’s most succinctly described as a `transmedia guru’ – someone who dwells in the overlap of different mediums.

According to Hayes, this venture of traditional gaming into social networking isn’t a short lived gimmick that just a few companies are toying with – it’s now a necessity for developers.

“From an economic point of view, given the massive rise of social games over the last couple of years, and the decline in console games generally (in June of this year there was around a 10% drop in total game industry sales, down to about $6.7 billion), traditional games developers – EA and Ubisoft and so on – are looking at social gaming as really a pretty important part of the mix that they need to be involved in,” Hayes says.

“It’s part of their survival,” he says. “There’s a quote from Alex St. John [DirectX creator and social gaming producer] who says that if a game doesn’t have a social element, it’s going to be dead before it starts out, in the future.”


Continue reading »

Sep 222009

I have been lucky to know Jim Shomos for nearly five years over here in Oz. The first time I met him I was actually mentoring him at a two day networking event in Melbourne and he had a new project called ‘Forget the Rules‘. Like most ‘pushing-the-envelope’ project discussions at these emergent media development labs around the world, they tend towards a list of ‘feature wish lists, bells and whistles’ that are unlikely to see the light of day – not because they are particularly blue sky but more about the time, effort and funding required to get perceived ‘unknowns’ off the ground. So I was delighted to see Jim and his colleague Paul Baiguerra persevere  during 2004/5 and create this first, notable innovative production.

Continue reading »

Nov 122008

What is Mixed Reality, why is it relevant and has TV become it’s background medium of choice for larger numbers of media consumers who around the world are spending more time in online pursuits than glued to the content breaks in-between the advertising slots of commercial TV. How will TV survive in a world where social and gaming worlds are far more compelling?

Latest video above available as a download (with better audio & creative commons) 50MB MP4 click here

I am up at the SPAA conference in the Gold Coast producing at and speaking on one of only two panels, for TV and Filmmakers, that are really focused on the ‘creative’ vs biz only possibilities of games and virtual worlds and (that oft misused term in these circles) New Media (that internet thang). The topic of our panel is “Where Virtual Worlds Meet TV, Where Films Meet Games” and features SilkCharm aka Laurel Papworth, Keren Flavell aka SL’s Starr Sonic ( and truna a great games evangelist and IGDA leader hailing from Brisbane.

This topic is driven partly by a mass fragmentation and a paradoxical reversal of fortunes at the moment – where TV itself is the actual snacking medium and where the longer form immersion is in online communities, virtual worlds and games. But can the mediums truly cross-over?

I have been talking for a couple of years now about the fantastic potential of the live and by implication shared TV experience to be enhanced by extending the world into online games. Where are we and where might we be in a few years once the ‘broadcasters’ realise that keeping an audience involved in the ‘IP’/programme in-between airings is a good thing. Good for the story creators, the latent creative audience and of course advertisers who need eyeballs/hands/ears/minds. Some older Personalize posts on this topic.

Anthony Zuiker creatorof the CSI franchise was at one point going to join my panel and his perspective is critical in this:

“The advertising model for TV is completely broken top to bottom. I’ve lost 20% of my viewership in a year and a half… where are they? They’re on the web and other platforms.” Despite a “revolution” in television that now delivers some of the highest quality programming in the history of the medium…Zuiker believes that the “technological boom” happening concurrently is negatively impacting the popularity of the medium.

MI6 Creative Keynote: CSI Creator Calls For Games, TV To Converge

A part of the SPAA panel here on the Gold Coast, is looking at the issue of where TV and Virtual Space ‘will’ cross-over, not some wacky installation in a dark art gallery but a new emerging element of the entertainment industry where film and games are joined at the hip and TV and social virtual worlds work in glorious real time – live broadcast driving and linking to highly social environments, what Interactive TV has always tried to be. FYI here is the synopsis of my panel “Where Virtual Worlds Meet TV, Where Films Meet Games”.

“Which side of the wall are you on? Are you ready for the Mixed Reality, Entertainment Perfect Storm? TV and Film OR Games and Virtual Worlds? That wall is about to crumble. This is a wake up call to all entertainment producers and consumers to prepare for an almighty collision. Audiences are already spending up to four times as much of their entertainment time in virtual spaces than they are watching TV. Gary Hayes says “I think we are really approaching a perfect storm, a mixed reality perfect storm, because we are seeing several things happening. The first one is a long history of games based on TV and films. Another ‘wind’ is virtual worlds, particularly the exponential growth of customisable social spaces and the important ability to integrate external media into them. The third force is audience behaviour, who involved in far more simultaneous activity particularly between broadband web and TV. The fourth element to this perfect storm is actually what is happening to TV and film, such as live reality TV becoming more game like and now over 100 feature films in production based on game worlds. All of these forces together are creating a really potent mix that all producers need to be aware of”

What is Mixed Reality?

Mixed Reality is a term that needs a more focused definition, as I believe it is also misused. So I see Mixed (or Cross) Reality as

“a creative or social activity that takes place simultaneously in real and virtual worlds and where the interaction in each are resonant and dependent on each other”

So typical practical examples – a conference that has active audiences in the virtual and real world and both communicate seamlessly with each other or where a kids TV show is broadcast while children collaborate in a virtual world and appear to help the progression of the story or a film on a DVD with clues that as you watch it allow you to progress with other players in an online game. But there is also another key aspect of cross or mixed-reality, that of virtual “˜things’ appearing around us in real physical environments. This is often called “˜augmented reality’ but I believe it sits under the broader “˜mixed reality’ umbrella.

Some folk also call this form ‘blended entertainment’ – but to me that doesn’t specifically suggest virtual space and is more related to the already out-dated ‘cross-media’ RIP (hello – Social Media Entertainment) or Extended Entertainment, which is more in my Level 1 or 2 cross-media definition on Wikipedia.

About the Mixed Reality Compilation Video Above

aug-01I think the potential of this two-way flow is significant for education and business but also for entertainment and advertising/marketing. So I put together the short film above that hopefully captures a sense of where we are with this circa 2008. I wrote a rather rushed voice over for the short film, a kind of pretrospective (writing back from the future) looking at a “Dawning of Mixed Reality“. As well as my music and voice over (apologies, it was done at 1am a few days ago)Â it contains many choice clips trawled off YouTube but also quite a few that I have worked on at LAMP, the Format Factory, BBC and various other hats.

Is this reality? Is this fantasy? In 2008 the great transition truly began. Whereas a decade earlier we used to “˜log in’ to cyberspace, now we began to semi-permanently inhabit virtual space and the digital world started to leak out into everyday life. It began to infiltrate our shopping malls and became integrated with our everyday lives. Humanity started the relentless journey towards the natural future where virtuality became a greater reality, and where digital fantasy and organic reality were inextricably intertwined.

Through the early 2000s pervasive wearable computing started to break down the walls that used to separate our virtual existence from our physical one. This same technology also allowed us to embed ourselves seamlessly into virtual worlds which is where we could truly experiment with future forms of entertainment, art, education and business.

In these new places we became the real time, living celebrity. Many of us became avatar stars, pixel gods – exalted as those linear film and TV icons. This became the only place for many. By 2007 hardcore gamers were already spending fifty five hours per week “˜inside’ these shared worlds and learned to forgot about the real world. This became the place where their most meaningful hours were spent. Everyone began to blend the two domains, combining them in new ways, driven by an invisible urge. Thus started the global meshing, of today’s mixed reality.

But in 2009 questions were asked. As in the many films that hinted at our transhumanist future, would the physical embodied self ever become redundant? Would we truly evolve as self contained, digital entities? May we as well be slumped in a chair with a cable plugged into our heads? Regardless, as the real world became inhospitable we all started to crave for the immersion and began moving our physical selves into these spaces. Even more than that we wanted our peers our un-virtual audiences to be able to share our experiences there. It became a contagious as the most poignant moments and memories were the ones in virtual space.

But we needed to look beyond these screens and into our own souls. Find the truth written along the “˜thin white line’ that separates analog from digital, person from avatar. This was the beginning. It always was. As far as the meshing or blending of reality and virtuality, in 2009 they were the equivalent of grainy black and white movies seventy years earlier. Our experiences were simultaneously part real, part virtual but it didn’t matter which was which anymore. Both were valid and they depended on each other. Those early parallel existences of twenty years ago started to move humanity forward, released our imaginations and we coexisted with our precious and ultimately fragile, physical selves. The great crossing had begun.

I was particularly interested in several new developments that are allowing “˜minority report’esque’ eye-catching. holographic human figures into real space. You will spot these in the film, the live Telstra conference or the models trying on clothes in store windows and closed displays. I tried to list most of the items below and have copy paste some of the text that describes the videos. If there are others you think are significant or that I have missed, please comment and as I do with other compilation films I will be updating regularly, particularly on the download versions.

The various clips in the film are detailed below:

  1. InLimbo Blended Reality – A.M. Architect sits In Limbo to converse and showcase their electronic music. Simultaneously to the broadcast, the show was recreated in the KRTU-Second Life complex, the audio streaming and photos of the studio dynamically updating
  2. Metaversatility brings Frogg Marlowe (Jeremy Works) to TX for a couple blended reality music performances. Footage from Limelight in San Antonio & Mozart’s in Austin, for the Second Life TX Meetup, hosted by Metaversatility.
  3. A short 3 hour Mixed Reality Game I devised for a LAMP residential – called the Old Forest more details on UTube
  4. Red Dog Clawtooth (Red Dog’s) Avatar makes his debut on Channel 9 news sharing the concept of Blended Realities!
  5. A Segment on CNBC asia about Adrian David Cheok work on ‘mixed’ or ‘augmented’ reality.
  6. Gizmondo Augmented Reality Game – Catapult. Please not this was never released it was in development when Gizmondo went under. There is no sound on this clip.
  7. Motion Capture & Augmented Reality. Virtual Dancers Part 1- A good sign the tracking is okay is that all this has been obtained on the first test and with no feedback for me when I held the camera. In brief, as long as it is a bit smooth, the real-time tracking is okay.
  8. Project exploring technique of green screening Second Life avatars and real life actors.
  9. A promotional video from The Format Factory
  10. This 3-D Mixed Reality 3-D Book teaches children about the layers of the earth, how are they discovered, how do they related and function, siesmic waves, comparative sizes, and many more.
  11. Larngear Technology’s Mixed Reality Learning Media being used in real classroom environment, during a high school chemistry lesson in Mater Dei School. This is featured in the TV progam Mega Clever
  12. Tim Johnson’s presentation was held at the Centre for Digital Media at Great Northern Way Campus – home of the Masters of Digital Media program. The event was simultaneously broadcast in Second Life at the school’s virtual world campus located at University Project.
  13. The worlds first holographic illision, designed for retail. Created in 2002. You can find the team behind this solution at
  14. Dreamoc, a holographic video display designed to create mixed reality. The world where real objects meets virtual fiction. The Dreamoc system is designed by realfiction
  15. Future Fun – Mixed Reality Lab, Singapore, Adrian David Cheok : A documentary about future fun
  16. DNP and metaio implement the first augmented reality museum application with a handheld system in LOUVRE-DNP Museum Lab, Tokyo.
  17. A re-invisioning of a 1916 Italian Futurist film using Second Life Augmented Reality Technologies, being developed at Georgia Tech. The short film is a hybrid live action and machinima captured in real time. There are no use of Green Screens. Pos-production is not used to composite the Second Life image and live image.
  18. Augmented reality driving from Total Immersion.
  19. Large scale implementations of augmented reality solutions from Total immersion.
  20. Aussie talkshow host Rove McManus made history yesterday at Auckland’s Vector Arena as the first person to have his hologram beamed across the Tasman. Multiple cameras beamed his three-dimensional image from a studio in Melbourne to the paper-thin Auckland screen, accompanied by sound effects from Star Trek’s transporter beam, enabling McManus to have a live discussion with Telstra-Clear chief executive Allan Freeth.
  21. Roy Block is an experiment, sort of game, with a tangible interface that touches the field of mixed reality. It is a project by Sebastian Schmieg, student at Merz Akademie, Stuttgart.
  22. 2004 Video, computer, pneumatics. A stack of books sits in a chair and, on them, a monitor abuts a small table. The surface of the table holds water, forming a small rectangular pool in front of the monitor. The monitor depicts a woman’s face, her chin resting on her folded hands.
  23. http://www.reactrix.comDog going crazy for the interactive soccer ball on the Reactrix interactive floor display
  24. Another sample of interactive floor projections: the Water Effect. Scenes creates custom interactive effects for any kind of event, on the floor as on the wall or screens.
  25. Technological platform for interactive floor ocean. Collaboration with Studio Azzurro. Museo del Mare, Genoa.
  26. Worlds first multi touch interactive musical wall. This interactive wall allows multiple users to conduct virtual symphonies.
  27. Multi-touch table. This table is a departure from earlier examples because it is relatively easy to use by non-technical content creators. It works in most lighting because it doesn’t use a camera
  28. – Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. By Johnny Chung Lee, Carnegie Mellon University. For more information and software visit
  29. News broadcast on CNBC – Mixed Reality Lab, Singapore, Adrian David Cheok : 22nd February 2002
  30. Larngear Technology’s story in the research and development, designing, and marketing of Mixed Reality technology is well describe and illustrated in the TV program Clickzone, supported by SIPA. The show is hosted by Tai, Chutima, from the movie Season Change.
  31. More to come…

There are several companies around the world developing Cross-Reality forms. One that I am heavily involved with, The Format Factory, are pioneering formats that bridge the space between compelling participatory TV and online game worlds. This is a kind of teaser video I created that metaphorically demonstrates some of the ’embedded’ world-within-worlds. Their sister company The Project Factory have been doing some basic but interesting cross-overs in Second Life too.

I also talk in more detail about Mixed Reality potential in this early 2006 seminar, here are the slides from that one containing many cross-branded vs cross-reality examples.

I will leave the final word again to Mr ‘CSI’ Zuiker who instinctively points out that particpant audiences expect to be involved in the story, to be the hero, the villain and to feel they are partly in control of their own personalised route through it. Customisable and game worlds with existing culture are the best tool at film and TV makers disposal.

“In the gaming area, you want to give people tasks, to shoot things and upload pictures… You’re doing this because you want these people to be creating their own story and it will be part of the crime on the broadcast… Even if it’s not the actual thing I shot, I was part of that experience, that community, that narrative.”

PS: BTW I think the term ‘participatory audience’ used in the title of this post/article, is a bit of an oxymoron. Audience as a term suggests passivity not participation…but I am sure you will let me off for that one?! Please.

Oct 222008

OK you should have spotted quite a few characters living on this post :)  Originally there were ‘video-real’ talking, salesy character centered on the page courtesy of CLIVEvideo but I still talk about them more below.

A few months ago I blogged about the new kid on the intranet block, those  ‘layered’ social virtual worlds. Quite simply they are communities of pseudo 3D avatars layered over the 2D web (browsers). I noted that these services are a transition to a ‘live’ collaborative web 3.0 world as this is more of a “let them dip their toes in” before committing to a higher bandwidth, more fully rendered 3D world such as many of those on my sticky video of the 08 metaverse.

I certainly think is the best approach for large numbers who wouldn’t be seen dead or alive in something like Second Life. This is another quick whistle stop tour of a quickly evolving player, Rocketon and also a recent Aussie company who have an alternative approach -  ‘live action’ video layered over the 2D web (Incidentally if everything is working you should have had a person talking to you in the middle of this post – if not it may be many months later and things have broken OR some other technical reason I cannot ponder at the moment – IE!). Even though I start by talking about Rocketon and it’s implications, having the privilege of playing with the demo of CLIVEvideo for a while I realised many points are relevant to both – bar the ‘big’ nay huge fact that Rocketon is social (shared, real time and partly pulled) and CLIVE is pre-rendered, pushed and fixed (although they tell me they are working on being a bit more web 2.0).

I have been beta’ing and playing with Rocketon for the past few weeks trying to see how it fitted in with my normal zillion web 2.0/3.0 application lifestyle and finding out where the real attraction is for large numbers to adopt this hybrid paradigm. Firstly it I noticed that with Rocketon in minimize mode, every web page I visited it seemed to be doing something in the background, watching? Spying? Regardless every hour or so it gave me a present – some pixel jewelry, a funny avatar – I have a massive collection of stuff now – what to do with it all and how does an emerald relate to me browsing a ‘map of sydney site’? I have still to work out what is going on with general browsing but two killer apps are evident with Rocketon after a few hours tinkering. 1 – Making existing branded websites fun/sticky and 2 – Making web surfing more social, gamelike and challenging.

The first image you can see above is me and SilkCharm being silly so and so’s dropping Burger King pixel toys on MacDonalds sites (only we can see it of course), but with a larger group like the top image, it starts to have significance…if only in the ‘power’ to do so and the fact that pictures/videos are taken and put on blog posts/flickr/YouTube (ah the old rippling impressions). I also made a quick film of a few of us invading the SMH webpage, partly Laurel and myself showing how ‘communities’ can and will make ‘statements’ – much the same as we do in group based social situations in the real world. The potential for positive product placement, interactive toys, loyalty benefits and so on will not go unnoticed by readers of this post!

But the more interesting element of Rocketon for me is where the community are given the tools to create quests, puzzles or games for each other. To demonstrate the potential of CCG (community created games) the Rocketon team set up a simple quest with pretty easy clues. The process, you are given a mission, you read clues, travel to websites (with the Rocketon layer activated) come back to a base and so on.

The thing I really like about this simple example is that you can embed pixel ‘treasure’ or goods on websites, without any recourse to the website owner of course. (I am sure Rocketon are thinking hard about the legal ramifications of hundreds of RTons heading off to litigeous sites to find inappropriate items and then posting the experience!). Anyways you can see in these two images I have been given a secret envelope and sent to ebay to collect a parcel to post and then await further instructions. Suddenly a couple of web pages turn into a scene from The Thomas Crown Affair.

I have quite a lot more to say about Rocketon and it’s distant cousins such as weblin but time is pressing and lots more to get on with. For the moment though all I can advise them is to enable tools for the community to develop their own fun or for marketeers to start to offer quite tricky quests for real world prizes – I am sure this is happening, it is the only path to really get the numbers up.

So to CLIVEvideo. I have literally been playing with this for less than an hour today after Scott from Maxy’s grabbed me on twitter! It looks very promising. I have seen many variations of this over the years but the implementation of this particular technology is pretty accessible and is squarely aimed at ad agencies, SMEs and larger companies and those who want to differentiate their website and make it a little more viral. As with the points above about Rocketon the real value of having layered personalities over the webpage is to build bridges between the layers (the avatars or video peops relating to what is below them) – or why be there in the first place. have some great tools to build ‘key’ed’ (invisible backgrounded people) sequences and to also add in sequence applications (person, flash demo, person, page link, person, product video demo etc) and are focused on sales or corporate messages at the moment.

But imagine a future where the keying is from 4-10 people, a webcam community, who start to act a little like we have been doing with Rocketon. Doesn’t have to be full body necessarily, but why not – webcam pointing at users in front of a green or blue screen in their office/bedroom. Then you really have some potential to make the 2D web much more fun and sticky. The applications for marketing, socialising etc start to kick in when you can (like some video chat applications) render pixel elements over the top of the live video image. Ummmm. *rubs hands*… It will certainly be a lot of effort for some, but having specially designed web pages for ‘Keyers’ (as they shall be known) would also provide Google Lively type integration – key yourself live into this and make the branded movie etc etc: This reminds me a little of the fun video I did at AFTRS recently with SilkCharm and lots of invited real people – keye’d into World of Warcraft – that I shall leave you with!

Finally, finally well still on this topic a new player that makes it even easier to meet and chat based on the web page your on is Live World. It’s product LiveBar is basically a ‘chat’ engine that detects the page your on and connects you to others that are also on that page.

Now we will really see how popular some webpages are 🙂