Sound recognition at GDC 2012 in San Francisco

Since the start of our own R&D activities (in collaboration with Ghent University), we have developed a couple of proprietary smart music/audio processing technologies and it's about time we show the world what we have to offer.

Technology
So this week, we are showcasing technology that allows game developers to control a game by using real-world sounds at GDC 2012 in San Francisco (USA).

Setting up our booth at GDC Play (Bram got it all working, yay!).

Our Percussive Sound Recognition technology (PSR) is a middleware package that analyzes incoming audio and listens for occurrences of pre-defined percussive sounds. You could see this as extra trigger events coming in, not through a physical game pad, but through an audio source, such as a microphone input.
PSR works in 2 steps:

• first you tell the system about the sounds you will be working with (this is used to train a sound recognition model)
• once the model is trained, you feed the system a new audio stream with sequences of these sounds, and it will trigger events as soon as one of these sounds is recognized

These trigger events can be used to control various gameplay elements: the sound of tapping on the table could trigger a jump action, the sound of hitting a glass could cause a protection shield to pop up, the sound of hitting a box could cause a canon to fire, etc...
In short: players can control a game by making real-world sounds.

Demo game
We have been working together with a team of game developement students from mADE/NTHV in Breda (NL), who have used SampleSumo's PSR technology to create a demo game called "Cows'n'order". It's a fun and collaborative game, in which 4 cowboys need to guide their cows over a track filled with obstacles. Each time an obstacle can not be avoided, a cow dies... Each cowboy can remove certain obstacles, but only if he is at the front of the group at the right moment. The fun thing is that you can make your cowboy move to the front by hitting the right real-world object: a glass, a box, a table, two spoons, etc...

Game credits: Karlijn Koning (team lead, artist) Dieter de Baets (tech lead, programmer) Rick de Water & Carsten Frentz (programmers) Bart Geels, Monica Pranoto & Janice Tirantautama (artists) Nils Desle, Robbie Grigg & Ronny Franken (supervisors).

This is the trailer they made for the game:

Of course, this is just one example of what is possible with the technology.

Interested?
The technology is currently available as a C++ library with documentation and example code for Windows and Mac, and iOS is on the list next. If you're a game developer and are interested in using this technology to build a game, please contact us.

We are still in San Francisco until Friday March 9th, so if you happen to be around, drop by for a live demo at booth #48 at GDC Play (Esplanade Ballroom at Moscone South hall). We still have 2 free Expo passes left, so let us know if you need one.

And yes, we did already see the morning fog and the sea lions at Pier39, crossed the Golden Gate bridge and had a nice little Californian earthquake wake us up this morning around 5:30 ;-) All warmed up for GDC Play! Bring on those game developers!

Post-conference update
Well, this was quite an interesting exeprience: we recieved very good feedback from several companies ranging from game developers, over toy manufacturers to even casino owners. Even though GDC Play was new this year at GDC, and some people didn't find out about it only after it had ended, we did manage to attract quite some attention. Making lots of noise while playing the demo game probably helped a lot ;-)
Scott Budman from NBC Bay Area news dropped by to interview us on our technology (aired on Tuesday in the 6pm news, and again Sat 24th at 6:30pm; we'll add it once it's online).
Brian Crecente from The Verge also wanted to do a report on what we developed: here is a link to their article. Their video is embedded below: