I love thinking about ways to make recorded music more engaging — more interactive. And I’ve posted before about ways to make music interactive — including modern sites like 3 Dreams of Black, more fluid experiences which border on synaesthesia, playful conducting for kids, and old-school music cd-roms.

So it was really cool to stumble across Motiv, Russ Maschmeyer’s recent thesis project at SVA. It looks full of potential to change how we experience music.

Motiv uses a Kinect camera to track a user’s body movements, and then takes those inputs to adjust playback attributes of a musical piece. The goal is to give “direct control of emotional expression by interpreting their physical gestures in real-time.” The potential of using your whole body (or maybe something as simple as a nodding head) to experience music offers a physical involvement with the piece — something described by a research subject in his thesis as “groove.” Way better than the intellectual approach many other solutions use.

I wanted to know more, so asked Russ a couple questions about Motiv, how it’s used, and what he sees in its future…

Q: Could Motiv be used to track dance moves? (ie. something less specific than conducting gestures.) Could that sort of more loosly-defined input could work?

That’s almost exactly what MOTIV was designed to track: dance in a very loosely defined way. With the right composition, a choreographed dance would work extremely well with MOTIV since it uses the velocity and magnitude of your movement to alter the emotional impact and intensity of a musical moment.

Q: Do you have a sense of whether Motiv would be more useful as an educational tool (to teach musical principles) or as an “playback” platform (to interact with existing compositions, and experience them more deeply)?

I’ve designed MOTIV as a performance platform, to take a sequenced piece of music and elevate its playback to allow for emotional improvisation in the moment of performance. Though I think it’s also extremely useful as an educational tool. I was really happy with how easily non-musicians seemed to pick it up and actually make some really nice moments happen in a relatively short amount of time—in just minutes with some. You can’t really say that about any other instrument (not that I’m really comparing this to a more traditional instrument). So I think it could go a long way to helping people understand the difference between playing music and playing music expressively.

Motiv: workflow

Q: How do you imagine artists, musicians, and labels might use a platform like Motiv to make songs (or pieces) more interactive?

I think once it’s capable of handling multiple users at once, the possibilities become exponential as to what can be controlled and how the parameters of control interplay. Imagine one player being able to control tempo and note velocity, while another player controls the instrumental layering at any given time and has special gesture cues for hits and flourishes, while a third controls the key, octave and arpeggiation patterns. All the while, the musical data from each player is shared with their co-performers in real time to create a synchronous musical experience. Player two’s hits and swells are echoed across player three’s actions and vice versa. It’s gets pretty rad.

Configuration interface in Max/MSP

Q: How much do you imagine the user would want to tweak the movement input parameters?

A: I think there are some basics that just very intuitive, notably that the speed of your movement maps to MIDI note velocity (loudness), but there are many that are a bit more up in the air like tempo for instance. I played around with a conductor’s tempo gesture at first, but found that ultimately it was very difficult to both accurately track the tempo and make the actual down beats line up with your gestures at the same time (it’s a mathematical average frequency determination issue), so I went with a much simpler tempo slider based on the position of your left hand. That was entirely arbitrary and worked mostly because of its simplicity. The instrumental layering is currently controlled by the overall movement of your whole body in space. The more your jump and twirl the more layers of instrumentation are added in. That works well, but I could certainly see room for customization depending on the needs of the performer.

Be sure to also check out his thesis book, which offers a great view into his thinking, research, and process for the development of Motiv.

Earlier concept

What is it like to conduct an orchestra? The easiest way is find out is to just try it yourself. Which is what this cute project from Magnetic North lets kids do. Wave your arms around and see how the orchestra adapts. It looks like a lot of fun. My only question is if it’s too unstructured an experience. Could there be a layer that helps guide kids to learn?

The project was part of the Manchester International Festival Music Boxes, short-run a melodic adventure created for children aged 6 months – 7 years. It looked like a great event!

(Link via Creative Review, where’s there’s also more info about the project.)

There’s been a lot of Kinect-based experimentation lately, and, like so much of it, this new research project from the MIT Media Lab looks pretty cool. But it’s more than just another Kinect-hack — it’s full of intriguing ideas of how video-conferencing can adapt if it’s more than just a “dumb” carrier of video. It asks the question, “what would video conferencing be like, if it knew about the participants?”

The team has come up with some pretty interesting concepts. “Talk to Focus” shifts the visual focus of the system to whomever is talking — so that non-speakers don’t distract you. “Freezing Former Frames” lets participants text, or step out of the room, without being noticed by the other side, or appearing rude. Plus there’s also “Privacy Zone” and “Spacial Augmenting Reality.”

Did I mention that the demo video is pretty cute?

More information is on the project website kinectedconference.media.mit.edu/.

Freezing Former Frames

Talk to Focus

(Link via Infosthetics.)

Kinect Gestural UI: First Impressions

From labs.laan.com

Microsoft’s just launched Kinect is generating a lot of buzz right now. But it’s not the platform’s games that are most exciting — instead it’s the implications for future gesture-based interfaces. For example, check out Wes Keltner’s article about the possibilities for Gesture-Based Advertising.

But I think what’s even cooler is the degree to which designers and programmers are working to hack Kinect and invent all sorts of new applications for the technology, and new way to think about gesture interactions. Microsoft seems to be pretty unhappy about this, but maybe, instead, they should embrace this energetic culture.

Much of the work is based on software which is downloadable from the OpenKinect project. People have been posting videos of their work, and it’s a lot of fun to check out. There’s something about the whole-body interaction that has shades of Myron Kruger.

Engadget has a good post about some of this early exploration for capturing motion and 3D data. And there’s tons more out there to explore, such as the YouTube channel KinectHacks.

Golan Levin hacking Kinect

UPDATE: Here is even more…