I love thinking about ways to make recorded music more engaging — more interactive. And I’ve posted before about ways to make music interactive — including modern sites like 3 Dreams of Black, more fluid experiences which border on synaesthesia, playful conducting for kids, and old-school music cd-roms.

So it was really cool to stumble across Motiv, Russ Maschmeyer’s recent thesis project at SVA. It looks full of potential to change how we experience music.

Motiv uses a Kinect camera to track a user’s body movements, and then takes those inputs to adjust playback attributes of a musical piece. The goal is to give “direct control of emotional expression by interpreting their physical gestures in real-time.” The potential of using your whole body (or maybe something as simple as a nodding head) to experience music offers a physical involvement with the piece — something described by a research subject in his thesis as “groove.” Way better than the intellectual approach many other solutions use.

I wanted to know more, so asked Russ a couple questions about Motiv, how it’s used, and what he sees in its future…

Q: Could Motiv be used to track dance moves? (ie. something less specific than conducting gestures.) Could that sort of more loosly-defined input could work?

That’s almost exactly what MOTIV was designed to track: dance in a very loosely defined way. With the right composition, a choreographed dance would work extremely well with MOTIV since it uses the velocity and magnitude of your movement to alter the emotional impact and intensity of a musical moment.

Q: Do you have a sense of whether Motiv would be more useful as an educational tool (to teach musical principles) or as an “playback” platform (to interact with existing compositions, and experience them more deeply)?

I’ve designed MOTIV as a performance platform, to take a sequenced piece of music and elevate its playback to allow for emotional improvisation in the moment of performance. Though I think it’s also extremely useful as an educational tool. I was really happy with how easily non-musicians seemed to pick it up and actually make some really nice moments happen in a relatively short amount of time—in just minutes with some. You can’t really say that about any other instrument (not that I’m really comparing this to a more traditional instrument). So I think it could go a long way to helping people understand the difference between playing music and playing music expressively.

Q: How do you imagine artists, musicians, and labels might use a platform like Motiv to make songs (or pieces) more interactive?

I think once it’s capable of handling multiple users at once, the possibilities become exponential as to what can be controlled and how the parameters of control interplay. Imagine one player being able to control tempo and note velocity, while another player controls the instrumental layering at any given time and has special gesture cues for hits and flourishes, while a third controls the key, octave and arpeggiation patterns. All the while, the musical data from each player is shared with their co-performers in real time to create a synchronous musical experience. Player two’s hits and swells are echoed across player three’s actions and vice versa. It’s gets pretty rad.

Q: How much do you imagine the user would want to tweak the movement input parameters?

A: I think there are some basics that just very intuitive, notably that the speed of your movement maps to MIDI note velocity (loudness), but there are many that are a bit more up in the air like tempo for instance. I played around with a conductor’s tempo gesture at first, but found that ultimately it was very difficult to both accurately track the tempo and make the actual down beats line up with your gestures at the same time (it’s a mathematical average frequency determination issue), so I went with a much simpler tempo slider based on the position of your left hand. That was entirely arbitrary and worked mostly because of its simplicity. The instrumental layering is currently controlled by the overall movement of your whole body in space. The more your jump and twirl the more layers of instrumentation are added in. That works well, but I could certainly see room for customization depending on the needs of the performer.

Be sure to also check out his thesis book, which offers a great view into his thinking, research, and process for the development of Motiv.