What could a future (software) interface for sound-design/mixing look like?

Question

Hi folks,

apart from my occupation as sound designer, my main focus (since I'm working at a university) lies in media (technology) research, software development, and (sonic) interaction design.

As such I'm excited about the quasi-revolution that is taking place in the user interface realm through the development of tablet devices and the like. Also, software companies such as Native Instruments or Ableton are currently pursuing new ways of human-computer-interaction.

Contrary to this, at the latest audio convention I attended, to me it seemed like the only innovation were the obligatory 16 channel strips added to the large mixing consoles, etc.

Since I'm somewhat rooted in both worlds, I find these developments very exciting, and am constantly thinking of how a new kind of user interface which sort of abandons the old "build a mixing console as a software package" way of doing things, could revolutionize sound-design and mixing. Without any more allusions from my side, I would really be interested in what kind of interaction you envision for your future (computer) work?

I'm also thankful if you have some thrilling examples of nice interaction design for me... I try to follow developments as tightly as possible, but there's so much going on at the moment...

All the best, Julian

Answer 1

As a professional interaction designer (and that's for context, not to set myself up as any kind of expert), I think the "future" is hard to nail down without sounding either kooky or like Ray Kurzweil (which some would argue is the same thing). However, I can offer what I've seen and heard recently that might offer some suggestions and ideas for your exciting investigation.

• For all the talk we do of placing sounds in the stereo (or any multi-channel) field, there's been no serious attempts to visualize it in a professional manner, covering spread, gain, and frequency.
• I agree with Tim that haptic feedback is sorely lacking. While one always wants to remove cognitive friction, physical friction is what makes things truly tactile. This is why I tape printmaking paper on my drawing tablet: Plastic on plastic doesn't feel anything like the real drawing or writing I do on a daily basis, and the lack of physical friction creates cognitive friction as I try to be visually inspired while managing the oddness of the experience. (Many haptic feedback actuators are driven by low-frequency sound files playing back, let that simmer in your brain for the possibilities...)
• I strongly believe that the future of interactions is a range of input methods, not just one. Not just gestural (massive fatigue and huge accuracy issues), not just keyboard (indirect, proxy control), not just speech recognition (failures in loud environments and in translation). We're in an era where we don't need to assume someone will input commands in just one way - remember that Kinect's most overlooked feature is that it's not just a gestural controller, but it does speech recog., too. That's a trend I think we'll see more of: Contextually-appropriate controllers.
• People think of touch and multi-touch in terms of touch*screens*, but that's limiting. The touch-sensitive encoders and faders on mixing boards really is a huge productivity changer. Heck, for scrubbing, I'd probably rather have a ribbon controller on my keyboard than a jog wheel.
• XY input devices like the Wacom Intuos and Bamboo, or even the XY touchpads on Novation controllers, allow for two axes of input rather than one, which is a huge think. Could the new Apple USB touchpad input device - heck, two of them at once - be an interesting model for mixing? (remember, though, that some things like faders probably still want strictly-controlled single-axis movement)
• In my opinion, all interface design should reduce the apparent distance between input gesture and the system's response, yielding at least a perception of more direct manipulation of a value, setting, or a piece of content.
• Investigate peoples' mental models of things like mixing and editing. Have them draw how they thing of sound data spatially, away from hardware and out of the studio environment. You might gain interesting insights...
• The radically new is almost never adopted by the market. Look at the QWERTY keyboard as the ultimate example. The key is to retain enough affordances from the old way of doing things that people are drawn in and somewhat comfortable, and then sneak in some innovation around the way the familiar things work, or a new thing in addition. Dreyfuss (who largely created what we now know as human factors) calls such things "survival forms:" If you design a pencil sharpener to be even vaguely pencil-sharpener-esque, people will know what to do with it, but if it looks like a featureless egg, no one will know where to begin.

Answer 2

I had a mixing lecturer who told us that our generation would be the one to make huge changes in the physical aspects of mixing (eg. faders, pots, etc.). I'm constantly trying to think about what would improve things.

One thing that comes to mind is some kind of X/Y touchscreen/tablet automation interface. You could set the X and Y axes to whatever parameter you choose (eg. volume/pan, volume/send vol, etc.). Maybe even add a third axis that's related to finger pressure. This might require greater finger dexterity, but i've been messing around with GRM plug ins lately (which have a similar X/Y axis with a dot on it that controls 2 parameters) and i find that having access to 2 parameters, with the one control, makes things more intuitive.

Also, my feet don't do much while i'm mixing. It'd be pretty cool if i could have pedals to skip fader banks or stop/play or something.

Answer 3

Force feedback is one aspect totally lacking from a lot of computer based ui design. Whether its knobs on an analogue synth, controls on a real mixing desk or playing an acoustic instrument, part of what allows people to develop skill & dexterity is friction - there is art in that friction! These empty weightless gestures on many of the new devices really do not feel like the future to me. A simple example: I'd like a surround panner on my iPad but I suspect if I get one I'll end up sticking things like plastic disks on it to provide physical friction and/or to restrict movement.

Answer 4

@Tim:

there is much truth in what you say. it reminds me of a short section in Theo van Leeuwen's "Speech Music Sound" (p. 138f):

"In this area, too, electronic sound presents a challenge, partly because it often becomes very much a 'mind event' rather than a body event, difficult to relate to physical, bodily experiences and physical, material objects, and partly because the kinds of actions we have to perform to produce them often bear little connection to the sounds themselves, as when a vibrato is produced by touching a button rather than by vibrating one's finger on the string, or the sound of a Spanish guitar by tapping the keys of a piano rather than by plucking strings. Sounds lose their materiality, their link with the actions that produce them, and as a result their meaning might increasingly come to be seen as conventional rather than motivated. If we are to keep mind and body together, and not reduce all semiotic action to moving a mouse on a mouse pad or tapping keys, we urgently need to develop new, subjectively meaningful 'interfaces' between physical actions and the sounds they produce."

I believe, though, that this gap can be overcome. Maybe I'll make a dissertation out of it :)

(sorry, this post was too long for a comment :) )

Answer 5

@Tim @NoiseJockey - Oddly enough I'm a huge fan of the surround panner install on the Digi/Avid ICON consoles. Touch screen integration (+ the joystick) as well as variable glide functions ( via variable speed/velocity glide ratio via the jog wheel + Hot Key/Keystroke control variables) and it also doubles as a Pong interface ala: some of the SSL J and G series consoles. Slick stuff. Mixing without one is actually kind of shitty and overly complex without one, especially once you get spoiled by access to one an you have to go back to multiple automation pencil drawings or multiple fader combinations.