I do use physical controllers for designing sounds. I use Pro Tools for my work environment, and Pro Tools is very bad at allowing out of the ordinary controllers to interact in meaningful ways with the main DAW features. A MIDI controller, usually a keyboard, is good for controlling plug-in instruments like a sampler for great expressive use of pressure, pitch and mod wheels, as well as various knobs and faders to control pitch, filter, sample start time, etc. Any MIDI controller can be used for this sort of thing. Plug-in maker Flux has started adding OSC control to their range of v3 plug-ins. So you can use an OSC controller like an iPad running TouchOSC or Lemur to control your IRCAM Spat panner/spatializer. However, the most generally useful controllers for Pro Tools are mixer-like, such as HUI protocol mixers and JL Cooper protocol panners, especially accessible is Neyrink V Control Pro for iPad, Eucon based controllers from Avid/Euphonix, or the various D-Command and Control devices made by Avid/Digidesign. These all look like mixers but can in fact be used to control practically any parameter in Pro Tools. They are useful for getting the real-time gesture or performance of expressive parameter modulation into Pro Tools quickly and with feeling!
Frequently controlled parameters are volume, send levels (usually to reverb or a sub-synth) and pan settings. But these are usually considered mix parameters. (It's all sound design!) It's fun to play in the more design realm with plug-ins like GRM Tools to play with bandpass filter parameters, doppler, pitch shifting, etc. Controlling these with faders and knobs while watching and hearing in real time is very powerful. Just putting the output of what you are playing with into a record track and going for possibly tens of minutes at a time can be very useful. Remaster the best bits and build a library, or perform direct to picture in sync. Sometimes I record automation and then tweak that by hand before recording the output of these sounds. The recorded sounds are cut into conventional tracks, tracks with this kind of plug-in inserts and automation almost never makes it to the mix as a live thing.
Ultimately the mouse or trackpad can be the most flexible general purpose expressive controller with the least amount of setup time. It mainly falls short in that you can only have one of them.