It's not the USB MIDI that is where the delay lies, it's the generation and digital to analog conversion of the sound. The amount of data that has to be processed to output a half second of audio is vastly greater than the amount of data processing needed to change a part of the display to show a character from the code page or show the mouse pointer in a different place.
Also note that changes in optical stimulus that take place in less than 1/30 of a second create the illusion of smooth motion, while the human ear can decode differences in two audio signals that differ in the range of 1/20,000 of a second.
Finally, when you are typing or moving the mouse, you are not trying to synchronize the response of the computer with anything external, so there might be latency that you don't even notice. When playing music, we are usually either trying to have the sound appear on the beat of a metronome or in our heads, or we are trying to synchronize with other instruments. It's much easier to hear the latency when you're playing with other than when you're just playing alone and are not trying to hit a precise time when you depress a key.