Indeed it's easy to generate very loud noises with a speaker system whose power is only a fraction of what's actually used in PA or HiFi systems. What's not so easy is to get any sound you want to that level, without severely distorting it.
Perhaps most relevantly for music: bass frequencies need way more power to be perceived as loud as a 1 kHz signal. And short transients can have enourmous peak power without sounding particularly loud. A bass drum has short transients in the bass range, and indeed the bass drum hits will by far dominate the power histogram in a typical live situation.
For studio records, mastering has as one of its goals to minimise the peak-power need (what they do is, maximise the loudness for a given sample peak cap); most important tool is various compressors, which roughly speaking shift some of the transients' energy into the gaps (at the cost of reduced dynamic range). But to really get the most efficiency, you'd need to do what speech application have done pretty much forever: reduce the signal for the most audible frequency range, i.e. cut both bass and treble. Then put in some nonlinearities (compressors/limiters or distortion, in doubt in the low-quality speakers) to also reduce the peaks, and voilà: you can build a fire siren with a 5-watt supply, or speak to an entire stadion with only a couple hundred of watts. It'll sound horrible though, definitely not useful for music.
