I have always been under the impression that it had to do with the type and construction of the voice coil. And the magnet size contributed to the dynamics and tone of the speakers functioning (hz. etc). With formers like paper vs. Nomax, it definitely changes the amount of heat dissipation, which lowers or increases the power handling capacity of the voice coil/speaker. Paper is the lowest I believe and nomax is a higher power material for formers. Now I hear about aluminum and other metal combinations for the voice coil former, that takes the power handling way up. Also, the type of wire used for the coil winding can effect power handling as well.
However, recently someone told me I was incorrect in my understanding and that the size of the magnet had something to do with the power handling....the person was from a speaker recone company. Unfortunately, I don't know them or how reliable they are, so I can't verify their expertise. I am asking the same question on several sites so maybe I will get clearer information.
That's my two cents. Hope its helpful.
PS. I just found the information below on the Jensen website, so take it with that background:
The wattage of your speaker is primarily determined by the diameter of the voice coil and the weight of the magnet. Generally a larger voice coil and heavier magnet can handle more power. Your amp set-up is generally most efficient when you match the wattage of your amplifier with the wattage of the speaker. This is ideal, but in some cases you may want a speaker with a higher wattage than the amp. This would lower the overall amp output, but may increase the low frequency response.
Though it is not recommended (due to the potential for speaker malfunction) you can use a lower wattage speaker in a higher wattage amplifier to achieve higher speaker distortion. Keep in mind that the total power handling wattage is the sum of all the speakers. For example, two 50 watt speakers can handle 100 watts from the amplifier.
The three different types of materials used in speaker magnets are Alnico, Ceramic (Ferrite) and Neodymium. Each material has a different effect on the tonal characteristics of the speaker.
Alnico, the original magnet material used in speakers, produces a classic tone. Alnico magnets tend to be a bit more expensive due to their cobalt content. These speakers sound warmer and sweeter at lower volumes and many musicians feel they react more quickly to the player's touch.
Ceramic magnets were developed as an inexpensive alternative to Alnico. These speakers have a few advantages: they do not cost as much, are more versatile and create a wide range of tones. Both the Vintage and MOD™ ceramic speakers tend to weigh more, generally handle more power and sound better at high volumes.
Neodymium is the newest material being used for speaker magnets. Price-wise they fall between Alnico and ceramic magnet speakers. These speakers respond to a player's touch much like Alnicos and they have a well balanced frequency response. The major advantages of these magnets are weight and efficiency. A neodymium speaker weighs about 50% less than other speakers without giving up power or tone. Neodymium speakers are especially good in large, heavy amplifiers or amplifiers that have more than two speakers