And if not, why is there a switch on the preamp? Why not just send phantom power all the time?
It depends. On a lot of things, but mostly on the particular microphone. Phantom power in general is not evil for almost all dynamic mics who don't need it, but if you for instance plug a ribbon mic in a phantom powered input without switching it off first, you might just destroy the microphone completely.
You will most likely find the answer inside the documentation of your particular microphone. :) If not, please share with us which microphone you plan doing this on, so we can help you further.
Phantom power is usually relatively harmless. Most of the danger consist when people have their mic-lines normalled on the patchbay and are crosspatching.
As said, refer to the documentation on that particular microphone. As a rule of thumb older ribbons might be completely unprotected (ie. absence of transformer or some kind of protection diode (quoting Daniel Rumley here :) ) but most modern dynamics, generally all the condensers - even the ones with their own psu - and even modern design of ribbon microphones will be alright with a constant feed.
why do they have a off button? again, you might have to use a patchbay to crosspatch, you might be in a live scenario where a second console is introduce - although if the setup is expensive enough you'll have some really expensive splitters that will make this not be an issue - and it's a good practice to power down +48dc on microphones before powering them off.