Because of where the wire is, I generally wear my headphones with the one marked left over my right ear (and the one marked right over my left). Does that matter?
I can see how it would for film or computer games, but for music?
Uh oh, don't do that... the stereophonic inversion police aren't up for any fun...
Truth is, many headphones are designed almost symmetrical, so if you turn them around and flip the stereo channels you get exactly the same experience. Flipping the channels is easy enough if you're working with a mixing console. In consumer devices there's seldom such an option, so unless you don't mind an extra patchbay or re-wiring the headphone plug, the stereo channels will be “wrong”. Now, as you conjecture this is usually not a big deal – music is mixed in stereo, but the choice of sides is pretty much arbitrary. It will feel a bit weird to classical-music listeners if the 1st violins are coming from the right, but it should hardly be essential to the listening experience.
In particular HiFi headphones often are not symmetrical, so it's simply uncomfortable to wear them the other way around. But it won't be, like, harmful.