First a short historical notice.
In the old days, professional headphones often where 600 Ohm, sometimes 250 Ohm. The IEC standard from 1996 recommended an output impedance of 120 Ohms for the headphone amp. This worked perfectly as the supply voltage in the headphone amplifier often was 20V or more. The value of 200 Ohms output impedance on the HX3 worked very well with that era of headphones.
Then came the consumer stuff, such as the iPod. Here the headphone amplifier only had a few volts to work with. This meant that old type headphones only could be driven to a low sound volume. Of course, as the consumers where many and had money, we started to see a lot of low impedance headphones. So much that low impedance now is more or less the standard. The low impedance headphone could be driven by the lower voltage to a high sound volume and the headphone amplifiers where modified to have a lower output impedance.
Now, what happens when you use a headphone amp where the output impedance of the headphone amplifier is larger than the impedance of the headphone? The first effect you will hear is that the sound volume will be higher -- often easily adjusted by turning down the volume. The second effect is more variable, depending on a lot of factors. As the headphone will be less dampened by the amplifier, it may create various types of sound "distortion". It could be variations in frequency response or harmonic distortions of various types. Highly dependent on the exact headphone and I do not believe there is any general answer for all situations.
The reason for the "distortion" is that the headphone amplifier is designed and optimized to work together with a type of headphone. This will include optimizingg the volume of the various cavities in the headphone, the dimensions and material used in the sound membrane, electrical components and so on. How these design decisions will interact in a different setting cannot really be generalized. Also remember that some headphones are designed to intentionally distort the sound, example may be "bass boost" on some headphones. How this bass boost will react to a different output impedance of the headphone amp is not really possible to predict.
(There are a lot of "crap" headphones out there, made only to sell cheaply. These will sound bad regardless but that is something different).