Definitely handling noise. I'm not a big fan of the little handle that came with the thing, so I jury rigged a mount 1/4" - 3/8" - 5/8"...the first is a standard tripod screw mount adapter, the second is a mic stand adapter. I epoxied them together so they don't make noise, added an o-ring at the 1/4" end to minimize noise at the h4n, and I've got the 5/8" end screwed to a table mic stand.
For handheld, I can unscrew the post from the base, wrap a foam handlebar grip around it, and viola...no noise. 3/8" female is also standard for boom poles...if you want to have your h4n on a boom.
Whatever you do, make sure you have an assortment of neoprene or rubber washers...they go a long way to minimizing sound at the connecting points.
Other solutions: I've got a tripod quick release mounting plate I can screw to the back, then set it on a tripod...hands-free = no handling noise.
One of the other miracle things is a fig-rig clamp. 3/8 AND 1/4 screws, and the inside of the clamp is lined with cork or rubber. You can clamp it to anything in the 3/4" to 1" range...lightstands, c-stands, broompoles for a cheap boom-pole...very versatile!
Adapters:
http://www.bhphotovideo.com/c/product/554344-REG/Manfrotto_120_120_Converter_Plate_for.html
http://www.bhphotovideo.com/c/product/305934-REG/AKG_KM216_BLACK_3_8_Male_to_5_8_.html
Stand:
http://www.bhphotovideo.com/c/product/154670-REG/K_M_23200_500_55_Table_Top_Microphone_Stand.html
Fig-Rig Clamp
http://www.bhphotovideo.com/c/product/381895-REG/Manfrotto_595CLA_595CLA_Fig_Rig_Clamp.html