Definitely handling noise. I'm not a big fan of the little handle that came with the thing, so I jury rigged a mount 1/4" - 3/8" - 5/8"...the first is a standard tripod screw mount adapter, the second is a mic stand adapter. I epoxied them together so they don't make noise, added an o-ring at the 1/4" end to minimize noise at the h4n, and I've got the 5/8" end screwed to a table mic stand.
For handheld, I can unscrew the post from the base, wrap a foam handlebar grip around it, and viola...no noise. 3/8" female is also standard for boom poles...if you want to have your h4n on a boom.
Whatever you do, make sure you have an assortment of neoprene or rubber washers...they go a long way to minimizing sound at the connecting points.
Other solutions: I've got a tripod quick release mounting plate I can screw to the back, then set it on a tripod...hands-free = no handling noise.
One of the other miracle things is a fig-rig clamp. 3/8 AND 1/4 screws, and the inside of the clamp is lined with cork or rubber. You can clamp it to anything in the 3/4" to 1" range...lightstands, c-stands, broompoles for a cheap boom-pole...very versatile!