Nature recording is a really tricky place to start with cheap gear, as ambiences tend to be reasonably quiet and cheap gear nearly always has a high noise floor! I've found most quiet nature ambiences I've recorded with a handheld recorder either with the built in mics or external mics are almost unusable because of the noisefloor.
One response is to get cheaper gear and concentrate on louder sources that you can record without worrying about noise...
However I've recently discovered that cheap, plugin power electret omni mics have very little noise and very high output, meaning they can be used with a cheap handheld recorder to get pristine ambiences! Although it's hard to reject unwanted sound with omnis, it's a mix technique championed by Chris Watson who is one of the UKs top wildlife sound recordists.
So if you add to whatever handheld device you decide on, some Clippy EM172 or Lom MikroUsi and some bubblebee wind protection, you'll add the ability to record really quiet sources for about $100
FWIW I've felt let down by almost all the built in mics I've heard apart from Sony, but Sony's don't have xlr which is frustrating when you want to experiment with external mics! I might suggest starting with a used Sony, then jumping to a SD Mixpre or Zoom F4 once you want to start using external mics.