My list of sounds to record on set is as follows (in order of priority):
If I only get to capture the dialogue, it is still a good day. If I fail to capture the dialogue, because I'm busy recording less important sounds, I'm not doing a good job as sound recordist. It is THE most important part of recording on set, and it is the most expensive sound to reproduce afterwards.
2- Wild tracks of dialogue
If for some reason I don't get a single clear take of the dialogue of a scene, I will ask for permission to do a wild track of the dialogue of the scene. This is not something which should happen with every scene, because you have to do it on set, just after the scene, while the scene is still fresh in the memory of the actors. You also have to keep everybody on set silent, so if you do it too often, people will just get tired of you and make it harder for you to capture clean dialogue in the first place. I do wild tracks less than once per day.
3- Walla from actors/extras
This Can also be pretty hard/expensive/impossible to do in post, so you should record this. Extras should be miming when in talking in the background, so you need their sound at some point. This can be planned in advance, but you have to keep reminding the AD of it, because sometimes they forget you and send all the extras home before you get your recording :-(
Try getting the sounds that are unique to the location. Background that may set a tone or be an interesting element in the sound design. I always carry a couple of zoom H4 and H2 recorders, so i can quickly go and pick up some interesting sounds, without having to carry my main recorder, which is set up for recording dialogue. Sometimes I record ambiences on the Zooms while the crew (and I) are shooting the scenes, because people are quiet at that time. I would also use the Zoom to get an interesting perspective on the action in the scene, for instance by mounting it close to the camera to get a point of view sound in chase scenes. All of this I try to do without interfering with the rest of the crew, sometimes delaying my lunch break by 10 minutes or so.
I don't spend any time recording IR's and I am generally not very interested i recording doors, but maybe that's just me being more of an ambiences kind of guy...