I use libraries when it simply isn't possible or practical to record specific material, be it for schedule/budget reasons or because I simply cannot access them.... But I also use them when I want as big a range of source material as possible eg I had some big fire scenes in the Cirque 3D film and while I have a lot of fire sounds in my library I also recorded a fire performer recreating the exact moves.... but I was also very happy to buy Franks Fire library, to expand my options beyond what I already had.... Every recordist records differently & finds different ways of manipulating props etc... You could give the exact same prop to five recordists & get (at least) five very different sounds....
I think another important aspect of using libraries is HOW you use them. Nothing makes me cringe more than hearing an obvious sound library effect, with no editing/alteration whatsoever.. eg that metal hatch door open from Hollywood Edge makes me feel a little sick, and I kind of laugh/kind of groan every time they use it in The Simpsons since for them its a running gag.. but I hear that sound in some big budget films & think WTF? How hard is it to record something unique? But if that isn't possible how hard is it it to layer it, pitch it, process it a little or something just so its not sitting there like a sign saying: LIBRARY EFFECT #017.... (maybe the picture editor cut it in as a temp effect & they fell in love with it?)
Regardless of the source of the sound, if you aren't personally invested in it - if it doesn't have some of your character (taste/aesthetics etc) in it - then I think there is something wrong...