It depends entirely on what you're doing. If you need it to respond very quickly (for example, if you are a musician and you want to monitor what you are playing in realtime!) then low latency is extremely important. I find that my classically-trained pianist friend starts to have a lot of trouble at around 10 to 16 ms, whereas I am more of a beginner on my instruments and generally have no problem below 16 ms.
But if you're just mixing or tweaking effects or something where you can handle a bit of a delay, much higher latencies are sometimes desirable because there's less likelihood that you'll run out of buffer - you can start loading up some more CPU and memory intensive effects without worrying about hearing pops and skips.
From my experience, more than 20 ms is generally not considered acceptable by anybody.