Play around with the ASIO buffer size, assuming you're on Windows. If you're on OSX, it's Core Audio instead of ASIO, but same idea.
Generally, ASIO4ALL (and actually, any low-latency audio driver) will expose a parameter for how large the ASIO buffer is. The buffer is a buffered set of samples between your sound hardware and your computer. The idea is that as your computer jumps between tasks, as computers do, there's a safety net in case the computer doesn't get around to processing more sound data. This prevents audible clicks, pops, and start-stop stuttering when recording or playing back audio.
The shorter the buffer, the shorter the latency, at the risk of an interrupted signal. You can experiment with different settings to see how low you can get the buffer before your computer is too interrupted by general use to keep it stable.
I don't know exactly why this is, but different combinations of audio hardware and computers seem to have different minimum useful buffer sizes. As an anecdotal example, my previous computer couldn't get anywhere under 30ms round-trip (input + output) without stuttering horribly, but a different computer using the same interface was getting closer to 16, and I've since switched out audio interfaces and now can usually get around 11 unless I've got a lot of CPU or memory intensive software running.