It's pretty simple actually, though it depends on the specific issue. The main thing is a combination of naturally having skills to deal with bad video and the extra information video provides.
If video is a little bit shaky or the color is a bit off or the angle is bad, these are all things that we deal with all the time in real life. Your eyes are always dealing with your head moving around. You are always adjusting for changes in lighting color, often even within one environment. You don't always see things from an ideal angle. You have a lot of practice naturally dealing with many of the things that make video "bad".
Similarly, if detail is missing from video, it's pretty easy for our brains to adjust for detail loss from compression artifacts and such because we are good at matching patterns of what we see and filling in the details. Really, this is what persistance of vision and our ability to move our eyes around without it being a hugely jarring experience does all the time (not to mention hiding the giant blind spots where our nerves enter the back of our eyes).
On the other hand, for audio, we have drastically better ability to perceive sound than we have to play it back. We have quite good positional noise filtering capability, so we can normally make decent sense of a noisy situation, but when it is recorded and then played back through a few speakers, the noise and the audio we are interested in are coming from the same place. Our normal tricks for filtering don't work and so we struggle.
Additionally, our senses don't work completely independently of each other. We are very good at combining what we see and what we hear and thus it is jarring if sound doesn't match up with what we are seeing (such as with out of sync audio or bad foley).
Many of the issues that can plague audio simply are things we are either not well adapted for or are actually outright adapted against. For video, it is the other way around. Our natural adaptations aid us rather than work against us.