I'd say "All of the above"
SSD/RAID for storage [with appropriate backups on & off-site]
I put my most-used sample sets on SSD & also record to it - with backups, of course, as I don't actually trust an SSD to live out the day ;-) So far, so good, I haven't lost one yet, but I like to be sure.
Fast architecture/CPU, high core count. [background was in an NDA conversation so I can't give details of 'how I know'] but audio apps use cores very well, with multiple plugins, audio streams etc.
I don't know which audio spec is fastest; I use ASIO with I would guess CoreAudio behind that somewhere, being Mac-based. I've never used ASO4ALL. There is reasonable certainty that the proprietary systems would be slicker than the open version, but that's really just speculation on my part.
The new 'trash-can' Mac Pros are supposed to be game winners at the moment, I've heard from some reliable sources - but boy, they're expensive. Add in a Thunderbolt RAID & it's dearer still.
Ultimately, the faster the machine can process, the lower you can set the latency - & for more of the overall recording/mix process - without it starting to break up.
I always have to raise the buffering/latency long before I'm getting towards the final mix, & by the time I'm at the final stage, I'm at maximum; so I tend to put down the latency-sensitive stuff at the start of the project, or switch off a lot of the higher-cost processing if I have to go back to do overdubs later.
Drive speed won't affect latency, per se, but will affect how much data you can throw around before the machine is waiting too long for the next bit to play.
That would be more noticeable with an orchestra of samples than a few tracks of simple playback..
Latency itself is simply 'how much buffer' - a dial-in setting; the less buffer the lower the latency - but at increased CPU stress, until the point that audio starts to break up. Therefore, faster computer, lower latency, simply because it won't break up at that setting.