I know this is bordering on consumer audio/home theatre, as that's what my end goal is, but bear with me for a minute.
Goal: to output 5.1 (as 6-ch analogue) from VLC in order to use an old, semi-retired Mac Pro 3,1 as an HTPC.
Problem: Controlling individual channels in the Mac before output to hi-fi amp.
In essence, I want to be able to tame the excesses of high dynamic range movies so I don't annoy my neighbours, whilst still being able to hear the dialog when they're whispering. My 5.1 amp has a simple low/med/high compressor, but it is just nowhere near comprehensive enough.
Analogue 5.1 o/p is necessary, as it's the only input spare on the amp.
I have at my disposal, an S/PDIF to 6-channel analogue converter & a USB to 6-channel analog device. Both are cheap & cheerful, but both behave 'acceptably'.
I have Rogue Amoeba's Audio Hijack, and (not currently installed but available) Loopback.
For compression I have everything built into macOS as AU plugins & also anything by Waves. My intention is to get a solid multi-band comp going so I can always hear speech on the centre without the explosions being able to bend the walls.
This would be the 'easy bit' if I could get the routing to work.
To the issue as I currently find myself.
Using Audio Hijack I can easily route audio through any compressor to my chosen output. How this actually behaves depends on whether I'm using the S/PDIF or USB output.
Over S/PDIF I can comp all channels - but the resultant audio seems to be interpreted by the converter as some form of stereo. All 6 channels work, but I'm not hearing real placement, just some kind of poor guess.
Over USB I have 'true' 5.1 visible to the OS, however in Audio Hijack I can only comp the front left & right channels - everything else just passes through untouched.
In either scenario if I attempt to use a Waves 360 [true 5.1] plugin, I just get noise, no signal. I have to use stereo plugins to get any sound through Audio Hijack. Presumably it is seeing the AC3 as 'stereo' before decoding to surround.
Can anyone even think of anything else to try?
I can add more detail if needed.
I'm stumped as to how to persuade what feels like a simple operation to work as intended. I have absolutely no issues dealing with 5.1 in a 'pro' environment, from Cubase etc, but for this simple pass-through, I seem to be stuck.
Is there even another app, similar to Audio Hijack, that I can plug my dedicated 5.1 comps into - or should I be looking at Loopback, which tbh I have barely looked at yet.
Fabulously-drawn diagram of current & desired situation...
The ice creams are supposed to be VU meters ;)
Late additional test
I tried Loopback as a 6-channel passthrough to the USB device. It complained 'VLC has only two outputs'