1st of all, if the voice coming from the speaker has some characteristics picked up by the fact that's miced and going through amps & speaker, maybe you could EQ those so they can pop a bit and create a contrast between the other sounds/people talking w/e..
If the "right next to the camera" talkings obscure the speech, sadly there's little you can do. The main thing that you should aim for is frequency separation to start with, if that isnt doable, dynamic separation is not doable either. (in your situation)
Now if all the above fails, and you really want this to happen , I would do it by hand, edit out any part that obscures the speech, it can be painful but could save some stuff.
But look for the frequency "footprint" first, see if you can boost that against the others.
EDIT after OP's comments.
To explain my answer further :
When you have a sound that you want to manipulate, for instance make it dryer (which is not such a simple task on its own) and separate it, the classic way is to boost it's unique frequency response and cut everything else.
As a simple example I could easily use a kick drum.
To separate a kick drum from a hihat, or a snare, first of all you would dump the frequencies that you don't care about, and then use other stuff to separate more, like gates or compressors..
But if you try to separate a kick drum from a bass track, it's nearly impossible. Though you will be able to only play sound when the kick drum hits (with a simple gate), you wont be able to take out the bass sound from the sample/waveform.
That's because they have very similar frequencies.
Now in the kick drum example we use a transient, a kick sound. With continuous sounds (such as talking) the job is harder by far.
The main body of the voice is somewhat close in frequency response so you end up with a midrange thing.
So even if the expander worked, when the expander would open you'd hear the talking on the main voice. Cause it just works like a gate - it opens when it listens to the sound and the sound by itself is not separated!
But the fact that you say you are sure it can get separated intrigues me, so to further advise I would like you to upload a sample of the problem. It's the only way we can have a more detailed discussion :)