Is it better a wall close to the sound source or far from it, for max db reduction?

Suppose a sound has to go from point A to B and there is a wall in between:

A -------||--------->B

Is the volume of the sound reaching B louder if the wall is close to A or if the wall is close to B?

The questions come from the fact that in my flat I can hear people shouting in the street, but they cannot hear me shouting. How is that possible? The distance is the same. The only difference is that people in the street is not as close as me to the wall. Does anybody have an explanation?

Two factors come into play.
I'm not going to do the math [never was my strong suit], but if you understand that sound level reduces in a logarithmic fashion - double the distance, quarter the volume*, that will help.

Sound travels from your noisy street to your wall. All that limits it so far is the distance.
Then it hits the wall, which cuts it by let's say 90 %.
That 10% volume has mere feet to lose energy as it comes the rest of the way to you.

Consider it the other way round.