This is obviously a touchy subject, but one of those things that unlike taking a photo, a recorded sound can be totally disassociated with where and when it was recorded when taken out of physical context. So in short, I don't think there's anything to be concerned about as long as you're not purposely stalking or being a voyeur. And when you have many layers of BGz in the edit, its nearly impossible to pick something out.
That said, I have a few rules I like to follow.
I personally make it a point to, as a sort of "sound effects recordist's code of conduct", to scramble or erase any conversation "callouts" which blatantly stand out in a crowd/ambience recording and are of private nature. If I'm recording and somebody notices or asks, I reassure them of the above and all has been fine. When in doubt, remove it. This is why I try to record 15+ minutes in people-heavy locations.
The other is I obliterate all brand names where possible. Rick's book make a valid point on this subject, one of very the few items in his book I actually agree with. Because if a walla track found its way into a film and it had the audible word "I hate this X,Y,Z place" or "z,y,z place sucks", you could be slapped with a defamation lawsuit. Probable? Not likely, but it's best to be safe. This is where I roll up my dialogue editorial sleeves and tweak any callout of this nature that features a brand name and either remove the callout, or tweak it so it says "That pizza" versus "That Dominos pizza" or something of that ilk.
I do the same thing when using cop radios, scramble them all up, because you never know the codes (except I've memorized that a 419 is LVPD for dead body/homicide thanks to CSI). Since every city has it's own codes and such, the last thing you'd want to have happen is during some sort of comedy scene have a code in the background that somebody recognizes as a code for rape in progress.