Which one of the lossy file formats like Ogg, AAC, Mp3, Wma has the best quality/size ratio? I've seen that AACs are smaller than MP3 at the same bitrates. Do they sound worse than MP3?
You may find a lot of subjective answers here, and some of the quality seems to be dependent on what you play the files on - for example AAC and low quality mp3s will be fine on your iPod through headphones, but pop them on a decent system with good speakers and they will sound crap (Skeptics question here)
Each of the lossy formats has its own way of deciding what to lose (the wonderfully named psychoacoustics) and you may prefer one over another.
I don't like the way AAC, Mp3 or WMA compression sounds on low or medium bitrates, but on high I can't tell the difference between them. Ogg I actually like slightly less at low bitrates than the others, but morally I like it :-)
So I use mp3 or ogg in all my audio players, at high bitrate/quality settings (or flac in the devices that support it)
A good place to look for this type of info is Hydrogenaudio. But... because this is so subjective (quality) it ultimately comes down to doing some listening tests yourself.
As a purely pragmatic thing, I'd use .mp3 for lossy compression as it's established itself as the gold standard for better or worse.
I personally think that WMA or Ogg sound better at lower bitrates than MP3. At higher bitrates I don't notice as much of a difference.
For my MP3 player I tend to encode everything at the smallest file size I can without constantly hearing the compression artifacts.
I currently am using Ogg at compression level 0 which is roughly equivilant to 64 Kbps. The only thing I don't like about the format is that Windows Media Player does not like it, so I must add my music to my MP3 player manually instead of being able to sync using WMP.
You have asked a question which DOES have a mathematical answer.
Here are a list of popular lossy compression methods: https://en.wikipedia.org/wiki/Lossy_compression#Methods
Here is a picture which shows the results of a listening test of compression quality as a function of bitrate comparing a few different algorithms.
Because the image above was generated with listening tests it is still completely SUBJECTIVE.
Someone COULD write software, and a paper where each algorithm is run on very high quality versions of the top 1000 radio songs of the last decade at different bit rates. Then a comparison could be made of the sound waves before and after compression with dumb monte-carlo integrals. The algorithms which on average have a smaller integral difference, are better.
Writing such a paper would be very useful for the public, but not worth any individual's time, because any given person is better off building a new algorithms rather than doing robust analysis on old ones. (There is little personal gain to be had answering this question)
To properly answer your question, someone has to write the unlikely math paper I just described above.