Whilst being true that you shouldn't limit yourself to one or another level, there is still a level you should be able to come back to. You should have a read through Equal loudness countour curves, or phons.
This are some curves developed some time ago (became known as Fletcher-Munson) that shows how messed up our hearing really is. It shows that our perception of high an low drastically change depending on levels. The higher you go, the flatter this is.
Now, since OSHA was so kind as to publish tables of how much we are damaging our ears and for how long we should expose them to certain levels, it's impractical to use a nice, flat curve of ie. 110dB, but I tend to go for something like 90dB.
If you are lucky enough to mix with something with a nice control room section (most modern NEVEs and SSL, most euphonix, D-Control/Command etc.) they will allow, for the same reason, for you to set a calibration level. It is good to reference at higher and lower levels now and then but you should have a common point to return to that you will be familiar with and confortable with. When I'm mixing in desks without this, I often take my lovely £20 SPL meter to have as a quick reference for the same exact reasons.
90dB SPL A-weighted seems to be a good point to stay at imo.