Once a signal is digitized it is treated as a number (as you quite rightly point out) and for 16bits the range of numbers are -32768 to +32767. The numbers are created by an analogue to digital converter.
The analogue to digital converter (ADC) will have a maximum input range from -X volts to +X volts (i.e. real signals that you could measure inside your mixer or on the input to a PC sound card).
This range of real signals is translated to digital numbers and if the real analogue signal exceeds what the input range is (volts) you will get a limit number of -32768 or +32767 i.e. the signal will be digitally converted but clipped to these limits.
And you are right in your assumption - a number that is -32768 or +32767 is regarded as 0dB.
So, if you had a repetitive signal that brushed against top and bottom limit on your digital VU meter (as per in a DAW or wave editor) it would display the peaks as 0dB. if the signal were numerically half the size i.e. -16384 to +16384, the VU meter would display the peak as -6dB.
A quarter sized signal would be -12dB. In fact, each time the amplitude in numbers (or volts for that matter) drops to a half, the dB level drops by 6dB. This is a rough guide to calculating other signal's dB amplitudes.
On playback, a digital to analogue convertor turns the numbers into voltages that ultimately go to your speakers (after an amplifier). The loudness totally depends on how loud you want it to be - volume controls, bigger amps etc determine this.
And yes, the minimum possible amplitude corresponds to silence and, for a 16 bit ADC, will be -96dB. It's no coincidence that a 16bit signal has potentially 16 x 6dB (96dB) of dynamic range. A 24 bit signal has potentially 24 x 6dB (144dB) of dynamic range.
EDIT - it's might be worth mentioning that back in the early (ish) days of telephony, Bell Laboratories conducted experiments; real folk were asked to judge perceived sound level intensities.
The result, for mid-spectrum sounds (1kHz area), for a halving of perceived loudness became the "Bel" and, one-tenth of a bell is a deci-bel. It was found that this tied in nicely with the fact that ten times the power of a sound causes a perceived doubling in loudness.
10x power = 1bel = 10 decibels increase. This is another way of judging sounds - if your peak signal reduces by 10dB then it has approximately halved in loudness (mid bands of spectrum).