I recommend you to start with pulse-code modulation files (.WAV, .AIFF, etc.): https://en.wikipedia.org/wiki/Pulse-code_modulation
What is stored in a typical CD-quality PCM file is nothing more than a few metadata markers, and a trunk of stereo data kept as numbers(note, they may be floating numbers or integers).
You can view PCM files as a "dump" from the internal buffers of an audio interface. For example, when the audio interface runs at 44.1Khz/16bit, you simply read short[...] (because it's stereo!) from a
.wav file and dump them into the audio interface and it will just work.
Depending on your OS/audio stack you may need different libraries for sending the "dump" into your soundcard. I had experience with https://github.com/thestk/rtaudio and it's cross-platform.
Anything beyond PCM will need a good load of math prerequisites to understand. But if you strictly follow the specs, doing every step correctly you'll be able to wrap up a decoder without understanding all the details.
You will be first decoding the stream, which effectively turns
.wav (incrementally), and throws the
.wav segments into the audio interface.
Update: btw, the container-codec model is almost universal in audio/video file formats. If you dig deeper into the
WAV format you'll find interesting stuff. :)