If I have the spectrogram and waveform of a clip of audio what sort of information can I gather by looking at those two things?
migrated from avp.stackexchange.com Jan 24 '14 at 12:01
This question came from our site for engineers, producers, editors, and enthusiasts spanning the fields of video, and media creation.
I'll start from the very beginning: sine signals.
Also known as pure tones, these kind of signals are periodic (i.e. repeat the same pattern over and over again). Anything that is periodic can be said to have a frequency, which is in fact f=1/T, being T the period. If you look at the image, you can see that the period is 2*pi and therefore frequency is 1/(2*pi) Hertz (Hz).
This one is, in fact, a time domain signal. A waveform. In this examples it's pretty easy to compute the frequency, but it isn't always like that. So the only real information you can extract out of it is the amplitude at each moment.
Sine waves contain a single frequency and, in fact, sound like this. If we apply the Fourier transform to a 200 Hz, we would get the following spectrum:
This means, roughly, that the signal's whole power is at 200 Hz. I will get more deeply into that later.
Speech, or music, are far from featuring just one frequency. You can easily distinguish a note played by an electric guitar from that very same note played by a trombone. Same musical note means same fundamental frequency, so there must be more frequency components. That would not be a pure tone, but a some of some of them. Look at what the signal looks like when you sum to sines (first row, a is time domain and c is its spectrum):
So now we have a time signal that seems periodic, yet more complicated. If we look at its spectrum, we now see two frequency components.
So let's generalize. According to Fourier's theories, any signal can be decomposed into infinite a sum of sine signals. If a sine signal had one component and its spectrum had one impulse, two sines had two impulses... infinite sines means a real spectrum, showing the most powerful frequencies as higher components. Infinite vertical bars mean "so many that you can't distinguish one from the other", so the plot appears to be a single one.
Now, let's get to practice. You can open any song in Audacity to have it plotted in time domain. If you select the whole song and plot its spectrum (Analyze > Plot spectrum...), what do you see? Well, you will probably see some sort of plot that means almost nothing. What would you hear if you played all the notes of a song at once? In the bottom row of the previous image, the two distinct sine signals happen at different moments but, as the spectrum of the whole signal was taken, it seems that it happened at the same time. So the same happened here: it's like printing a book in a single page.
So the real interest is taking a small sample of the audio (let's say, 0.1 s) and plotting its spectrum. Now we will see what's happening in that time lapse. You can, for example, see if low frequencies (bass) are too loud.
An interesting experiment is to take a brass instrument and plot its spectrum. You have samples here. As you are analyzing a single undistorted instrument playing a single note, you can see its harmonic structure: if you have taken a Tuba playing F2 (87 Hz), its harmonics would be 87*2=174, 87*3=272, etc.
But, in general, as I said, the most interesting information appears when you take a little time lapse. Wouldn't it be great if some sort of software looked into every little fragment and plotted it to you? That's the spectrogram, which shows how much power is into every frequency at every moment.
An example: this is the spectrogram of someone saying "Rice University". The spectrogram has a bunch of info impossible to obtain just looking at the waveform, i.e. the frequency components contained in each phoneme.
etc. This is this way up to a point that if an experienced phoneticist looks at an spectrogram can indeed "read" what the person said.
Here below is an image of a waveform (top part) and a spectogram (lower part).
This is a two-dimensional graph where the amplitude of the signal (your audio) is shown over time. The amplitude is interpreted as the volume, so you see your total volume progression over time.
This graph has three dimensions. X-axis is the time, Y-axis is the frequency and the colour represents the amplitude of the signal. Here below is a simple example to show you how the spectogram works.
First a 400Hz tone over the duration of 8 seconds: You see that the lightest colour (highest intensity) is on the 400Hz line of the Y-axis. Some other frequencies slightly higher and lower than 400Hz also exist in the tone, but those have a lower intensity (darker colour) and are not the main tone.
Now a 200Hz tone shows the same behaviour. Highest intensity at the 200Hz frequency: If I play the 400Hz and 200Hz tone simultaneously I see on the spectogram that both the 400 and 200Hz have a high intensity.