What exactly is a sample anyway?

I have watched many videos, purchased courses, and programmed a few audio visualizers, but something is still not clicking and it's causing tons and tons of confusion for me. And I think it boils down to the word "sample". Either I am not understanding something, or different people use it in different ways. Or it has multiple definitions depending on context, which makes things worse when trying to learn.

Let me start by a) describing what I'm trying to do and b) what I think I know. And then I'll ask my question.

As I briefly mentioned earlier, my goal is to create an audio visualization framework in a video game I'm working on. More precisely, I want to have a list of values where each value is between 0 to 1 and represents a range of frequencies. It's an extremely common representation of audio in many places (see figure 1). Little did I know that it's actually not as straightforward to produce, and that's because, unlike my very early assumption, audio is not stored that way.

Now, given how common this is, there are a plethora of tutorials that walk you through how to do it. But it is precisely those tutorials that seem to cause me more confusion. The code works, I am able to produce the visual I want, but as soon as I need to fix a bug, or alter its logic, I just am lost.

So what do I think I know?

I understand that ultimately audio is stored as a series of values(?) that represent a sine wave. I understand that a sine wave is the sum of multiple sine waves, each of which are of different frequencies (figure 2). I understand that in order to represent a frequency, you have to sample at least double that amount (i.e. if 20kHz is the limit of human hearing, you need to sample at least 40kHz + some room for falloff, thus 44kHz being the standard). The video that explained it made if very clear why we need double. I understand that we use Fourier Transform to deconstruct a single since wave into the multiple sine waves it consists of (more specifically, we receive the coefficients(?) or some other part of the equation of the frequency that we can later mathematically reproduce the sine wave). I understand that bitdepth is how close to the sine wave a value can get (and that anything above 24bit is unnecessary). I kinda think I understand what windows are, but this is where I think I'm starting to lose it.

So, from all of the above, and because I really like representing ideas visually, I will start with a visual representation of my understanding. The best representation I've found that relates samples to frequencies is a 3D spectrogram like this: Moving from left to right (X axis), is time. From bottom to top (Y axis) is amplitude. And front to back (Z axis) is amplitude of a specific frequency.

So, in my understanding, when we sample, we are basically recording a single value from 0 to MAX_BITDEPTH that represents the sum (or average?) amplitude of all the frequencies at that very specific moment in time. Which, on its own, doesn't tell us much. First, we need a wave, and second, we need enough samples of that wave to be able to deconstruct all the frequencies we care for. So, for 20kHz, we take 44,000 samples in 1 second. Right? So the minimum amount of time we need to deconstruct anything, would be 1 second. Right? And because we know that not all sounds start and end at the exact same time, sometimes we... (this is where I start losing things) use multiple samples to represent one? Overlap different samples? Wait... what is a sample again?

I've lost it.

If 44k samples is required to represent 20kHz frequency, then why do so many code examples call an API that returns 512 samples? For example, Unity has a method to receive "spectrum data": AudioSource.GetSpectrumData. This is their example code:

using UnityEngine;

[RequireComponent(typeof(AudioSource))]
public class AudioSourceGetSpectrumDataExample : MonoBehaviour
{
void Update()
{
float[] spectrum = new float;

AudioListener.GetSpectrumData(spectrum, 0, FFTWindow.Rectangular);

for (int i = 1; i < spectrum.Length - 1; i++)
{
Debug.DrawLine(new Vector3(i - 1, spectrum[i] + 10, 0), new Vector3(i, spectrum[i + 1] + 10, 0), Color.red);
Debug.DrawLine(new Vector3(i - 1, Mathf.Log(spectrum[i - 1]) + 10, 2), new Vector3(i, Mathf.Log(spectrum[i]) + 10, 2), Color.cyan);
Debug.DrawLine(new Vector3(Mathf.Log(i - 1), spectrum[i - 1] - 10, 1), new Vector3(Mathf.Log(i), spectrum[i] - 10, 1), Color.green);
Debug.DrawLine(new Vector3(Mathf.Log(i - 1), Mathf.Log(spectrum[i - 1]), 3), new Vector3(Mathf.Log(i), Mathf.Log(spectrum[i]), 3), Color.blue);
}
}
}

Notice that is passes a float array of size 256. It is important to know that this is a method of AudioSource, which is responsible for playing audio. So while audio is playing you may call this method at every tick of the game engine, and receive "spectrum data" of that... moment? Of the last 256 samples? Wait... what is a sample again?

I've lost it, again.

When I tried to watch a YouTube tutorial that explains audio visualization, the person said:

"The complete spectrum of Fast Fourier Transform, in this case, could be about 20,000 Hertz. So it would create 20,000 different samples. And that's a bit too high for our game to work with"

source: Audio Visualization - Unity/C# Tutorial

Wait, what??

...

Further down the tutorial series, the author tries to explain how, instead of having 1 bar for 1 frequency, we want 1 bar to represent multiple frequencies. Perfect! That's what I want!! And then they start using the word "sample":

"the song we are playing has a hertz of 22,050. and we have 512 bands. which, if you divide this, it's about 43 hertz per sample"

(╯°□°）╯︵ ┻━┻

Clearly, there is something very fundamental here that is not clicking for me. I feel like I'm missing something extremely obvious that I just need somebody to show me for it to click.

Please somebody help! What am I missing? What is a sample?

I quote « First, we need a wave, and second, we need enough samples of that wave to be able to deconstruct all the frequencies we care for. So, for 20kHz, we take 44,000 samples in 1 second. Right? So the minimum amount of time we need to deconstruct anything, would be 1 second. »

No, you can uses less samples and be able to compute frequency amplitudes with a Fourier transform.

The Fourier transform will get you a frequency precision which is the inverse of the size of your buffer. If you feed it with 1s of samples, you will have a 1Hz precision. If you feed it with 1/10s, you will have a 10Hz precision, and so on.

Generally, the Fourier transform is computed with 2 exponent n samples where n is an integer. This permits fast computations (see Fast Fourrier Transform).

Think of samples as frames in cine film.

No single frame shows movement. It's only when you run them as a sequence that you can see the motion.

This is kind of how samples work. At 44Khz, the computer/processor is sending out a measurement of how far to push a speaker cone 44,000 times per second to create the overall sound. There are no overlaps.

There's a good explanation of sample rates here: https://www.bbc.co.uk/bitesize/guides/z7vc7ty/revision/2

To measure audible frequencies on a digital system, you will need to analyse the differences between groups of samples. At 44Khz, to accurately find a 20Hz wave would take 2,200 samples or 0.05 seconds because that's how long the wave takes to complete a full cycle.

I can only really explain the relationship between samples and waves here but I believe, with the right processing, it should be possible to detect a given frequency using a fraction of its wavelength. For example, it may be possible to detect a 20Hz signal by only finding half a cycle (1,100 or 0.025s)

• I think you should be able to get it from a quarter wave [zero to the first peak], as in effect it's mirrored twice, once horizontally, then again vertically. [idk what practical use this is, but in theory it can be done.] Sep 19 '21 at 9:04
• I only mentioned the possibility because OP seems to be trying to create visualisations. The quicker they can pick out a given frequency, the quicker they can present it and move on to the next frame. If they can get it from a quarter of the wave then that's twice as fast as I thought might be possible. Surely there are off-the-shelf solutions for frequency analysis these days? Sep 19 '21 at 19:23