How does "transcribe!" or guitar tuner work?

Question

Suppose that we have a microphone-recorded note, say played on a guitar, and we want to write a computer program which will detect the note frequency. There are already plenty of such programs. How to write a new one from scratch? And, as a sidenote, how does a human ear (maybe for those with absolute pitch) detect the note frequency?

It may seem that the question already has an answer "read on Wikipedia about Fourier transform / FFT", but what is missing here is the window width, this is what the question is about. This question may seem a bit technical, so let me give some background first.

The sound is transmitted through the air with variable pressure. The microphone identifies air pressure at given time for several consecutive time points. Our ear, I suppose, works in a slightly different way, but I am going to return to this later. Everyone interested in acoustics knows that the pure sound is formed by a sine function sin(f*t) where f is frequency, t is time; the real sound with given base frequency is formed by the sum of sine functions. There is a lot of questions and answers here on music.stackexchange about what are overtones and possible inharmonic frequencies, this is fairly well understood.

In order to decompose the sound "back" into the sum of sines, there is a so-called Fourier transform. Mathematicians know that if your function f(x) is periodic with period T, i.e. f(x+T) = f(x), then you can decompose the function into the sum of sines with coefficients, which are all computable.

From a musical point of view this means that if you have some low-frequency note, say C1 played in a low octave, and also multiple frequencies of it, say C2, G2, C3, E3, G3 and so on (harmonics), with different amplitudes, then

• from the shape of the waveform you can recover the amplitudes of the harmonics
• from the amplitudes you can recover the musical instrument that played this note, by comparing to already pre-recorded profiles of amplitudes.

But here we had an assumption that the function is periodic. In real life that doesn't happen because many instruments play many different notes and the notes change, so the period also changes. Fourier transform is of little help here, unless we know the true base frequency of the note, and we can throw out other notes as "noise" because they don't fit into the periodic picture. Note that a human ear of a trained musician solves this problem perfectly, detecting the base pitches (or at least, relative pitches), so all the information contained in the sound package is enough to infer the answer. I know that the ear contains some "hair" that vibrate in response to particular frequencies. Maybe this approach is developed to some extent?

Question. How to detect the note or a set of notes which sound at given point/period of time on a sound recording using computer only?

Also, "transcribe!" is a sort of computer program that can do that (I only heard but never launched), so I suppose that this is not so hard to understand on idea level.

Some related links:

https://music.stackexchange.com/a/48255/40204

https://stackoverflow.com/questions/8928695/sound-convertion-to-frequency-in-android

https://stackoverflow.com/questions/3904488/determine-the-audio-frequency-of-sound-received-via-the-microphone

https://stackoverflow.com/questions/1354084/real-time-pitch-detection

Answer 1

Windowed FFT is indeed one of the usual approaches to find an approximation of a spectrum within every window - and there is a trade-off here, as wider windows mean worse time resolution, while more narrow windows mean worse frequency resolution. Wavelets try to sidestep this trade-off to an extent, as does constant-Q transform.

But even after you find a good approximation of a spectrum, it does not tell you which notes are there, as spectrum is about frequencies.

Let's say, your first window has 30% (by power) of frequency 440 Hz, 30% of frequency 880 Hz, and 40% of 1320 Hz. Does it mean someone played a chord of A4/A5/E6? Or maybe just a single A4 with harmonics? You really need to know the timbre of the instrument used - which is complicated by the fact that there may be multiple instruments playing, and even for one instrument the timbre can change based on how exactly it is played.

So whatever method you use for spectrum extraction, you probably need some machine learning model to recognize chords from the spectrogram.

One obvious simplification (which your question seems to imply) is assuming only one note sounding at a time. In this case, it is possible to work with harmonics directly. Note that it is not enough to just take the lowest pronounced peak in the spectrogram as the fundamental frequency - first, you will need to define what a "pronounced" means, and second, some timbres have the fundamental frequency absent, but the ear still detects it as if it were there! E.g., if there are peaks at 800, 1200, 1600, and 2000 Hz, the ear would recognize a fundamental at 400 Hz. An easy way to work around this is to autocorrelate spectrogram, and to take the highest peak as the fundamental.

A more advanced method (still starting with autocorrelation) is described in YIN, a fundamental frequency estimator for speech and music.