0

Suppose that we have a microphone-recorded note, say played on a guitar, and we want to write a computer program which will detect the note frequency. There are already plenty of such programs. How to write a new one from scratch? And, as a sidenote, how does a human ear (maybe for those with absolute pitch) detect the note frequency?

It may seem that the question already has an answer "read on Wikipedia about Fourier transform / FFT", but what is missing here is the window width, this is what the question is about. This question may seem a bit technical, so let me give some background first.

The sound is transmitted through the air with variable pressure. The microphone identifies air pressure at given time for several consecutive time points. Our ear, I suppose, works in a slightly different way, but I am going to return to this later. Everyone interested in acoustics knows that the pure sound is formed by a sine function sin(f*t) where f is frequency, t is time; the real sound with given base frequency is formed by the sum of sine functions. There is a lot of questions and answers here on music.stackexchange about what are overtones and possible inharmonic frequencies, this is fairly well understood.

In order to decompose the sound "back" into the sum of sines, there is a so-called Fourier transform. Mathematicians know that if your function f(x) is periodic with period T, i.e. f(x+T) = f(x), then you can decompose the function into the sum of sines with coefficients, which are all computable.

From a musical point of view this means that if you have some low-frequency note, say C1 played in a low octave, and also multiple frequencies of it, say C2, G2, C3, E3, G3 and so on (harmonics), with different amplitudes, then

  • from the shape of the waveform you can recover the amplitudes of the harmonics
  • from the amplitudes you can recover the musical instrument that played this note, by comparing to already pre-recorded profiles of amplitudes.

But here we had an assumption that the function is periodic. In real life that doesn't happen because many instruments play many different notes and the notes change, so the period also changes. Fourier transform is of little help here, unless we know the true base frequency of the note, and we can throw out other notes as "noise" because they don't fit into the periodic picture. Note that a human ear of a trained musician solves this problem perfectly, detecting the base pitches (or at least, relative pitches), so all the information contained in the sound package is enough to infer the answer. I know that the ear contains some "hair" that vibrate in response to particular frequencies. Maybe this approach is developed to some extent?

Question. How to detect the note or a set of notes which sound at given point/period of time on a sound recording using computer only?

Also, "transcribe!" is a sort of computer program that can do that (I only heard but never launched), so I suppose that this is not so hard to understand on idea level.

Some related links:

https://music.stackexchange.com/a/48255/40204

https://stackoverflow.com/questions/8928695/sound-convertion-to-frequency-in-android

https://stackoverflow.com/questions/3904488/determine-the-audio-frequency-of-sound-received-via-the-microphone

https://stackoverflow.com/questions/1354084/real-time-pitch-detection

  • This seems off-topic here, as it is not about music practice, performance, technique, theory, etc. Also probably off-topic for Stack Overflow for being too broad. – David Bowling Nov 17 '17 at 12:00
  • Regarding the window width, I think it has to be dynamic, using a trial and error method until the FFT yields useful results. In the old days, tuners would have range settings which would help narrow the guessing space and they would respond more slowly. Now that there’s so much more compute power available so cheaply, dynamic windowing works better and faster. The way ears work is an entirely separate and large topic. Both topics really require a lot of research and don’t really work as focused questions with definite answers. – Todd Wilcox Nov 17 '17 at 13:16
  • @ToddWilcox Thank you for migration. For me, music theory is closely related to acoustics and mathematics but maybe David is right and we should separate solfeggio from physics. I mentioned the human ear because I think that the physical intuition behind the vibration in human ear can be applied to note recognition. I think what you suggest is a good approach to imitate how ear works: by selecting a range of different widths (e.g. 35000 of them) we put each width into correspondence with a single hair (or with a subset of hairs). – Sergey Dovgal Nov 18 '17 at 5:26
  • I can’t migrate, and I didn’t, but I’m glad your question got migrated. – Todd Wilcox Nov 18 '17 at 5:35
1

Windowed FFT is indeed one of the usual approaches to find an approximation of a spectrum within every window - and there is a trade-off here, as wider windows mean worse time resolution, while more narrow windows mean worse frequency resolution. Wavelets try to sidestep this trade-off to an extent, as does constant-Q transform.

But even after you find a good approximation of a spectrum, it does not tell you which notes are there, as spectrum is about frequencies.

Let's say, your first window has 30% (by power) of frequency 440 Hz, 30% of frequency 880 Hz, and 40% of 1320 Hz. Does it mean someone played a chord of A4/A5/E6? Or maybe just a single A4 with harmonics? You really need to know the timbre of the instrument used - which is complicated by the fact that there may be multiple instruments playing, and even for one instrument the timbre can change based on how exactly it is played.

So whatever method you use for spectrum extraction, you probably need some machine learning model to recognize chords from the spectrogram.

One obvious simplification (which your question seems to imply) is assuming only one note sounding at a time. In this case, it is possible to work with harmonics directly. Note that it is not enough to just take the lowest pronounced peak in the spectrogram as the fundamental frequency - first, you will need to define what a "pronounced" means, and second, some timbres have the fundamental frequency absent, but the ear still detects it as if it were there! E.g., if there are peaks at 800, 1200, 1600, and 2000 Hz, the ear would recognize a fundamental at 400 Hz. An easy way to work around this is to autocorrelate spectrogram, and to take the highest peak as the fundamental.

A more advanced method (still starting with autocorrelation) is described in YIN, a fundamental frequency estimator for speech and music.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.