What is the current state of the art in source separation? What tools are available?

Question

I've seen a few questions around there at ultimately seem to be asking for software that does source separation, which refers to techniques for separating mixed-down audio into some approximation of its original sources. The human brain is pretty good at separating the components of clean signals -- in real time, even -- so it seems reasonable to expect that it's easy to do in software.

Unfortunately, it's not -- it's definitely one of the harder problems in DSP.

The easiest kind of source separation problems are ones where the two desired sources are mixed in stereo with different panning, or more generally, you have at least as many audio outputs as inputs. Situations such as two or more sources mixed down into a mono track are extremely difficult, often referred to as blind source separation (BSS) since there are no such spatial cues.

I'm not asking here how to unmix a specific track. Rather, I would like to know:

• What is the current state of the art in source separation for audio?
• What tools are out there for doing source separation -- either for programmers or for casual end users?

Answer 1

The separation of instruments from a mix is something that seems easy to us. Humans can concentrate, for example, on a specific instrument being played in an orchestra. We can do this because we are very good at grouping harmonics into a 'tone', and associating characteristic timbre with an instrument. We are also good at filling in what isn't there with what should, probably be there given what we know and what we expect.

The last time I looked for a tool that could separate instruments was a good few years back, but I don't think things have progressed much since then. The software I was advised to try back then was Celemony Melodyne (v3), which I was told could edit individual instruments separately in a mixed audio stream. I remember trying it for a day and being quite disappointed with the results. Looking back with a bit more experience and knowledge, I think the problem was that I wanted a tool that could split an audio mix to its source components, but what I got was a pitch editing tool for correcting the pitch of individual instruments and vocals.

The software did this quite well for say a piano and vocal mix, adjusting a bad singer's voice or correcting a wrong note struck on a piano. This is very different from a source separation tool because only the harmonic content needs to be adjusted with pitch correction, the rest of the instrument's signal, which is important to the instrument's character, is not adjusted. This means that the instrument's pitch can be changed without the instrument having to first be separated from the rest of the mix. Note that a more complicated mix would confuse the software, so individual changes were not possible.

I don't think source separation will be at a usable level for a long time, if ever, because it is impossible to distinguish inharmonics and characteristic sound of different instruments without an exact spectral print of the source instruments. Think of a flute for example, a very clear tone with little harmonic content above the fundamental. But it wouldn't sound like a flute without the characteristic wind sound, which would be pretty impossible to remove from a mix of instruments because there is nothing to tie it to the harmonic content, aside from our perception that it is the wind from the flute.

I think the only way that would get close would be a self-learning, intelligent system which analyses and categorises thousands of instruments and makes a guess based on that information, then analyses the result as a whole and estimates the chances that it is correct based on past results being correct or not.

I'm sure that when source separation is at a usable level, we will all hear about it. I know that a lot of questions on this site would have new answers.

Answer 2

I think the leaders in this field are without a doubt iZotope and Zynaptiq.

iZotope RX is just the most incredible tool for separating sounds using FFT and spectrographic analysis: https://www.izotope.com/en/products/repair-and-edit/rx.html

To see what it's really capable of you should check out the work that Michael Wabro did separating the soundtrack and dialogue audio from the classic film Brief Encounter: http://www.pro-tools-expert.com/home-page/2014/7/22/podcast-extra-interview-on-using-izotope-rx3-to-remove-the-m.html

You also need to take a good look at Zynaptiq's plugins like Unmix and Unfilter. Some of their software is pure sorcery: http://www.zynaptiq.com/products/

Answer 3

My website, monotoSTEREO.info, and its companion Facebook page may be of interest. It is dedicated to providing a collection of resources on audio upmixing using processes such as source separation and spectral editing, with links to over 1,000 research papers, presentations, etc. in the field as well as media samples.