5

I've seen a few questions around there at ultimately seem to be asking for software that does source separation, which refers to techniques for separating mixed-down audio into some approximation of its original sources. The human brain is pretty good at separating the components of clean signals -- in real time, even -- so it seems reasonable to expect that it's easy to do in software.

Unfortunately, it's not -- it's definitely one of the harder problems in DSP.

The easiest kind of source separation problems are ones where the two desired sources are mixed in stereo with different panning, or more generally, you have at least as many audio outputs as inputs. Situations such as two or more sources mixed down into a mono track are extremely difficult, often referred to as blind source separation (BSS) since there are no such spatial cues.

I'm not asking here how to unmix a specific track. Rather, I would like to know:

  • What is the current state of the art in source separation for audio?
  • What tools are out there for doing source separation -- either for programmers or for casual end users?
  • Could you be more specific what kind of sources or sound elements? I think the question is quite broad and could be answered by mentioning any number of audio tools. Given the well established process of recording-mixing-mastering, I fail to see where the value of source separation comes in. If you're looking for analysis, I'd suggest iZotope, Flux or Avisoft. – Joel Pinteric Oct 6 '16 at 13:46
  • 2
    The question is actually quite clear but it relates to a very specific research topic with almost no mature enough musical application today. For the state of the art, maybe try to find a paper published at the last edition of the DAFx Conference. Here are audio examples from LABRI in 2006, and here from INRIA in 2014. – maxime.bochon Oct 9 '16 at 16:15
  • The aim here is to create a linkable resource that we can direct people to when they ask about source separation. It's a pretty common question here. – wwww Oct 15 '16 at 16:46
5

The separation of instruments from a mix is something that seems easy to us. Humans can concentrate, for example, on a specific instrument being played in an orchestra. We can do this because we are very good at grouping harmonics into a 'tone', and associating characteristic timbre with an instrument. We are also good at filling in what isn't there with what should, probably be there given what we know and what we expect.

The last time I looked for a tool that could separate instruments was a good few years back, but I don't think things have progressed much since then. The software I was advised to try back then was Celemony Melodyne (v3), which I was told could edit individual instruments separately in a mixed audio stream. I remember trying it for a day and being quite disappointed with the results. Looking back with a bit more experience and knowledge, I think the problem was that I wanted a tool that could split an audio mix to its source components, but what I got was a pitch editing tool for correcting the pitch of individual instruments and vocals.

The software did this quite well for say a piano and vocal mix, adjusting a bad singer's voice or correcting a wrong note struck on a piano. This is very different from a source separation tool because only the harmonic content needs to be adjusted with pitch correction, the rest of the instrument's signal, which is important to the instrument's character, is not adjusted. This means that the instrument's pitch can be changed without the instrument having to first be separated from the rest of the mix. Note that a more complicated mix would confuse the software, so individual changes were not possible.

I don't think source separation will be at a usable level for a long time, if ever, because it is impossible to distinguish inharmonics and characteristic sound of different instruments without an exact spectral print of the source instruments. Think of a flute for example, a very clear tone with little harmonic content above the fundamental. But it wouldn't sound like a flute without the characteristic wind sound, which would be pretty impossible to remove from a mix of instruments because there is nothing to tie it to the harmonic content, aside from our perception that it is the wind from the flute.

I think the only way that would get close would be a self-learning, intelligent system which analyses and categorises thousands of instruments and makes a guess based on that information, then analyses the result as a whole and estimates the chances that it is correct based on past results being correct or not.

I'm sure that when source separation is at a usable level, we will all hear about it. I know that a lot of questions on this site would have new answers.

  • TL;DR "I don't think source separation will be at a usable level for a long time." I must agree. – user9881 Oct 10 '16 at 19:30
  • I appreciate this excellent and thorough explanation of why separation is so hard, but I'm seeing a lot of presumptions extrapolating from experiences you had with just one software program years ago. – wwww Oct 15 '16 at 17:09
  • Yes, but this experience of mine is common even today. Melodyne is still one of the best pitch-correction programs, used in studios around the world. Not much has changed since then, a little refinement maybe. Separating two signals from a multi-channel signal i.e. dialogue and soundtrack is easy as long as certain criteria are met. But to people who don't know, this looks like a breakthrough. Truly separating two unique and spectrally similar summed signals would be revolutionary. – Marc W Oct 27 '16 at 19:28
  • Tools like iZotope RX are inching us closer, but this is just old tools used in a new, clever way. You can infer the state by reading questions and answers on this site and all over the net on this subject. Good answers will contain the fundamental building blocks used in tools like RX. – Marc W Oct 27 '16 at 19:28
4

I think the leaders in this field are without a doubt iZotope and Zynaptiq.

iZotope RX is just the most incredible tool for separating sounds using FFT and spectrographic analysis: https://www.izotope.com/en/products/repair-and-edit/rx.html

To see what it's really capable of you should check out the work that Michael Wabro did separating the soundtrack and dialogue audio from the classic film Brief Encounter: http://www.pro-tools-expert.com/home-page/2014/7/22/podcast-extra-interview-on-using-izotope-rx3-to-remove-the-m.html

You also need to take a good look at Zynaptiq's plugins like Unmix and Unfilter. Some of their software is pure sorcery: http://www.zynaptiq.com/products/

2

My website, monotoSTEREO.info, and its companion Facebook page may be of interest. It is dedicated to providing a collection of resources on audio upmixing using processes such as source separation and spectral editing, with links to over 1,000 research papers, presentations, etc. in the field as well as media samples.

  • Sorry, but I had to tone down some of the wording in your answer to make it more appropriate for this site. Very cool resource, however! Thanks for sharing. – wwww Dec 2 '16 at 21:02
  • 1
    Even some small exerts of relevant data would improve this answer. – user9881 Dec 12 '16 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.