I'm remixing two songs, and their drum patterns are somewhat incompatible so I'm try to cut out the drums in one of them. The problem is that the non-bass drums are at the same pitch as some of the vocals and piano notes, so I can't just filter out that pitch. (Obviously I am working with the final mix and not individual instrument tracks).
Is there a way to do this in Adobe Audition or Soundbooth, or in any free tools? Or a good algorithm that I could code as a plugin?
In general, the short answer is simply 'no'. Cutting out an instrument entirely in the mastering stage of a recording, is completely impossible. Even if you are having balancing problems that are not due to frequency ranges in your listening equipment, it is hard to believe that the balance was good when you were actually mixing it down.
This is a problem that should always be fixed while mixing the separate tracks together, you really can't properly get it done while mastering. If you can't get it done during the mixing process either, there were crosstalk problems during the recording session, that had to be fixed back then.
My advice: go back to the mixing stage and make a new 2-track mixdown without the drum tracks. That should save you a lot of trouble.
If it happens to be the case that you trashed all the instrument tracks, well, there is not much left to say but "make sure that will never happen again!"
You cannot solve for x and y in the equation:
x + y = 5
because you do not have enough information. If you knew what x was, you could find y, and you can find what x and y are in relation to each other (e.g. x=5-y), but you cannot find both values without more info.
Likewise, once two signals are mixed, you cannot find what either sounds like by itself from the mix alone because you have the sum of those signals.
Now... If you have the original drum signal you want to remove from the mix, you could add an inverted version of it--i.e., subtract the signal from the mix. You will want to tweak the volume and tone to be identical to what is mixed prior to inverting it. This is a basic noise reduction technique. I understand noise reduction works this way. If you do not get the volume (and any other effects) pretty close and the timing perfect, it will not work. Besides, if you have access to the original sounds, you can still edit the mix, right?
Mastering engineers would apply multiband compression to this kind of problem to try and 'minimise' the excessive frequencies in a certain range. You can boost then selectively control if you exceed a threshold, but only for a particular frequency range. One of the great tools in my mastering toolkit.
Combined with a wide-Q EQ ("broad smile") to curve across the frequency range and lessen the impact of certain ranges, this often has a very positive effect upon the resulting sound - when done judiciously! Unfortunately without high quality plugins if you try to do this you'll often get sub-optimal results (although there are some great high quality and free mastering plugins available).
