I have a stereo mix file at about 0 dbfs. When I put a high pass filter and cut below 30Hz it boost the signal +3db up. Is this normal? I tested it with different filters (plug ins only) and I had the same result. DAW was ableton but I remember I had the same issue with Pro Tools in the past.
It is not inevitable, but nor is it surprising – yes, it would call it normal.
"Low cut" doesn't specify what the filter does exactly. Better models will either give a whole bunch of possible settings, or specify clearly the characteristic. If nothing is specified, the filter will generally be either
A first-order IIR (aka HP6). The advantage of this simple design is that it never boosts the total peak level: the filter simply follows the signal as-is, disregarding high-frequency peaks. So evidently this is not the design you're dealing with – indeed it wouldn't be much use either, since these filters have a very shallow cutoff (6 dB/8ve): set at 30 dB you'd scarcely notice any difference, and set to higher frequencies you'd affect also the desired bass range.
A resonance-free second-order IIR would be simply two of the above in series. That increases the cutoff steepness at low frequencies, but it still leaves a vast range where the signal is attenuated feebly.
A second-order Butterworth filter (aka HP12) mitigates this problem: it adds a slight resonance peak at the "knee" of the filter, not so strong that the signal is actually boosted anywhere, but a resonance it is nevertheless. This is most likely the cause of your issue.
To understand why the peak level increases even though the filter does not boost any frequencies, you need to consider the phase relations: different frequencies are shifted by different "angle". Perhaps easier to understand: like the first-order IIR (actually, that can also be called first-order Butterworth), it follows the signal shape, but to avoid the "cutoff knee", the Butterworth2 "echoes" oscillations in that range. Therefore, short bursts can come out with higher amplitude (though never higher RMS).
Butterworth filters of higher order have basically the same issue, only worse; same goes for Chebyshev-, Cauer- etc. filters. I don't think those are used in any widespread filter plugins.
Bessel filters are specifically designed to minimise overshoot/phase distortion. But even they don't completely remove it. Indeed this isn't possible with analogue filters other than LP6-cascades!
What is needed for zero-overshoot without a big filter knee: add some resonance, but with "symmetric" character. This is only possible if the filter can look "forward in time". Which isn't possible in live sound, but certainly in digital recording. Note that this will also cause "inverse reverb" artifacts, i.e. you might be able to hear sounds in the recording before they actually occur – mostly an issue for phase-linear brickwall FIRs. Those can literally cut away all frequencies below 30 Hz, but this will severely mutilate the time-domain of pulses, with tens of milliseconds of "pre-smearing". (Brickwall FIRs are an essential part of proper resampling though).
Something of a ideal compromise is offered by Gaussian filters. These are phase linear with no overshoot, have pretty steep cutoff, and yet largely preserve the shape in time-domain. This is the filter type preferred by well-designed phase-linear EQs, and would propably the best fit for your problem.
Note that all that phase buzz is only relevant for the peak level. Loudness is in fact far more accurately described by RMS, which doesn't consider phase. The only reason peak level is relevant is that today's standards (still?) alas have a cap on peak, not on RMS or actuall loudness. It is this fact that's largely responsible for the loudness war. I recommend you look more at RMS levels than peak, except in the final mastering stage.
Indeed I generally prefer IIR filters despite their phase distortion, because they tend to have better or at least more predictable transient response. A second-order Butterworth high pass is just fine, simply apply it before the final compression/limiting stages. That will probably sound better anyway, because the compressors won't need to care about subsonic artifacts (which can cause unpleasant pumping / fluttering).
(Phases are also relevant when EQing different mics that have bleed signals of the same source, but I don't think that's an issue in your application.)
I have wandered about it a while back while using a low cut eq in logic, using a Linear Phase EQ instead of a regular eq solved this mystery.
Unfortunately, I don't know the theoretical explanation behind it but I suspect that a regular eq can create unwanted phase artifacts and hence those picks in your signal.
Could it be the filter resonance?