Thanks to Tetsujin, I have the answer to my question: There is nothing wrong with my audio interface. I just had an inapporiate use of it.

The issue was caused by the audio source being stereo coming into a mono input.

For what I understood (but I sure am no expert), the input connector beeing a XLR/jack combo, is balanced. So it excpects the right signal from the cable to be the inverse of the left signal. As I enter stereo with similar L and R, the result was sort of a cancelation of the signal.

The next step for me is finding some kind of stereo to mono splitters in order to plug my Pocket Operators properly.