Thanks to Tetsujin, I have the answer to my question: There is nothing wrong with my audio interface. I just had an inapporiate use of it.
The issue was caused by the audio source being stereo coming into a mono input.
For what I understood (but I sure am no expert), the input connector beeing a XLR/jack combo, is balanced. So it excpects the right signal from the cable to be the inverse of the left signal. As I enter stereo with similar L and R, the result was sort of a cancelation of the signal.
Edit: by left and right I mean "tip" and "ring" of the connector, which are used as positive and negative polarity for balanced mono signal (ref).
The next step for me is finding some kind of stereo to mono splitters in order to plug my Pocket Operators properly.