Well, there are ways that a signal could be clean, i could experiment with that kind of noise removal.
Just to give an answer , there could be some ways you could remove this sound, one is to try and make it exactly the same, with some noise generators and stuff and put it right underneath and flip the phase.
i believe that the de noiser has really killed it...
If you have a boom mic, you could for an instance change it by editing/cutting a small chunk and after doing some processing substitute it with the chunk of time that contains the cloth noise.
I think that the best denoiser you can make can be done with a way of gating called subtractive gating.
In essence this are 2 channels and one has a limiter and the other is clean and phase flipped.
Try to find the sweetspot and you can have some good results.
When you end up having a problem like yours there's only magic that can help.
The last way you could clean a signal is Crossover Distorion.
Crossover distortion is a channel with a clipper VS the clean channel with the phase flipped, again trying to find the sweet spot.
This technique actually happens in tape machines.
This is not a linear process! So careful about your levels.