As Mark said, there is no way of magically removing your voice without affecting the recording's quality. However, if you are willing to try, then here are some options:
Reverse the polarity of your recording and play it in sync with the recording containing both, your and the other person's voice. This will help you cancel your voice from the second recording. However, you will have to tweak the volume so that your voice gets completely cancelled. And also, if there's soom room ambience (obviously) then you might not get desired results. Also you might face some phasing issues due to this.
Use an equalizer to remove your voice. This will only work if most of your voice is in a different frequency range than the other person's voice.
If you both were rarely speaking at the same time, then you can just trim out the parts when you were speaking and insert silence there. To deal with the residual noise, you can de-noise the audio. De-noising will not affect the audio as much as the above methods do.
The success of using these advices to get what you want, largely depends on the recording quality and the amount of voice that you need to remove. However there's no harm in trying these out. Good luck!