I understand that setting your audio interface to 96Khz or 192Khz can change the sound of a software synthesizer. So would it be best to record the synthesizer at 96Khz or 192Khz?
What you are dealing with here more likely than not is the software synthesizer not distinguishing between sampled and continuous signals. An analog synthesizer applies functions to a continuous signal. This is not the same as applying functions to a strictly lowpass filtered continuous signal (because many functions will cause higher frequency contents to "fold down" into the audible domain), nor is it the same as applying the function to discrete-time samples (applying functions to samples cannot increase the bandwidth of a sample beyond the Nyquist frequency while applying them to a low-pass signal can).
Now even with an analog synthesizer, accuracy and reproduction will taper off comparatively quickly as you venture into ultrasound, not least of all because in the interest of noise and circuit design and oscillation tendencies there is little incentive to extend the operating domain significantly into ultrasound. As opposed to minimally sampled materials (48kHz, say), accuracy and reproduction will not rapidly cut off but rather taper off.
It would be my guess that with 96kHz you'd usually be at a good compromise where the digital/analog border is far enough from the audible domain that conflating sampled and continuous signals is not going to cause a lot of trouble. You also avoid most of "frequency warping" artifacts when translating IIR filters into their (comparatively straightforward) bilinear transform equivalent. So for practical purposes and emulating analog synthesizers (rather than experimenting with something purely digital), I'd consider 96kHz sufficient. Yes, there will be artifacts, but they will probably be dwarfed by the artifacts an analog synthesizer produces due to not being designed for ultrasound veracity.
There may be a point in asking on the signal processing stackexchange: the respective mathematics may be better known there.
Neither is best. You do what creates the best sound according to the ideals you want to achieve. 44.1 kHz or 48kHz are perfectly usable as recording sample rates, going higher will only make minor changes, not necessarily better, but slightly different. To some ears these changes are important, to other ears not so. So the recommendation is to try it and use your own ears and listen. My bet is that you will end up at 44.1kHz unless you are into video where 48kHz is used.