Oh, the possibilities are endless!
Depends on what type of "whoosh" you want to create:
- Long vs. short
- High freq vs. low freq (or mid)
- Sharp vs. smooth
- Peaceful vs. scary
- Simple vs. complex
- Singular vs. multiple peaks
Working with wind and air sounds will usually yield smooth, long whooshes which can add body to your sound design. Or they could be used in an emotional context, such as peaceful, happy, "airy", calm, etc.
Thunder and gunshots will go a different route, usually yielding undulating whooshes with sharp peaks (as per Ryan's example).
There's so many other ways to go, too. Try working with your own voice; record yourself singing a steady tone, whistling a note, blowing steady air, etc. Work with doppler/flanger/modulator plug-ins to add movement, then work in some reverb for added depth. Delays are helpful here, as well. And if you don't have access to fancy plugins, you can create very effective sounds with simple techniques, such as EQ sweeps and fader moves. Just like in the good 'ol days before fancy digital doodads.
Layering is a good way to build complex whooshes. Try stacking a pole swish, reversed transient attack (ie. metal hit or drum hit), and a vocal element together in your DAW (syncing up the peaks) and see what ideas you come up with.