That's really fascinating. Reading your account reminded me of the book Musicophilia by Oliver Sachs. He's a neurologist who studied the connection between sound and the brain. There are several stories in a sense similar to yours, of people having powerful experiences in their life where sound plays a part. As an example some of the book is about a 42-year-old man struck by lightning who suddenly experiences an unquenchable thirst for music, learns to play the piano, and starts to compose. I recommend it to anyone here actually, it's also quite an easy read - not full of jargon. So I'm not suggesting that you will find some all encompassing answer in the book, reading it just highlights how complicated the connection is between what we experience, perceive and how we try to interpret meaning from that.
Back to your question though. As was suggested you should definitely look up Synesthesia, but be aware that it is a neurological condition - the connections have been made by the brain of the person who experiences it. Why they have this experience and whether it relates to anything measurable and coherent is the interesting thing about it. From a scientific standpoint colours don't literally have equivalent sounds but they do have a specific wavelength in the same way individual frequencies of sound have a wavelength:
The connection has been exploited by VJ's and programmers writing sound visualisation software extensively over the years. Essentially if you analyse and map from one to the other, in a sense you can create images from sound or convert image data into sound.
So exploring this sort of thing is a route you could take, and if you find it interesting then why not! Everyone on these pages has some kind of (perhaps unexplainable) fascination with sound so you'd be in good company. I'd say though, that working with sound and image at this sort of level is fairly advanced so don't expect to get results right away. Another route you could take is to hook up with a sound designer/artist who can work with you on the project, you might be able to find someone working in those areas already who has an interest in your story.
I'd also suggest that if you can, perhaps you should speak to a neurologist or other professional about your experiences - they might be just as interested as you are, and you may find as many answers there as you will pursuing your ideas here.
All the best,