Follow the signal flow from input(whatever is amped) to output(mixer/pa) and also tell us explicitly what the output is on the amp. It isn't totally clear. This is assuming Eugene is right about the cable you used, since you didn't specify male or female for RCA to 1/8.
What kind of amp, and is the output:
2 RCA outputs? (Scenario 1) unbalanced signal, stereo
is it balanced mono(trs) 1/8 output? (scenario 2)
Or 1/8 Headphone out which is stereo unbalanced? (#3)
What level is the output? Headphone or line?
Scenario 2 is rare from any amp I can think of. If its scenario 2 try an 1/8 trs male adaptor to 1/4 inch trs female t(plug in amp output), and use a trs 1/4 inch, not instrument (ts) cable to connect to the mixer. If you don't you'll barely get signal from instrument cable
For 1+3
if your mixer takes unbalanced mono inputs and has the gain to make up the level difference from headphone to line level, no problem but noisy.
If it has a stereo 1/4 input, you could use one input instead of splitting to two mono.
Likely your mixer wants to see a balanced mono input from the 1/4 input. This would be a trs cable. If your inputs aren't wired to expect a ts cable you may get nothing in either scenario regardless of how you adapt it. So try these options:
Scenario 1
Take the male trs 1/8 inch end and use a f 1/8 trs (stereo) to 1/4 trs male adaptor. This also requires a stereo 1/4 input on the mixer. If you don't have one, you should use a f 1/8 trs to 2 female ts adaptors and use 2 instrument cables.
For sc 3 you could try to use 2 RCA f (stereo) to ts f adaptors, and plug 2 inst cables into the inputs on the mixer.
If you have a stereo 1/4 input you could use a 2 female RCA to 1/4 inch female adaptor, but you would have to use 1/4 trs cable, not instrument.
The issue is that a balanced input will phase out your unbalanced signal.
If this happens, Mic your amp, buy a headphone amp, or buy a box to convert your unbalanced signal to balanced.
Most importantly be more specific or we have to guess what you mean and spend forever answering your ambiguous question's various possibilities.
Don't say mono or stereo say trs or ts, because a trs cable can be unbalanced stereo or balanced mono. I hope this sheds some light on the differences for others seeking help.
