I'm not sure I follow your argument here... it seems to me that you would be getting three independent signals on each eye, thus the perception would be effectively a 6-dimensional color space.
That said, I think I've come around to the idea that we could see multiple signals. I could probably identify "right-brightness" & "left-brightness" as separate from "right-desaturation" & "left-desaturation". But can I also separate those from "right-CCW-hue-rotation"? Can I pair it with "right-greenness/blueness/redness" in a way that I am gaining a full independent channel of information? Maybe it's like 2.2 channels worth. :P
I guess I shouldn't speculate at all; it's easy to be misled in hypotheses about perception.