We don't even know whether we see the world the same to begin with. We just use the same labels.
Maybe my red is your green but we both call it blue.
We actually do know quite a bit beyond the labels. We measure excitation in neurons. We can tell from DNA sequence what kind of color blindness (if any) one has.