No – significant preprocessing of photoreceptor cell signals occurs in the eye, in the retina. The "raw" signals from the three cone types are processed into luminance and two color-opponent channels before being sent to the visual cortex. (While there are around 130 million retinal receptors, there are only approximately 1.2 million axons bundled in the optic nerve.)

Actually they do, and beyond that, the retina contains neurons that do the basic processing before travelling down the optic nerve to the brain. The retina is also basically brain tissue and is part of the central nervous system.

To reinforce what you're saying, I just want to point out that the retina is actually a part of the central nervous system. It grows from the same tissue that the brain does during fetal development.