It was a single spatial point of ray intersection with your sensor or eye. You’d need color filters/retinal cells to pick apart the frequencies in the complex waveform.
After rainbow separation, the components are spread across multiple sensors giving you a frequency domain view.
Goodman is great, Hecht and Zajac covers more fun with optics at an intro level.
Some more info here
Miles V. Klein, Thomas E. Furtak - Optics 2nd ed, Wiley
Joseph W. Goodman - Introduction to Fourier optics, W.H. Freeman
https://www.youtube.com/watch?v=Y9FZ4igNxNA
Adding that, again, optics is a difficult and frustrating field. Don't worry that you're struggling to connect it. It took me a few years in an optics lab working hands on with light every day to really come around.