when you look outside the window, any sunlight that enters your eyes from the angle of the sun is the sun because the sun is so far away. this is why we have well defined shadows.
any light that comes at an angle is all the light that did not come directly from the sun. light coming at an angle is directly aimed at your pupils if you're looking at the source.
what you said, the only way it's true, is if you're always looking directly at the sun.