No it does not matter at all. The optics focus it much further away. As far as your eyes are concerned the display isn't right in front of them.
In fact if those optics weren't there you wouldn't see anything except a few coloured blobs.
The amount of brightness your eyes are ingesting is also much lower than walking around on a sunny day without sunglasses. Remember our eyes work logarithmically so the outside could be tens, hundreds of times as bright.