This is because your brain needs to do more legwork to figure out the underlying shape when you look at reflective suefaces that are mostly flat and show you parts of a sky that you may or may not see at that point.
Reflectivity can also work in your favour, to highlight shapes, especially when the paint additionally has a diffuse color (think of cars) and they are in eye level (less confusing sky reflections), but highly reflective glass buildings with nothing else to gisually hold on will most of the time just reflect a random patch of the sky.
What visual readability is, is the answer to the question how fast and effortless a image can be read. A certain degree of unreadability may be interesting even, but I am not sure if architects are that aware of the reflective part of their buildings.
E.g. I happen to have the luck to have a second sun in the end of the day despite my east-facing windows, because one of the glas buildings opposite. I am sure they didn’t simulate that. I am also sure this increases the temperature in the city.
Most architects I met just think glas looks rad and all their collegues use it and so they do too.