All new artforms are derided as not being "true" art by the establishment, who are invested in the status quo. Even contemporary art, intended to expand our ideas about art, is often only accepted under certain existing categories; performance art, mixed media, others.
Once could hypothesise that when non-gamers think of games, they think of the most popular games, which means the most commercially viable. I would equate it to thinking of advertising illustration or graphic design when talking about "visual art".
Of course, people often think games should be fun in some way, so the kookier games that are artistic expressions might not become popular, because people don't want to experience those works.
Some examples of games I consider arty:
- Moirai [0]
- Sisters of the Amniotic Lens [1]
- Gone home [2]
- The Stanley Parable [3]
[0] http://store.steampowered.com/app/496920/
[1] http://store.steampowered.com/app/287140/
[2] http://store.steampowered.com/app/232430/
[3] http://store.steampowered.com/app/221910/