Calling music by this term communicates a powerful judgment about its place in musical history. And it is simply, unavoidably incorrect. It has nothing to do with the music itself - it has to do with his musical process and how it jives with the basic musical environment of his time.
If the whole musical world were like John Cage, Philip Glass's works would definitely be avant-garde. But because the musical world is a lot more like Philip Glass, using the word "avant-garde" to describe his contributions is simply untrue.