Do we? That's only one possible implementation, and not a very efficient one for most uses.
and in the special case of extremely sparse graphs like trees, the adjacency matrix isn't going to win very often.
your copy of CLRS should discuss all of this. and for practical uses of non-adjacency matrix representations of trees, you can take a look at every functional programming language, ever.