That said, patio11 is talking about writing books in highly specialised academic areas. Linear algebra is standard fare for an undergraduate curriculum - taught at thousands of universities - using innumerable textbooks. So it is not something highly specialised. It is conceivable that your book was largely inspired by other linear algebra books, if indirectly. For instance, you might have had strong opinions about why a particular approach does not work [1], and chose not to follow it. The search space has been reduced by authors who have come before. An academic writing a book in a more specialised area would probably not have the luxury of a writing precedent and would have to figure out a lot themselves. This translates to more time spent on a book for which they would otherwise receive no reward.
Of course, it is possible that this premise is false and that it is easier for a researcher to write a book on their own research specialty.
I would imagine that your research appeals to 'a very, very niche audience'. Would you consider spending a lot of time and effort on a book (.pdf) in your research area - for no material reward? To be read by a very small audience? Orders of magnitude smaller than undergraduates looking for a good linear algebra text?
[0] http://joshua.smcvt.edu/linearalgebra/ [1] I certainly do - I dislike Anton and Rorres