In the case laid out in the OP, it is a little subtle, but it's really about status games, not the knowledge itself. Otherwise he wouldn't need to make someone feel like shit for not knowing (or remembering) the term.
For example, I will readily admit I had to look it up because I don't use the term often -- I remembered learning it, and vaguely that it was related to the diagonal/determinant, but that was about it. That doesn't make it trivia, but it doesn't make me an idiot either.
So, it's not trivia if one needs to use it, but that's not how the protagonist in the essay used it. You can imagine how differently things would have gone if he kept his judgement to himself, had sharing information as the real goal, and simply explained what the trace was in the context of the conversation without the putdown. It wouldn't be trivia in that context, because it would be relevant to a useful conversation -- not a status cudgel.