To be more specific, it’s not based on knowledge of art but knowledge of community. The art market is a subculture in every sense. It has only a superficial relationship with objective human creativity or even art history.
“You have to know what you’re looking for” is both correct and misleading. The thing you’re looking is predominantly a record of aristocratic relations. This was not always the case, but it is very much is now.
The way to view it is as a bubble. It’s a bubble based on the notion that common people cannot be trusted to evaluate art. This claim does hold water, but it was historically based on cultural intelligence which is no longer synonymous with capital wealth.
The average art history knowledge an art investor has is fairly superficial, which is probably best, because the market could care less about theory and critique.