Everything you say is true in that old world.
In today's world, Google is optimizing for ad revenue.
I could see how optimizing for PageRank could go out the window in such a world.
Even if PageRank still has influence, it's not clear to me the site doesn't have a large web community with a lot of links. Maybe it does, and the ad revenue considerations are overriding that.
But for whatever reason, the reality seems to be a relative few websites are selected to provide "good enough" answers on shallow queries for a huge number of topics or objects, Wolfram Alpha style.
Maybe this is due to developer laziness, gradual dilution of cutting edge engineering, explicit desire for simpler solutions or real practical limitations, even at Google's scale. In any case, the result appears to be that sites like yours are poorly surfaced because they're not very generalizable. Whereas CB Insights claims it knows about your query already, there's an internal API setup for exposing CB Insights already, and CB Insights passes a basic sanity check on the answers.
This is really unforunate and I sympathize with you. I think there is a lot of opportunity for a search engine that can optimize for very specialized queries across many domains, including arbitrarily historical resources. But I think such a search engine would have very different goals and incentives than Google from the outset.
Actually, that’s almost exactly how Google described the algorithm update...