Have you considered that useful predictive models for reality are simply irreducible to being comprehensible by swollen savannah monkey brains? It's not magic. It simply cannot both be explained to a human and be worth anything.
From Wikipedia: It is stated that "every good regulator of a system must be a model of that system".
How is that related to comprehensibility? Or going extinct?
A lot of smart people seem to be surprised that anything can be comprehended at all:
https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
The lack of mathematics effectiveness in description of intelligence seems unsurprising to me.
Supposedly the development versions we the general public can’t get yet are getting pretty good and would meet your “most of the time” bar on public roads.
But we don’t really know how good AI is at any given moment, because those that are leading in any AI area sometimes have reasons for not telling yet. Or because we don’t believe them when they do tell us.