I think the mini failed because most of those customers bought the second-generation SE.
The only people I know who pick SE are those who either still want the physical home button or just want the cheapest new iPhone. In all other metrics, including smaller physical size, Mini wins hands down.
So I find your claim about people who loudly demanded a smaller iPhone cannibalizing Mini sales by picking a larger SE instead a bit difficult to believe.
Also, it is worth noting the desire for small phones is mostly due to a desire for easier one-handed use, and the SE is as easy to use as the Mini in that regard, since the user does not need to reach the top of the device and the home button is centrally located, and thus easier to reach.