People say they 'just' want a 4" modern phone, but they define it as a phone with multiple cameras, high storage capacity, and fast processors, etc. etc.. It's a nontrivial engineering challenge to get everything into a smaller package, and this at a time where the market at large has picked bigger phones as the winner.
Also, don't forget that people generally expect something that's smaller to be cheaper. Reality is, it might actually cost more money to deliver a smaller product, and at that point you're swimming against people's assumptions, not a great place to be.
So, you sit down to design a small modern phone and BAM it's just nothing but obstacles in every direction. It doesn't surprise me that the conclusion manufacturers are making is that the only winning move is not to play.
I'm not saying it's the _right_ decision necessarily, maybe someone will make an excellent small phone in the near future and the whole world will flock to it. But nothing has pushed the industry in that direction yet.