I feel like there's no point in trying to chop it down any smaller than that. All you're doing is shifting trust from "system A" to "system B", without really changing anything.
The reason for putting the encryption in system B AKA the keypad and display controllers is, if you want end to end encryption, the keypad decoder and the display controller are the closest to the end points you can get. And they are single purpose devices that don't run arbitrary code nor accept arbitrary input.
Compare that with laptop CPU's and complex operating systems with their huge attack surfaces. Running software written by completely untrustworthy corporations and individuals.
But be careful designing firmware upgrades though. Also I really don't see what kind of attack it prevents when you have a trusted keyboard and display but otherwise compromised OS.
What I'd like to know is why people though it was ever fully secure in the first place. It really never was.