The Pi Zero can do this as a USB client device for a host PC, including the QRCodes with the Pi camera, buttons/switches, and a small touchscreen for host-isolated verification and pin entry. Beaglebone Black (possibly also Pocket Beagle) can do it too, and the am335x does have some minimal crypto acceleration.
They're not high performance, they aren't "security focused" hardware, nor are they perfect fit for the task, but they're reasonably well understood and broadly available. The Pi Zero does have a closed "GPU" firmware, but there is an effort[1] to run open firmware on it.
However if you expect the host to attack the device, USB OTG (i.e. Linux USB gadget drivers) may not be a good choice, you may want to access it via network instead and that opens up more choices (though most will not be as small).
The other alternatives are basically going to be micro-controllers, for example FST-01[2]/Gnuk[3], and FPGAs which are still more of a black box than CPUs at the moment.
[1] https://github.com/christinaa/rpi-open-firmware