You can go smaller.
Here I made 3x3, 3x4, 3x5:
https://twitter.com/tosuto_studio/status/1520071869529001984...
https://twitter.com/tosuto_studio/status/1520071869529001984...
Edit: I was wrong. Some of Simplifier characters like j actually use full 4 pixel height and poke out of 3x3 block.