I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware.

https://arxiv.org/abs/2310.11453

https://github.com/cpldcpu/BitNetMCU/blob/main/docs/document...