I'm interested in the Banking77 example you gave which I want to reproduce. Can you give some details for this run? Also, does the <10 MB size include the embedding encoder? Would appreciate any configuration or code to look at!
https://gist.github.com/nicobrenner/056a5aaff5d0119c0032ecda...
The <10MB does not include the embeddings encoder
The gist uses BAAI/bge-large-en-v1.5, which is 1.2GB approx. You can replace it for all-MiniLM-L6-v2 (91 MB @ fp32 or 45 MB quantized fp16) small enough for mobile/edge. With all-MiniLM-L6-v2 it still gets 93.0% on Banking77, only 1.3 points behind bge-large at 15x smaller