hi, it's cpu, makes full use of simd on multiple cores and i got 3 tok/s. it's also memory mapped, so it stays in ram once it's loaded. the weights are 8 bit quantized. it can run as a server and is the backend of
https://x.com/paulpapers - a x bot, that reads all llm arxiv papers and shares the ones that are about running llms more efficiently.