Currently I am using llama.cpp for an interactive repl chat. I was previously using Alpaca (a GTK GUI), but was annoyed with how slow it was and some random crashes. I am transitioning some of this to self hosted in the cloud for things that can't run on my laptop.
I am looking to get away from my current interface, and write my own. Mostly for experience of deeply integrating agents into a program. If anyone knows a good library for interacting with a local model that doesn't involve standing up a webserver I am interested :)
My daily driver is gemma3n. Its been a nice balance between speed and performance without spinning up my laptop fans.
I am super interested in local models, partially because there is no friction from managed services, but also because I think as small models become more viable we will see an explosion of apps incorporating them.