Show HN: Kalosm – a local first AI meta-framework in Rust

Kalosm is a meta-framework for AI written in Rust. Kalosm supports local quantized large language models like Llama, Mistral, Phi-1.5, and Zephyr. It also supports other quantized models like Wuerstchen, Segment Anything, and Whisper. In addition to local models, Kalosm supports remote models like GPT-4 and ada embeddings.

What does Kalosm do?

1) Gather context from URLS, RSS feeds, web crawling, a search query, local files, or even your current surroundings!

2) Insert that context into an easily searchable vector database with Bert or a keyword search index

3) Generate text, images, or structured data with your context!

One of the features I am most excited about is constrained generation which lets you control exactly what the language model outputs. This makes it possible for even very small models to generate coherent JSON or conform to your custom grammar. It also unlocks advanced tool support with custom constraints for each tool input that ensures the text is valid.

Why did I build Kalosm?

I am currently working on a node editor for AI workflows called Floneum. I found that the existing tools didn't help users gather and integrate context with language models, so I built Kalosm. Kalosm will also enable Floneum to more easily gather context and richly integrate that context into multiple modalities.

Everything in Kalosm is local first!

Local context, local models, local inference: Kalosm is built from a laptop to run on your laptop. It doesn't require a powerful GPU to run any of the models Kalosm supports. Just download the library and start tinkering!

If you are interested in Kalosm, have questions, or run into issues, consider joining the discord (https://discord.gg/dQdmhuB8q5)! I have tried to make kalosm easy to get started with, but if you run into issues or unclear APIs feel free to open an issue or discussion. There are over 20 examples throughout the repo and additional examples in the inline documentation. Every public API is documented if you run cargo doc.

Let me know what you think about Kalosm. I would love to hear your feedback or feature suggestions! If you like it, consider giving it a star