Chat with, and help host, a free community LLM "horde"
lite.koboldai.net
lite.koboldai.net
- The AI Horde hosts a web app (Kobold Lite) geared towards LLM chat/instruct. Its mature, predating LLAMA and GPT 3.5 and largely developed when the RP community was running GPT-J finetunes. There are desktop apps that can access this API as well.
- The user sets the chat syntax/format and picks a LLM host (or multiple hosts).
- These hosts run API endpoints on their own PCs/servers for Horde users to access. The backends du jour are koboldcpp, a frontend for llama.cpp which is excellent, portable and literally one click, and KoboldAI, with the very fast and vram-efficient exllamav2 backend:
https://github.com/LostRuins/koboldcpp
https://github.com/henk717/KoboldAI
- Hosts pick a quantized community LLM to run, which is (IMO) the real magic of this system. Cloud services tend to run generic Llama chat/instruct models, OpenAI API models, or maybe a single proprietary finetune, but the Llama/Mistral finetuning community is red hot. New finetines and crazy merges/hybrids that outperform llama-chat in specific tasks (mostly Chat/Storytelling/RP) come out every day, and each one has a different "flavor" and format:
https://huggingface.co/models?sort=modified&search=mistral+g...
https://huggingface.co/models?sort=modified&search=13b+exl2
https://huggingface.co/models?sort=modified&search=20b
- The horde workers then earn "kudos" for serving requests to clients. Anyone can use the kobold horde with no login, but when requests are queued, kudos earn hosts priority access to LLMs other hosts are running.
I really like this scheme. Perverse incentives for spamming and such are minimal. Its free and easy for users on old hardware, but excessive leechers are deprioritized through the kudos system. It encourages experimentation with hosting (and trying) new finetunes, and gives hosts access to models they normally wouldn't know about or can't run. The horde API seems relatively simple and cheap to host, since its just text prompts/responses bouncing around. Its easy for me to host on a spare laptop, or in the background when I am not stressing my desktop GPU.
There seems to be tons of interest in making LLMs "easy" to play with and giving access to GPU-poor users, but Kobold Horde has somehow flown under the radar.