Show HN: Singulatron – On-Premise AI Platform | Hacker News Reader

Show HN: Singulatron – On-Premise AI Platform

github.com

Great job! Can't imagine amount of hours invested into it. How does the offline version perform in different areas comparing with known LLMs (e.g. ChatGPT 4o, LLama, etc)?

Thanks gpaslari,

We are running known models like LLama, StableDiffusion etc. So pretty much the same :)

We are not planning on building our own models yet.

Hey folks,

Op here. Trying to build a distributed AI platform - lots of work, fairly early phase but I think it's already useful. Hope you'll like it.

Thanks!

Why reinvent Kubernetes?

I'm not planning to :)). Singulatron is not about scheduling, etc.; it operates at a much higher level of abstraction. To put it in less fancy terms: it's an app, albeit one designed to run on many servers simultaneously.

Why reinvent half of Kubernetes?

So any web app that runs on multiple nodes reinvent k8s? I'm not sure about that.

> So any web app that runs on multiple nodes reinvent k8s

Most of them only reinvent 1/10th-1/100th of Kubernetes.

Are you distributing the LLM generation tasks at a low-level (server 1,2,3,4 each have a GPU and each get 25% of the model + work somehow)? Or is this just a way of interfacing with an existing LLM using a standardized interface on front and back ends?

Hey!

At the moment the distributed features are entirely missing from the repo, sorry to disappoint on that front despite being part of the readme. It's hectic!

But to answer your question: I plan to sidestep most difficulties of the distributed story by offloading into the usual distributed tools - horizontally scalable DBs and distributed locks, queues etc. This will have the added upside of being able to use your favourite tools for which you already have dashboards, alerting etc.

There will be no fancy resource management (an other comment touched in "reinventing kubernetes" which is definitely not a goal at all), but there will be some simple optimizations in terms of where to run which prompt depending on if the model is already running on a node or not etc.

So there will be some knowledge of the nodes' capacity (at least available VRAM etc), but the picking of the prompts from the queue will be a simple unit testable algorithm.

At least this is the plan but it's only half implemented in my mind.

Reply on news.ycombinator.com