Comment by Someone1234 | Hacker News Reader

Comment by Someone1234 | Hacker News Reader

Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist:

- Easy plug & play model installation, and trivial to change which model once installed.

- Runs a local web server, so I can interact with it via any browser

- Ability to feed a model a document or multiple documents and be able to ask questions about them (or build a database of some kind?).

- Absolute privacy guarantees. Nothing goes off-machine from my prompt/responses (USP over existing cloud/online ones). Routine license/update checks are fine though.

I'm not trying to throw shade at the existing ways to running LLMs locally, just saying there may be room for an OPTIONAL commercial piece of software in this space. Most of them are designed for academics to do academic things. I am talking about a turn-key piece of software for everyone else that can give you an "almost" ChatGPT or "almost" CoPilot-like experience for a one time fee that you can feed sensitive private information to.

A few folks and I have been working on an open-source tool that does some of this (and hopefully more soon!) https://github.com/jmorganca/ollama

There's a "PrivateGPT" example in there that is similar to your third point above: https://github.com/jmorganca/ollama/tree/main/examples/priva...

Would love to know your thoughts

I'd love to test this out as soon as you get Linux or Windows support going!

Me too! I starred the repo and am watching releases, excited to try it.

I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission.

The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property.

What you are describing would help alleviate the concern about issue 2, but I’m not sure if it would help alleviate the concerns with issue 1.

Change company. Honestly. If you go as far as to forbid your partners in crime (workers sigh..) to explore new uncharted territory at all - well ya know someone will/might just win by not doing that.

Working for an 80.000+ employee company, one has already accepted a certain degree of inertia.

This is particular, specifically problematic territory. I cannot imagine handing over proprietary data to a third party without a contract in place for how that data is stored and used. It’s not about innovation, it’s about using someone else’s tools without ownership. For the other case, it’s both about integrity in owning your own work, and a shield from legal consequences. These things should be very relevant to any business.

I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. I am more bullish on LLMs than most of my coworkers. I think there is a lot of potential there. I do not see that potential in the current commercial offerings, and the financial outlay to fine-tune an open-source model and run it at scale is…prohibitive.

> I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy.

Really? I'm academic now but I find Copilot at least moderately helpful when I'm writing a library. It's pretty good a lot of boilerplate functions, docstrings, regex, etc. I certainly don't want to go back to not using it, my code is a lot closer to production quality now and looks nicer.

Thinking back to my days in back-end it seems like it would have been very helpful/sped things up so I'm surprised to hear it's just a toy but I've been out of the professional game for a while now. What's the main criticism?

That's not banning all uncharted territory, it's banning specific legally fraught territory.

It's basically the same thing in our company, too. They basically put a similar rule in place that prevents anyone from using e.g. Chat GPT. Little do they know that all software devs within the company are using co-pilot and the company is even paying for it. It's quite a funny situation tbh..

> Little do they know that all software devs within the company are using co-pilot and the company is even paying for it.

Just like annual sexual harassment training - it's mostly corporate CYA on liability. If it ever goes to court, they'll plead ignorance and blame the employees who should have known better as they were trained/informed on what they ought not to do.

Paying for co-pilot could bite them though, so I suspect it's a case were the one part of the organization isn't aware of what the other is doing

All of your assumptions are exactly right. They (mostly managers with little to no IT background) want to cover their own asses in case shit hits the fan (unlikely scenario if you ask me, because the company is just overrating the value of their data. Nobody gives a fuck about us anyway...) and many parts of this company have developed their own habits... The company is just very big and I can understand why they might be afraid, but come on, nobody will take that policy seriously forever. You need to eventually put some reasonable rules in place that allow you to make use of such Innovations...

Except, at my company they block software like that. Not only do they block it, but if you try to go to it a security person will immediately call your manager and ask what you are doing.

And people can access those systems via their own devices without the company knowing.

Well yes, but then I’d have to pay for it. That ain’t happening.

There may also be a third more important concern: AI/LLM generated works are not generally copyrightable.[1]

The Copilot lawsuit should answer concern #1 more definitively.

#2 is already solved by running your own model or using Azure OpenAI.

[1]https://www.copyright.gov/ai/ai_policy_guidance.pdf

we try to eliminate this problem by using code models trained only on permissevely licensed code, then you can run them locally without sending code anywhere

GPT4All satisfies these requirements, except for (AFAIK) running a web server

https://gpt4all.io/index.html

It runs a web server too - if you start up the desktop app it can run a web server with an API on a port for you.

I have been using refact.ai on my laptop, it has been quite good.

https://github.com/smallcloudai/refact/blob/main/README.md

Refact has worked for me. Hopefully exllama will support CodeLlama.

I wish these things worked with anything other than VSCode or JerBrains tools!

VSCode is such a bloated hog of an editor!

Every time I open VSCode it’s bugging with badges to update extensions… and it’s so slow!

having more plugin support is in our plans for sure. We're also open for contributions.

Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now.

That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.

I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup.

I've not yet been able to solve the challenge of needing CUDA etc for some models though!

Plugins so far: https://llm.datasette.io/en/stable/plugins/directory.html

Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.

Did you try Oobabooga (https://github.com/oobabooga/text-generation-webui) yet?

Oobabooga is a great tool but it still has a long way to go in term of user-friendliness. It's absolutely not plug and play the way that chatgpt is; It requires research, trial and error, and knowledge of the tech to make the model work to its full potential. It's great once you finish setting it up, but it does not compare to what you would expect from a commercial product aimed at normal end-users.

Things like bad default values, no tooltips, an no curated model list to one-click download is what separates a tool like Oobabooga from a paid commercial product. These things require time/money and it would be very unlikely that an open source tool could find resources for all the testing and R&D.

I think there is a big market for products where you pay and can just start chatting with the model without having to ever go to the settings tab or google anything unless you need to do something out of the ordinary.

What I want is even simpler: just an API that you make requests to and receive answers back. Surprisingly hard to find, outside OpenAI that is.

I'm not into open source LLMs in the slightest, and yet even I've trivially found tools to do what both you and the poster above you wanted

lmstudio actually does what both of you want: provides an easy GUI and serves up your model over a local endpoint that mirrors the OpenAI API.

There's just too much noise in terms of the tooling for LLMs, the solution is fewer higher quality solutions, not more solutions

https://gradient.ai/ is doing that with llama2

Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?

Good question, esp as Gradient fine-tuning is so much cheaper than Open AI's

Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.

Oobabooga exposes an API.

We’re trying to do this at https://openrouter.ai

We are doing this with https://Faraday.dev - would love to chat more - my email is on my profile!

You want a polished product and something that can run all sorts of different models. I don't know that both are possible at the same time.

we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE

I’ve used llama.cpp easily for some local things, but it does lack a good ui.

Reply on news.ycombinator.com