Hacker News Reader

Comment by alchemist1e9 | Hacker News Reader

The market for completely anonymous and uncensored models via API must be huge. How come nothing exists yet?

Completely anonymous and uncensored anything is hard to come by. You really can't legally provide a service like that in most Western countries. Every service that has tried has ended up realizing why it doesn't exist.

Even if you tried to base your business outside of the West, you're left figuring out how to collect payments for it without getting blocked. LLMs are far more expensive to provide than plain text communication or a VPN.

Eventually it'll probably happen, but there are a lot of logistics to sort out first, and in the meantime the open source models still have a ways to go before they're good enough at arbitrary problems out of the box to be worth paying for.

Why trust someone else if you want it to be anonymous? Buy a proper GPU, run a local model and that's it handled. Basically all open source inference engines support OpenAI's API format and act as drop in replacements.

"just spend $1000 on a new GPU"

How much do you think those API calls will cost you that require someone else to buy the GPU, maintain the server architecture, and also take a cut in the process? It only takes a 3-4 months to amortize if you want the instance to be up 24/7. And in the end you can still resell the GPU.

I rent spot instances on GCP with big GPUs on demand. It works quite well and since I'm just playing around I pay a very small amount per hour and kill it after I'm done.

Does this work on-demand? Could I build something I can query from my phone as needed?

Depends on your budget. If it's the absolute minimum, it won't be real time responses. As provisioning infra isn't instantaneous.

Just want to point out that $1000 is nothing to a lot of potential bad actors. $1 million is nothing.

Plenty exists

LM Studio lets you load in models from a built in HuggingFace browser, and sets up a server that accepts OpenAi 3.0 API requests, so everyone that built an app around ChatGPT API plugs in exactly the same

I'm working on this problem in my spare time, it requires a GPU (or two) RTX 3090 or 4090 because of the need for vRAM memory but I could also forsee APIs in front that allow worker nodes to run the quantized models for cheaper. the problem is model switching is hard so you would have to pick which model to run upfront for inference

openchat in particular ships with an inference server compatible with openai

follow me here: https://www.youtube.com/@fxhp1

https://replicate.com does in fact already exist.

Can’t this bot developer just switch to them then? as replicate can’t know what it is being used for I would assume. Perhaps there was fine tuning or RAG being used that still needs a private and anonymous service solution.

They have a list of prohibited uses which, though not has extensive as OpenAI's, includes this gem at the end (emphasis added):

> To engage in any other conduct that restricts or inhibits anyone’s use or enjoyment of Service, or which, as determined by us, may harm or offend Company or users of Service or expose them to liability.

Edit: Also, their privacy policy makes it clear that "completely anonymous" is not a thing they offer— they collect emails and IP addresses, transmit them to the United States, and will deliver them to law enforcement upon request.

https://replicate.com/privacy

So it’s not anonymous then because they see what the customer is generating? Otherwise how can they know what it’s being used for?

Yes, it's definitely not anonymous. I added to my comment right as you replied—they collect IP addresses and usage data and explicitly say they will share that with law enforcement upon request. It's a US-based company, so they kind of have to.

Let’s say there exists some “illegal” or “criminal” output of an LLM, doesn’t that mean by definition the government has criminalized speech and thought? since the output of an LLM is by definition tokens of language.

Now image generation I guess we can see as different since we do agree some images or videos have content society sees as illegal.

The whole topic is fascinating to me. Censorship and regulation around generative AIs.

The issue with this is the same one that exists with commercial VNP offerings. It is impossible for the user to verify claims of any company that guarantees anonymity/doesn't track/doesn't keep logs/etc.

Reply on news.ycombinator.com