we recently shipped secure mode on https://www.agentsea.com.
With Secure Mode, all chats run either on open-source models or models hosted on our own servers - so you can chat with AI without worrrying about privacy.
edit: there are some instances where i would like to be able to set the same seed repeatedly which isn't always possible online.
No, they can run quantized versions of those models, which are dumber than the base 30b models, which are much dumber than > 400b models (from my use).
> They are a little bit dumber than the big cloud models but not by much.
If this were true, we wouldn't see people paying the premiums for the bigger models (like Claude).
For every use case I've thrown at them, it's not a question of "a little dumber", it's the binary fact that the smaller models are incapable of doing what I need with any sort of consistency, and hallucinate at extreme rates.
What's the actual use case for these local models?
If anyone has a gaming GPU with gobs of VRAM, I highly encourage they experiment with creating long-running local-LLM apps. We need more independent tinkering in this space.
Again, what's the use case? What would make sense to run, at high rates, where output quality isn't much of a concern? I'm genuinely interested in this question, because answering it always seems to be avoided.
This is the exact opposite from my tests: it will almost certainly NOT work as well as the cloud models, as supported by every benchmark I've ever seen. I feel like I'm living in another AI universe here. I suppose it heavily depends on the use case.
For me, none really, just as a toy. I don't get much use out of online either. There was Kaggle competition to find issues with OpenAI's open weights model, but because my RTX gpu didn't have enough memory i had to run it very slowly from with CPU/ram.
Maybe other people have actual uses, but i don't
1. Profound tone-deafness about appropriate contexts for privacy messaging
2. Intentional targeting of users who want to avoid safety interventions
3. A fundamental misunderstanding of your ethical obligations as an AI provider
None of these interpretations reflect well on AgentSea's judgment or values.
Anyone with half a brain complaining about hypothetical future privacy violations on some random platform just makes me spit milk out my nose. What privacy?! Privacy no longer exists, and worrying that your chat logs are gonna get sent to the authorities seems to me like worrying that the cops are gonna give you a parking ticket after your car blew up because you let the mechanic put a bomb in the engine.
Just not a very good argument.
To play devils advocate for a second, what if someone that’s mentally ill uses a local LLM for therapy and doesn’t get the help they need? Even if it’s against their will? And they commit suicide or kill someone because the LLM said it’s the right thing to do…
Is being dead better, or is having complete privacy better? Or does it depend?
I use local LLMs too, but it’s disingenuous to act like they solve the _real_ problem here. Mentally ill people trying to use an LLM for therapy. It can end catastrophically.
> Is being dead better, or is having complete privacy better? Or does it depend?
I know you're being provocative, but this feels like a false dichotomy. Mental health professionals are pro-privacy AND have mandatory reporting laws based on their best judgement. Do we trust LLMs to report a suicidal person that has been driven there by the LLM itself?
LLMs can't truly be controlled and can't be designed to not encourage mentally ill people to kill themselves.
> Mentally ill people trying to use an LLM for therapy
Yes indeed this is one of the core problems. I have experimented with this myself and the results were highly discouraging. Others that don't have the same level of discernment for LLM usage may mistake the confidence of the output for a well-trained therapist.