Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI | Hacker News Reader

Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

together.ai

54 pointsbehnamoh22 comments

Better privacy might be running Llama2 locally, offline.

If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards.

  ! git clone https://github.com/ggerganov/llama.cpp.git

  ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P llama.cpp/models

  ! cd llama.cpp && make

  ! ./llama.cpp/main -m ./llama.cpp/models/codellama-7b.Q8_0.gguf --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"

Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well

just tried it and its not good. I think one of the better feedback was asking for play an adventure in the style of Space Quest 1

Or, for the easy way of doing it, just install Ollama and run: `ollama run codellama:7b`.

If you want the q8 version (which you probably don't), it's `ollama run codellama:7b-code-q8_0`.

Or if you're on macOS just hit 'Download' and use Ollama locally. [0]

[0] https://ollama.ai/download

It's quite easy to opt-out of OpenAI using your data for training. API and ChatGPT Enterprise are private by default.

# OpenAI Products and Privacy

- ChatGPT Free - trains on your data unless you Opt Out

- ChatGPT Plus - trains on your data unless you Opt Out

- ChatGPT Enterprise - does _not_ train on your data

- OpenAI API - does _not_ train on your data

Opt-out link: https://help.openai.com/en/articles/5722486-how-your-data-is...

I had no idea! Just submitted the form.

That said, the Enterprise version of ChatGPT is not easy to get by individuals.

Edit: the wording is confusing: "fill out this form to opt-in to share..." might lead the user to think they must submit this form to opt "out".

It’s not just training, but data retention and the ability for employees to access the prompts and completions.

You only touch half the problem

I can't recall hearing about this website before. In fact, only one link to this website has ever gained any traction here[0], and it wasn’t much.

Why is it surprising that a random website offering AI services might not have a great privacy policy?

I feel like there must be dozens of websites like this that offer hosting of Llama2 by now. My default assumption would be that they’re not as trustworthy as OpenAI, who has a lot to lose, where smaller companies have basically nothing to lose by comparison. If privacy is a primary concern, companies have to work to earn that trust.

[0]: https://news.ycombinator.com/item?id=36916762

This is not a random website - together is a prominent AI research company. Their chief scientist invented flashattention, which is used to train most LLMs. They release open source datasets/models like RedPajama. And they've published research around speeding up model training and newer model architectures.

I don't have any affiliation with them, but Together is hardly random and very important to the open source community, putting out some of the most useful work around things like new architectures, distributed training, datasets.

Any particular problems? Just linking to their policy is a bit confusing. You might consider submitting this as a question and linking to their policy in your summary.

Read the data collection part:

``` 2. Collecting and Using Your Personal Data 2.1 Types of Data Collected 2.1.1 Personal Data While interacting with the Services, You may provide certain personally identifiable information that could be used to contact or identify You. Personally identifiable information may include, but is not limited to:

Your user name for the Services

Text or image prompts you input into the Services, or public chats you maintain on the service

Your IP address

Usage Data

Tracking Technologies and Cookies

Contact Information

Organizational Information like your company title

Your Email

Other data that you elect to send to the Company ```

That's the normal way a privacy policy is worded. They start by describing the data they get from you, then they describe how that data is used. Obviously you're sending your prompts to them, so they note that and how they'll refer to that data in the rest of their terms.

The thing to pay attention to is how they describe the uses and sharing of that data. And yeah, those sections are super-broad amounting to "we can use all of your personal data however we want" (including "improve our Service" and a super wide "for other purposes").

> Personal data collected includes user names, text or image prompts, IP addresses, contact information, and more. The data is used for various purposes, such as providing and improving the service, managing user accounts, contacting users, and analyzing data. Personal data may be shared with service providers, third-party vendors, and other business partners. The company retains personal data for as long as necessary and transfers it to locations where data protection laws may differ. The security of personal data is important, although no method of transmission over the internet is 100% secure. The service is not intended for children under the age of 13, and the company does not knowingly collect their personal information. The privacy policy also includes information about links to other websites, changes to the policy, and supplemental terms and conditions for certain regions. For any questions or to exercise privacy rights, users can contact the company via email.

Generated here: https://dstill.ai/hackernews/item/37436498

This reads like something that a site using AdSense would use.

2.5 is interesting

> Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ from those in Your jurisdiction

> Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer.

And the table in the middle of the policy...

> Category Collected? Disclosed?

> Identifiers. Name, alias, postal address, unique personal identifier, online identifier, Internet Protocol (IP) address, email address, account name, or other similar identifiers YES YES

> Commercial information. Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies. YES YES

also interesting...

Invested in / partnered with NVIDIA & Hugging Face and others...

I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable.

If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claude).

I've had some success using vast.ai[0] with the Oobabooga LLM WebUI (LLaMA2) instances. One click to start up, minimal editing in the interface settings to enable OpenAI compatible interface.

[0] https://cloud.vast.ai/

We offer Llama-2-70B-chat api[0] at Deep Infra. $1 per 1M tokens, support streaming and OpenAI compatible endpoint[1].

0. https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf 1. https://deepinfra.com/docs/advanced/openai_api

Nice, awesome that you can just repoint an OpenAI client to this service. I'll give it a try.

Do you have future plans to support fine-tuning via pay-per-token billing?

Related: Found an interesting Reddit post (rare) discussing how OpenAI manages to do this by loading the LoRA layer at runtime. https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_...

You can run models at https://shadeform.ai in your cloud accounts or ours, and we don’t track or keep anything.

Reply on news.ycombinator.com