Hacker News Reader

Comment by specproc | Hacker News Reader

I don't agree it's cheap. For generation at fairly small scale, sure, but generation is just the party trick. The real power for my use case lies in how much better it seems to do at traditional NLP tasks than an out-of-the-box model, with no further fiddling and faffing required.

Say I've got a corpus of ~1m documents, each of 10+ paragraphs and I want to run quote extraction on them (it does this beautifully), vectorise them for similarity search, whatever. This gets pretty expensive pretty fast.

View on HN

Tangential: you can finetune something like flan-ul2 to do quote extraction using examples generated from chatgpt. If you have a good enough GPU, it should help cut down costs significantly

Nice, that sounds like it's worth exploring. Much appreciated.

Again though, it's the zero-effort part that's appealing. I'm on a very small team and getting that to close to the same standard will take time for a ham-fisted clod like myself. Worth giving a shot all the same though, thanks again.

It's interesting what you can do with ChatGPT with few shot learning. It generalizes at the drop of a hat, often correctly.

The zero shot ability is convenient. But for tasks that you need to get done millions of times, I’d much rather spend $10 on GPU compute and maybe a day of training data generation to train a T5 which I then “own”.

Also, running your own specialized model locally can be much faster than using someone’s API.

Sure, purely a time issue for me. I'm not the most skilled in this area, and I've got a load of core stuff I need to keep on top of.

I think we're not far off having something equivalent that can be pulled from Huggingface and run on a near consumer grade GPU.

For now, I'll hang tight and see how things progress. Don't disagree.

Maybe one day you’ll be able to tell ChatGPT what kind of model you need and it’ll automatically select the right architecture, gather the training data, and commission the training using the cheapest and/or fastest provider. :)

Don't they have in the ToS you aren't allowed to use outputs for training downstream? Which is a little ridiculous, considering it's ToS.

But yea, they cheap cost and lack of training is making me a take a long hard look at how I'm implementing more traditional NLP solutions.

> Don't they have in the ToS you aren't allowed to use outputs for training downstream?

you mean this? "Data submitted through the API is no longer used for service improvements (including model training) unless the organization opts in" https://openai.com/blog/introducing-chatgpt-and-whisper-apis

was referring to "(iii) use output from the Services to develop models that compete with OpenAI; (iv) except as permitted through the API, use any automated or programmatic method to extract data or output from the Services, including scraping, web harvesting, or web data extraction;" ~ https://openai.com/policies/terms-of-use

I think I missed the exception for API, how ever not sure where they are, but seems to be fine based on alpaca. Also interesting they are so hard on web scraping and and extraction, lol. But wow, that is a poorly worded paragraph.

I do this. It works.

Can you elaborate? Did some brief Google searching but had issues putting it together. We have thousands of documents and data stores we'd like to parse using GPT-3.5 (or the new ChatGPT API) and have been thinking of pretraining to cut things down. Thank you!

contact me at the email in my profile

1000x this. Entity extraction from unstructured text with zero/few-shot is fantastic.

I've got a use case where I need to extract model numbers from text - these LLMs are so good at it with very little work.

Be careful with NER, while it is very good, it is not perfect.

Example, I tried to extract skills from a job posting. ChatGPT did well, but there were skills missing.

It is good to find some entities but then you need to extend the labeling manually.

For sure. If perfect accuracy is important, it's still good to do sampling and human review to figure out accuracy rate and decide if further checking is required. But it still beats the pants off accuracy of other methods for the amount of work required.

That said, with fine-tuning, `davinci-003` is _excellent_ at the types of entity extraction you're describing.

There is no such thing as davinci-003 per se. There is text-davinci-003 which you can't fine tune, and davinci which you can.

Sure, but SOTA on NER is around 90% for things like Names and Places. For skills, a Spacy model will get around 60-70%. And training a Spacy model takes a bit of fiddling.

Checkout Argilla for annotation. You can use vectors to speed up the annotations and you can also start with zero-shot feedback to improve the training as you go

https://www.argilla.io/

The cost may be high but still worth it depending on the alternative. I wonder what is happening with this in the evidence mining tools lawyers use for example.

I’d wager it could cost anywhere between 1-10k to do that, which is a considerable amount of money. Might still be worth it though? If the alternative is mechanical turk, that would probably cost x1000-10000? Are there any ML alternatives that reliably produce useful results?

It would cost less than $50 of compute to finetune an existing open source LLM to do the job for another $50 of computer. OpenAI's APIs are incredibly expensive compared to powerful open source LLMs which run on very accessible hardware.

As one data point, LLaMA-13B beats GPT-3 175B in benchmarks, runs on a single 8GB VRAM consumer GPU, and takes only 24GB of VRAM to fine tune. (Though this particular model can't be used for commercial purposes.)

I actually read about LLaMA after posting the above comment, and I do agree with you. You probably can't use it OOTB right now because LLaMA prompts seem harder to get right, but seeing the rate of progress, that will probably be solved in a few weeks.

What’s the alternative? Hiring humans to do the job for you? Probably much more expensive.

A cheaper alternative is fine-tuning more specialized open-source model and hosting it on your own hardware.

What happens when you hit a guardrail because there was a politically-charged or socially-condoned phrase? What if the documents are potentially sensationalist news articles or Wikipedia articles with potentially-questionable topics?

Yes, however, it can generate a training dataset to fine tune a smaller model really cheaply, which can then scale.

Isn't there a limit on the input length? Like you couldn't put a whole research paper into it?

How expensive, if you would actually run the numbers for your use case?

How much would that example cost, for a sense of scale?

I suspect the author doesnt realise one request with hardly anything returned is many hundreds if not thousands of "tokens". It adds up very fast. Just some debug effort on a nonsense demo learning project cost $5 in a couple of hours. For maybe a hundred or so requests.

I used it for dozens of requests yesterday and that amounted to less than 7 cents. I used MacGPT for that.

So 7 cents for dozens of requests is only about 1/10th what I was saying. So could be I have the old API, but even that 7 cents for 10s of requests is not cheap compared to executing a model yourself at scale.

It's cheap enough to not bother about the price even for casual use.

That's straight up not true, unless that "demo learning project" is feeding GPT the entire Bible or something.

I have a project that uses davinci-003 (not even the cheaper ChatGPT API) like crazy and I don't come close to paying more than $30-120/month. With the ChatGPT API, it'll be 10x less...

You're right it was $3.74 https://ibb.co/gRLQBZ4 Returning under 300 characters and the prompt sent was about 10 words in length.

You could have saved some money by writing tests. How much text were you sending at a time? I’ve been summarizing multiple 500 word chunks per query in my app as well as generating embeddings and haven’t broken $10 over the course of a couple weeks.

Sure but at some point you're testing prompt generation and what happens with the model, thats what Im talking about. This is a basic session with a couple of people clicking cost that much. So clearly Im doing something far more wrong from the API side to get whats clearly magically worse billing than everyone else here.

They charge per 1k tokens so you must have high volume somehow, are you maxing out the prompt length every time? That’s the only thing I can think of besides sending a ridiculous number of requests that would cost that much in an evening.

It is not possible to pay anywhere close to $5 for a hundred requests, even if you used the max payload size every time.

Is it possible you had a bug that caused you to send far more requests than you were intending to send? Or maybe you used the older models which are 10x more expensive?

Could be I used an older API with the newer model. But there was no loop around the request only human input with mouse clicks from two people. Whatever was happening on the billing side there is zero chance Id ever post a project to HN for example.

         fetch("https://api.openai.com/v1/chat/completions", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                    Authorization: `Bearer ${window.localStorage.getItem('apikey')}`,
                },
                body: JSON.stringify({
                    "messages":[
                        {"role":"system","content":""},
                        {"role":"user","content":""}
                    ],
                    "temperature":0.9,
                    "max_tokens":300,
                    "top_p":1,
                    "frequency_penalty":0,
                    "presence_penalty":0.6,
                    "model":"gpt-3.5-turbo",
                    "stream":false}),
                })
            ])

Weird. My other guess was that you might be using best_of but you aren't doing that either.

I can understand making a mistake on the Internet, but to say it with such snarky gusto is inexcusable.

I’ve been playing with davinci pretty extensively and the only reason I’ve actually given OpenAI my credit card was because they won’t let you do any fine-tuning with their free trial credit, or something like that. You’re off by orders of magnitude, ESPECIALLY with the new 3.5 model.

Youre reading the snarky gusto in your head. My point was literally that small mistakes and even something operational scaled beyond extremely small user bases is not "cheap". If two humans clicking is five bucks in an afternoon. Regardless of how it happened. If id linked whatever I had done here Id be easily looking at 10k for people like you to assume bad faith. Its especially not cheap compared to using a smaller language model locally for anything but generation.

Reply on news.ycombinator.com