Hacker News Reader

Comment by tikkun | Hacker News Reader

This is the first time I've felt like Anthropic may be a true competitor to OpenAI.

I see 6 ways to improve foundation LLMs other than cost. If your product is best at one of the below, and has parity at the other 5 items, then customers will switch. I'm currently using GPT-4-8k. I regularly run into the context limit. If Claude-100K is close enough on "intelligence" then I will switch.

Six Dimensions to Compare Foundation LLMs:

1. Smarter models

2. Larger context windows

3. More input and output modes

4. Lower time to first response token and to full response

5. Easier prompting

6. Integrations

View on HN

7. Price!

GPT4-32K costs ~$2 if you end up using the full 32K tokens, so if you're doing any chaining or back-and-forth it can get expensive very quickly.

Oof, got access to the 8k model recently and was wondering what costs would be on the 32k one. That's brutal.

Why? 32K tokens would cover creating source code for a small to medium programming project. THat would easilly cost hundreds of dollars if done by a freelancer, and could get it in almost real time for just $2.

32k tokens is 500-1000 lines of code, so more like thousands of dollars, unless you're comparing against a landing page or CRUD tool arranged through Upwork. On the flip side, before I dove into AutoGPT I couldn't get GPT-4 to iterate on something as simple as a TypeScript function that deeply converts { foo_bar: string } to { fooBar: Date } without running out of context and cyclically reintroducing regressions.

Advice on where to get started with auto-gpt?

Only if you can get that result in one pass.

The real utility of LLMs is that they can be called in a loop to scan through many web pages, many code files, issue tickets, emails, etc...

There are already demos and experiments out there that for every input, 4x outputs are generated, then those are fed back into the LLM 4x for "review", then the best variant is then used to generate code which then automatically tested, errors are fed back in a loop, also with 4x parallel tries, etc...

It's the throughput compared to humans that is the true differentiator. If hooking up the API in a loop up ends up costing more than a human, then it's not worth it.

> It's the throughput compared to humans that is the true differentiator. If hooking up the API in a loop up ends up costing more than a human, then it's not worth it.

If its better in another dimension (e.g., calendar elapsed time), it may well be worth being more expensive.

Two dollars spent over and over would still take a while to equal a software engineer's salary, and if the end result is "produce a fully functional codebase in minutes" the premium might be worth it even if it exceeds that much.

We all know the game Sam Altman is playing. Eventually you can squeeze the full salary in cost out of companies who have laid off their workers and it would still be a good deal because AI does not need health care, pay taxes, hardware, sick days and so on.

Also, the azure offering is 10 times the openai price, so not really usable right now.

It's exactly the same pricing (East US):

https://azure.microsoft.com/en-us/pricing/details/cognitive-...

https://openai.com/pricing

Oh boy. I hope Microsoft doesn't get tired of subsidizing my access through Bing. I quite like it.

Bing seems to be using the stock 4K window size, which is likely why it's limited to that many messages in a single chat session before it forcibly shuts it down.

« other than cost »

>Six Dimensions to Compare Foundation LLMs

I'd add open source to the list, which neither "open"AI or this is.

now that I think about it

is it that important to open source models that can only run on hardware worth tens of thousand of dollars?

who does that benefit besides their competitors and nefarious actors?

I've been trying to run one of the largest models for a while, unless 30,000$ falls in my hand I'll probably never be able to run the current SOTA

> is it that important to open source models that can only run on hardware worth tens of thousand of dollars?

Yes, because as we've seen with other open source AI models, it's often possible for people to fork code and modify it in such a way that it runs on consumer grade hardware.

I agree utility of open source for personal usecase is overblown.

But for commercial usecases, open source is very relevant for privacy reasons as many enterprises have strict policy not to share data with third party. Also it could be a lot cheaper for bulk inference or to have a small model for particular task.

However, the same thing could be achieved with closed source models. There's nothing to stop an LLM being made available to run on prem under a restrictive license. It would really be no different to ye olde desktop software - keeping ownership over bits shipped to a customer is solved with the law rather than technical means.

That said, I really hope open source models can succeed, it would be far better for the industry if we had a Linux of LLMs.

> Keeping ownership over bits shipped to a customer is solved with the law rather than technical means.

Yes in theory... In practice, what happened with LLaMA showed people will copy and distribute weights while ignoring the license.

Locally hosted instances that don't report on prompts is important for personal privacy.

Even a small startup, a researcher or a tinkerer can get a cloud instance with a beefy GPU. Also of note, Apple's M1 Max/Ultra should be be able to run it on their GPUs given their 64/128GB of memory, right? That's an order of magnitude cheaper.

I am confused. Those amounts are ram, not gpu ram, aren‘t they? Macs cpus are impressive, but not for ml. A most realistic one for a consumer is a 4090 rtx 24 GB. A lot of models do not fit in that, so A6000 48GB and over for some professional cards. That might be around 9000€ already.

> Macs cpus are impressive, but not for ml

On Mac GPU has access to all memory.

Apple Silicon has unified memory - all memory is accessible to both the CPU and GPU parts of the SoC.

But they comes at max 32GB model?

Mac Studio (desktop) is up to 128GB, and Macbook Pro is up to 96GB.

I overlooked the unified memory on those machines. Can it really run this performantly?

I run Vicuna quite well with my M1 Pro, 32GB.

Yes, because it can always be down ported by people with more constraints than the original authors. We’ve see a lot of this in the LLM space, and a lot of other OSS efforts.

When linux was first released in 1991 a 386 to run it would cost about $2000.

We've already seen big advancements in tools to run them on lesser hardware. It wouldn't surprise me if we see some big advancements in the hardware to run them over the next few years, currently they are mostly being run of graphics processors that aren't optimised for the task.

$30000 is less than price of average car that Americans buy (and most families have two of them) - that's definitely in the realm of something that affluent family can buy if it provides enough value. I also expect price to go down and at $10k it's less than mid-range bathroom update. The question is only if it provides enough value or using in the cloud better option for almost all families.

They don't only run on high end systems. Good models can run on a desktop you have at home. If you don't have a desktop... I'm not sure what you're doing on HN.

You have a weird definition of "good model"

Llama 7B is NOT a good model.

You can run much larger models than llama-7B. Galpaca-30b or Galactica-120b for example.

30B is still not good enough.

What kind of desktop are you running a 120B model on with reasonable performance?

I would disagree that 30B is not good enough. It heavily depends on which model, and what you're trying to use it for.

30B is plenty if you have a local DB of all of your files and wiki/stackechange/other important databases places in a embedding vectordb.

This is typically what is done when people make these models for their home, and it works quite well while saving a ton of money.

While llama-7B systems on their own may not be able to construct a novel ML algorithm to discover a new analytical expression via symbolic regression for many-body physics, you can still get a great linguistic interface with them to a world of data.

You're not thinking like a real software engineer here - there are a lot of great ways to use this semantic compression tool.

If I just wanted a fuzzy search engine for local data, I'd use a vector DB - there's no need for an LLM on top of that.

It will create price competition for different providers of the model though, which should drive down prices

"It only benefits bad people" is a pretty shitty argument at this point tbf. You can apply this logic to any expensive thing at this point.

I can for example, afford the hardware worth tens of thousands of dollars. I don't want to, but I can if I needed to. Does that automagically make me their competitor or a bad actor?

I don't think most of the large customers will care about OSS AI. Over the last decade they've learned (trained themselves?) where to put their money towards (cloud vs. in-house infra for all manner of things, for better or worse) and I think AI tools will follow similar trends.

Businesses will certainly care about cost, but just as important will be:

- Customization and fine-tuning capabilities (also 'white labeling' where appropriate)

- Integrations (with 3rd party and in-house services & data stores)

- SLA & performance concerns

- Safety features

Open Source AI will have a place, but may be more towards personal-use and academic work. And it will certainly drive competition with the major players (OpenAI, Google, etc) and push them to innovate more which is starting to play out now.

Companies that aren't mindful of vendor lock in aren't long for the world.

Though those cloud platforms all have their own proprietary components most users are savvy enough to constrain and compartmentalize their use of them lest they find themselves having all their profits taken by a platform that knows it can set its prices arbitrarily. The cloud vs in-house adoption is what it is in large part because the cloud offerings are a commodity and a big part of them being a commodity is that much of the underlying software is free software.

History is littered with companies that went dead because they focused on things that don't matter (open source, anti-microsoft, pro-linux).

There will be a time when those things matter when it hurts the bottom-line (Dropbox), but to prematurely optimize for that while you are finding product-market-fit is crazy and all companies are finding product-market-fit in the new AI era

On the other-hand, companies that fall behind their competitors because they are spending time on adjacent activities rather than leveraging new capabilities on the market will also lose out. It isn't clear at this time that LLMs fall into that categories ... and smart applications of in-house systems can be as much as an advantage as badly done NIH projects can be an albatross.

A lot of B2B startups can technically the cloud API to provide value added applications to Enterprises, but often the banks and healthcare companies will not want their data running through startups pipes to OpenAI pipes.

We provide a low code data transformation product (prophecy.io), and we’ll never close sales at any volume, if we have a to get an MSA that approves this. Might get easier if we become large :)

> I don't think most of the large customers will care about OSS AI.

One would think the same in the 90s but yet, for some reason, Open Source prevailed and took over the world. I don't believe it was about cost, at least not only. In my career I had to evaluate many technical solutions and products and OSS was often objectively superior at several levels without taking account the cost.

The first really successful alternative to "Open"AI will:

* gather many talented developers

* will quickly become a de facto standard solution

* people will rapidly start developing a wide range of integrations for it

* everybody will be using it, including large orgs, because, well, it's open source

True, but the difference here is that running a performant and capable AI solution will be infrastructure-dependent, which has real costs.

Yes, and I believe it will develop in two ways over decades. Just like all major legacy hosting companies such as DigitalOcean, OVH or Hetzner have been offering a kind of public cloud services (at various levels, and after much feet dragging), they - and new AI hosting providers focusing exclusively on this use case - will provide the necessary services.

The other trend is the one we are already seeing right now: more and more mature solutions that you can use even on your laptop with a relatively new GPU. I'm sure we'll see some interesting results in this area, too.

Open Source hasn't really taken over the world, if you look at end-solutions.

As a software developer I might use an open source database, but as end-user I'm probably not going to use open-source accounting package - but I will use an accounting SaaS system that happens to be implemented with that OSS DB.

As a software developer I might use an OSS operating system, but as end-user I use a software that has been packaged and maintained by corporation like OSX, or even if OSS in license, has been fully packaged like Android.

Yes, but I think for most companies this has more to do with cost. They're not going to pay for the OSS model, and if they can use an OSS model + fine tuning, they'll choose to save the money.

Here's a really important reason to care about open source models: prompt engineering is fiddly enough without the risk of your model provider "upgrading" the model you are using in a way that breaks your existing prompts.

OpenAI already upset a lot of (admittedly non-paying academic) users when they shut off access to the old Ada code model with only a few week's notice.

I’m curious about how enterprises will manage model upgrades.

On one hand, as you mention, upgrades could break or degrade prompts in ways that are hard to fix. However, these models will need constant streams of updates for bugs and security fixes just like any other piece of software. Plus the temptation to get better performance.

The decisions around how and whether to upgrade LLMs will be much more complicated than upgrading Postgres versions.

Why would the models themselves need security fixes? The software running the models, sure, but you should be able to upgrade that without changing anything observable about the actual model.

Not necessarily. The insecurity from LLMs comes from the fact they’re a black box - what if it turns out that particular version can be easily tricked into giving out terrorism ideas. You could try to add safeguards on top, but they’ve already been bypassed if it has been used for something like that. You might just have to retrain it somehow to make it safe

LLMs (at least the ones with read/write memory) can exactly simulate the execution of a universal Turing machine [1]. AFAIK running such models will therefore entails the same fundamental security risks as ordinary software.

[1] https://arxiv.org/pdf/2301.04589.pdf

Paying users who need this kind of stability are more likely get access to those models via Azure rather than from OpenAI directly, which comes with the appropriate enterprise support plans and guarantees.

The OpenAI APi has model checkpoints, right now the chat options are:

gpt-4 gpt-3-5-turbo gpt-4-0314 gpt-3-5-turbo-0301

Notably absent from the available model list is code-davinci-002 - a lot of people were burned by that one going away.

Those are ChatGPT models. The code-davinci-002 model is still available - they responded to community requests to keep it up.

The 3.5 legacy model disappeared from the ChatGPT UI recently. Is it still available via the API?

Midjourney does this, as well.

>I don't think most of the large customers will care about OSS AI

Problem again, is centralization of LLMs by either the governments (and they always act in your best interest, amirite?) and corporation, which Non-FOSS LLMs prevent.

Democratization of the models is the only way to actually prevent bad actors from doing bad things.

"But they'll then have access to it too" you say. Yes, they will, but given how many more people who will also have access to open LLMs we'd have tools to prevent actually malicious acts.

A good guy with an LLM stops a bad guy with an LLM. - This message brought to you by the National LLM Association

Ah yes, the only other alternative is to give companies and the State sole access to LLMs, and guns, and drones, and militarized police. Because they'll surely only think of YOUR benefits, right?

> I don't think most of the large customers will care about OSS AI.

OSS AI will open up more diverse and useful services than the first-party offerings from relatively risk averse major vendors, which customers *will" care about.

The thing to remember when selling to businesses is that a business is just a stack of people in a trench coat. This might sound a tad evil but you don’t have to offer something that benefits the business as a whole, just something that benefits the person who holds the purse strings.

This is why cloud services are so popular. They’re easy and they don’t cost the decision makers personally.

Considering the very smart people asking for a moratorium on AI development, and it's potential to disrupt a lot of jobs, this may be a good thing.

One question is how much other factors really matter compared to the raw "intelligence" of the model--how good its completions are. You're not going to care very much about context window, prompting, or integrations if the output isn't good. It would be sort of like a car that has the best steering and brakes on the market, but can't go above 5 mph.

Strongly agree. They are ordered by how much I think they generally will lead to users choosing one model over the other.

Intelligence is the most important dimension by far, perhaps an order of magnitude or more above the second item on the list.

On that note, can anyone speak to how Anthropic (or other models) are doing on catching up to OpenAI for pure model intelligence/quality of completions? Are any others approaching GPT-4? I've only used GPT-based tools so I have no idea.

The best claude model is closer to GPT-4 than 3.5

Big question on that for me is that there's a variety of "completion styles" and I'm curious how "universal" performance on them is. Probably more than this, but a quick list that comes to mind:

* Text summary/compression

* Creative writing (fiction/lyrics/stylization)

* Text comparison

* Question-answering

* Logical reasoning/sequencing ("given these tools and this scenario, how would you perform this task")

IMO, for stuff like text comparison and question-answering, some combo of speed/cost/context-size could make up for a lot, even if they do "worse" versions of stuff just that's too slow or expensive or context-limited in a different model.

We already see how that goes. Stackable LoRa like in stable diffusion.

I don't know. While using Phind I regularly get annoyed by long prose that doesnt answer anything (yes, "concise" is always on). Claude seems to be directly geared towards solving stuff over nice writing.

I generally add to my initial prompts to GPT4 to: From now on, please use the fewest tokens possible in all replies to save tokens and provide brief and accurate answers.

Or rather, more analogously, a self-driving car that has a range of 10 000 miles but sometimes makes mistakes when driving vs a self-driving car with a range of 800 miles that never makes mistakes. Once you've have a taste of intelligence it's hard to give up.

However, in many applications there is a limit on how intelligent you need the LLM to be. I have found I am able to fall back to the cheaper and faster GPT-3.5 to do the grunt work of forming text blobs into structured json within a chain involving GPT-4 for higher-level functions.

Until they actually make any of it available in anything but an obscure expensive API you have to request access to, they might as well not even exist.

there are many services that integrate with them that would allow you to self-serve signup

I've been using it through poe and I prefer it to ChatGPT but can't pinpoint why. It just "gets" me better I guess?

The landing page says "Easy integration via standard APIs Claude can be incorporated into any product or toolchain you’re building with minimal effort." Then there is a big button "Request Access", which for me right now just does nothing. OpenAI has really faced the pain to make their product available via an API to the general public at scale, but Anthropic/Google/etc. don't quite seem to be there yet. It's frustrating.

Try a browser or a clean profile without any ad blocking turned on. It took me a couple of tries to figure out how to get it working but you should see a modal with a form when it works.

FYI the waitlist form submits a regular POST request so it'll reload the main page instead of closing the modal dialog. I opened network monitor with preserved logs to double check that I made it on the list :facepalm:

I don't think the person you're responding to wants a network based or cloud based solution.

When someone says they want it available they mean running on their own device.

This is hackernews, nearly everyone on this site should have their own self hosted LLM running on a computer/server or device they have at their house.

Relying on 'the cloud' for everything makes us worse developers in just about every imaginable way, creates a ton of completely unnecessary and complicated source code, and creates far too many calls to the internet which are unnecessary. Using local hard drives for example is thousands of times faster than using cloud data storage, and we should take advantage of that in the software we write. So instead of making billions of calls to download a terabyte database query-by-query (seen this 'industry-standard' far too many times), maybe make one call and build it locally. This is effectively the same problem in LLMs/ML in general, and the same incredible stupidity is being followed. Download the model once, run your queries locally. That's the solution we should be using.

When I want code that has a reasonable chance of working, or to bounce ideas off of someone decently intelligent, or just to talk philosophy, I’m not going to get great results out of the kind of model I can feasibly run at home. Even 30b parameters isn’t enough. That’s 75% of what I want out of an LLM.

Google models are now available as API on Google Cloud.

Faster, cheaper fine-tuning and training

If I could train a useful model, on my own data, in a reasonable time

I would want to have a CI-training pipeline to always have my models up to date

Yeah I remember in undergrad I was working on using transformation learning to train an object detector. Basically you only needed 100ish images to get the model to detect that new object really well.

I'm not sure what the analogous term is for a similar process on LLMs, but that will be huge when there is a service for it.

LLMs can do that without any examples (zero shot) or with one or a few demonstrations in the prompt, if you can describe the task in the limited context window.

If you want for example to train the model to learn to use a very large API, or access the knowledge in a whole book, it might need fine-tuning.

Could I just train a very small LLM with an English dictionary + Python + large API documentation + large Python code base?

Then do some chat fine tuning (like what HF did with StarCoder to get ChatCoder)

And get a lightweight LLM that knows the docs and code for the thing I need it for

After that, maybe incrementally fine tune the model as part of your CI/CD process

How similar were the object to other objects?

E.g., were you trying to distinguish an object vs nothing, a bicycle vs a fish, a bird vs a squirrel, or two different species of songbird at a feeder?

How much would the training requirements increase or decrease moving up or down that scale?

The PaLM 2 stuff released yesterday has fine tuning for their newest large models as a core feature.

Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685

Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...

How much would they cost aprox per training?

What would the quality of the model be, compared to what Karpathy uses on his video “let’s build gpt from scratch”?

In that video, he builds a decoder-only transformer model, that learns Shakespeare from 1MB of data and trains in 15min

These just fine-tune existing models, so the cost is single digit dollars. Whatever it costs in electricity to run a desktop GPU for a day or a few days.

Just use a machine learning method other than an LLM. If you're going to go to the effort of fine tuning and training, pay people to collect and label data like we used to do before Feb 2023.

If your main concern is question answering or summarization or code completion there are plenty of ways to do that now. If you really require the advanced emergent properties of LLMs, you'll have to work with a company that can afford to train a transformer on the Entire Internet.

We use Anthropic Instant in production and it has been much faster than Davinci/GPT4 for awhile. In terms of quality, Instant is at least as good as GPT3.5.

Privacy as well! I'd much rather run a model locally than in the cloud even if I lose out in performance in both speed and quality.

Reliability surely? They still haven't managed to make a model that says "I don't know" rather than bullshitting. That's by far the biggest unsolved problem.

Don't forget the ability to fine tune based on one's own data sources. For me, this is more important than any of the six reasons you mentioned.

Also if you allow users to receive vector representations of context and provide such representations as side information when querying LLMs.

For me I’d say speed trumps all else. It’s impossible to truly reach scale with the glacial response times you get from current API.

>speed trumps all else

Then use GPT-2

I actually do prefer 3.5-turbo over 4 for many tasks.

More languages?

Agreed. This is an important feature. Not all people speak English.

Reply on news.ycombinator.com