Hacker News Reader

Comment by martinald | Hacker News Reader

Yes I generally agree with this. Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite hard), which is not what I expected at all when the LLM boom started.

I think a lot of LLM usage is actually pretty 'simple' at the moment - think tagging, extracting data, etc. This doesn't require the very top state of the art models (though Llama3.1 is very close to that).

Hopefully OpenAI have some real jumps forward in the bag otherwise I am struggling to see how they can justify the valuations being floated around.

View on HN

IMO, it’s basically a generalist vs a specialist. LLMs are the generalist. They’re amazing at a wide variety of things, but unable to go deep.

The “wrappers” are the specialists. They’re doing the hard work of product discovery. This takes time, money, and a lot of trial and error to get it exactly right.

I think a big sticking point is chat isn't UX, unless of course 10 years from now we're all actually living in the movie Her...

Rebuilding existing apps with AI as a first class citizen seems like the name of the game right now

One killer feature is OpenAI’s vector database. I was surprised you can throw gigabytes of documents at it and chatgpt can see it all. It’s hard to simulate that via context window.

That’s not necessarily a moat, but OpenAI is still shipping important features. I wonder how hard it is for Claude et al to replicate.

They are outsourcing that to Qdrant, anyone can replicate it.

What do you mean, does it do anything other than the usual RAG?

Whatever it does, it’s indistinguishable from stuffing all the documents into the context window. Or at least I haven’t seen it fail yet.

Stuffing everything into the context window fails horribly every time i try it.

The model just doesn't seem to be able to really process the entire input.

I’m also confused what they’re talking about.. Does OpenAI have some feature I’m not aware of?

There's a way to upload documents that can be referred to in chat. I think they call it custom GPTs (which seems like a poor name): https://help.openai.com/en/articles/8554397-creating-a-gpt

Wait if this is what OP is referring to I am even more confused because the last time I built a custom GPT it was horribly slow and very inaccurate to look up uploaded information

also you certainly couldn't upload gigabytes of your own PDFs. Did anything change??

I’m referring to this specifically: https://platform.openai.com/docs/assistants/tools/file-searc...

When these companies are getting literally multiple billions of dollars of funding thrown at them, I can't think of a single feature that can be a moat. If it's truly a feature that leapfrogs competitors, there's just no way that someone like Anthropic spending $1B on engineering resources can't replicate the same exact feature that engineers elsewhere implemented.

If it’s simply an engineering feat I agree. Very, very, very often people tend to mix up science being done in technology spaces with engineering. Just because it’s being done in software and perhaps even without rigor doesn’t mean you’re not doing something novel and quite different at a fundamental perspective than someone else. Sometimes it could just be luck of the draw in an implementation approach that gets you there but that’s not always the case.

I’ve worked in scientific computing for awhile now and there are countless subtle decisions often done in the implementation phase that, from a theoretical perspective, aren’t definitively answered by the science working behind the scenes. There’s often a gap in knowledge and implementers either hit those gaps by trial and error, luck, or insight. That’s my opinion, at least. So I don’t think it’s always “just an implementation problem” some will claim, as if the science is well understood and solved. Perhaps it is, but from my experience that tends to not be the case.

This 100%. And LLMs and the applications around them (even something as simple seeming as ChatGPT) have more subtle decisions than any other type of software that I've ever seen. Everyone claims there isn't a moat, I bet there is.

Really interesting perspective

Yep I agree with that. I will say though that people, teams, and even entire companies (like what happened at Inflection) get poached everyday so maintaining a moat that way is tough. Also, even though it could happen in the future, is OpenAI's lead due to a moat of scientists with ideas so novel that no other AI company can compete? Certainly not because even though ChatGPT took the world by storm, numerous other companies built LLMs in a very short time span that now perform at very similar levels, both subjectively and based on benchmarks.

This practice supercedes modern science and was originally called trade secrets.

And these are generally "defeated" through spionage. 1 billion $ let's you do a lot, including completely legal things such as poaching someone that has a basic understanding of the secret sauce and then copying that from the abstract description

"The secret of success in business is knowing something no one else knows." — Aristotle Onassis

aka insider trading

Or being extraordinarily observant, creative and skilled in predicting trends.

Agreed, on another note I also struggle to see how no one could create a (better) CUDA implementation for e.g. $1B engineering budget

I purposefully used the word feature in my reply and I don't think CUDA is a feature. I see that as a massive ecosystem built on a proprietary platform. For that, it takes orders of magnitude more money and something else money can't buy.. time. Time to have the platform adopted by countless vendors and the ecosystem built up.

GPUs are scarce, expensive and in high demand. Devs are cheap compared to the expense of training a huge model. If Cerebras or some other hardware company develops a viable competitor to the H100, it doesn't matter if the ISA is 36-bit VLIW documented in Vedic Sanskrit, they'll have infinite demand.

AMD has a viable competitor to H100 but no one buys it because it doesn’t support CUDA.

It’s actually better than the H100 by a mile in some inference workloads, but in others falls behind.

that's surprising to me. Torch supports ROCm, why are startups wasting their money on H100s if they can get a better deal with AMD GPUs?

I'm sure OpenAI has some secret sauce that makes their RAG better than others, but LMStudio did ship the feature in 0.3.0 last week

https://lmstudio.ai/blog/lmstudio-v0.3.0

It’s not hard, we’re doing it with postgres and azure blob storage.

there are third party vector databases though

I used FAISS at work in the beginning of this year, and it was fantastic.

Company I worked for has massive private medical datasets and will never agree to non-local models or methods.

FAISS [0] is wonderful. Give it a try.

You can work with FAISS with LangChain, llamaindex and the like.

[0]: https://ai.meta.com/tools/faiss

Thank you! I’ve been wondering about something like this. Much appreciated.

The wrappers don't really have much of a moat either, OpenAI is just bad at front-end dev, so their velocity there is low.

Since most of their code is GPT-generated, their velocity should increase with better models.

Interactive use definitely wants the best model possible so that you have a higher chance of getting a correct and useful response.

It might be hard however to decisively convince people that one model is significantly better than another though, so branding/first mover/etc. probably plays a big role.

> UX etc is actually quite hard

It's analogous in the API space. OpenAI is demonstrating to be reasonably good at it for developers. They've been shipping significant features and improvements. Unless they lose pace, they have a moat, at least temporary.

The models are so general that it is much easier to swap out GPT-4 for Claude or Gemini than most other APIs like payments for example

Swaping models isn't the hard part. The prompts are.

Coming from a regulated industry that have well established rules about AI/ML in Production. In my opinion both of these model changes and prompt changes are governance issues that should be handled by your friendly CI/CD or Risk Management team(s).

I think the problem with valuations is the never ending financial history of hypes. The hype is correlated with the technology or industry but follows its own speculation hype. Even if the valuations and speculations are correct, they could be correct in years with ups and downs. If the dot com era is a good example it took half a decade or more to materialize.

Other aspects that are not well studied yet is how online advertisement will be affected if most people around the world end up using a single interface such as ChatGPT. How SEO/SEM/Ads will work in that world? Does someone look at what sites benefit being listed in ChatGPT (e.g. Wikipedia).

It's definitely what I expected. As soon as you get outside of the specific use case of chatbots, "AI stuff" is a feature, not a product.

> how they can justify the valuations being floated around

I'm sure they have at least a couple jumps in their bag.

Let's not forget inertia, as well. Migrating models is not a trivial project. To gain OpenAI customers, competitors need a considerable jump to justify, which I don't expect any of them to achieve before OpenAI itself delivers their jumps.

Products like Cursor literally have a dropdown that lets you switch between models seamlessly. From the surveys I’ve seen, most companies already use more than one model provider. The switching cost is fairly low.

The main problem is actually in making your prompts perform similarly enough with a new model. For sure, switching models is trivial, but it'll disrupt the quality of your service without significant effort in migrating your prompts.

I'm not saying this is an intrinsic moat, just that there's a barrier to change.

That same problem exists everytime your model served from a third-party is optimized/lobotomized. Only and offline model guarantees this wont happen.

This demonstrates a need for more robust engineering practices. Typically you would solve this by testing a validation set against a new model. Some tools like DSPy or Agenta are helping to encourage this. But creating good evaluators for generative responses isn’t easy, and in my experience people tend to punt.

Prompting is different than programming. Auto-testing your prompts is good practice and helps during a migration. But it doesn't change the fact that the prompts won't work the same in a different model.

It's like saying having Python tests eliminates bugs when rewriting a codebase into JavaScript.

To be fair though, you typically end up doing that process even when you remain within the OpenAI ecosystem, because they're putting out new (cheaper) models.

For example, a couple weeks ago I migrated a few game mechanics from GPT-4-Turbo to GPT-4o and it caused significant enough issues that I needed to revert them all until I could go back and retune all the prompts, which ended up taking 2-3 days. That's probably about as much time as it would have taken to tune prompts for any other model, especially since I'm using an API that standardizes format/structure across whatever model I select (as simple as a dropdown) and lets me just focus on prompt messages independent of which LLM they're going to.

With OpenAI's valuation and current interest rates, I don't really understand why they aren't visibly under pressure to show there's a lot of price tolerance, and until they attempt it, no one will see how quickly those model migrations get prioritized and completed.

Because people are already leaving for free options found in search engines on one end and on the other many are moving to other providers who offer multi-models or something different.

I think that they can get away with not exploring pricing as much because they’re trying to drive costs down instead and expand the market first. But I agree I’m surprised that prices aren’t being explored more.

Their new mini model is supposedly a lot smaller and cheaper than anything they’ve released before. If it compares to v3 which started this craze, then it’s clearly good enough to capture imagination and drive usage. People have posited that it’s cheap enough for per-click advertising to fund too. OpenAI claims that the lower cost model has doubled demand, which would be a good sign for supply-side growth.

> Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite hard), which is not what I expected at all when the LLM boom started.

In interviews lately, Sam Altman has been alluding to this more and more.

OpenAI has their own wrapper too and they have great UX.

Yes, details matter. The whole idea of creating AGI that is simultaneously a generalist seems more and more like wishful thinking. The reality is that to solve real problems, a large number of correct reasoning steps are required, along with the ability to make choices about which type of inference is useful at each step to avoid the explosion of complexity inherent in any brute-force approach. This suggests that we will have AI experts in different domains, perhaps superior to humans, but we will have thousands or even millions of narrow areas of expertise. To create something akin to an all-knowing superintelligent deity, we would need to combine thousands of experts, which would also consume unsustainable amounts of energy. I wouldn't bet on AGI in the coming years; it's just hype and distracts the discussion until big money finds a way to establish monopolies. However, if both UX and reasoning expertise require deep customization and specialization, we have a real chance to use AI to solve deep social problems rather than transforming society into a dystopia where humans are morally and intellectually surpassed, and those remaining are controlled by corporations that could at any moment be taken over by sociopaths.

Current models are very cheap. Like GPT-4 cost ~$40 million to train, right? Once we get 100 billion dollar models I suppose it'll be different.

Reply on news.ycombinator.com