Hacker News Reader

Comment by sillysaurusx | Hacker News Reader

One killer feature is OpenAI’s vector database. I was surprised you can throw gigabytes of documents at it and chatgpt can see it all. It’s hard to simulate that via context window.

That’s not necessarily a moat, but OpenAI is still shipping important features. I wonder how hard it is for Claude et al to replicate.

View on HN

When these companies are getting literally multiple billions of dollars of funding thrown at them, I can't think of a single feature that can be a moat. If it's truly a feature that leapfrogs competitors, there's just no way that someone like Anthropic spending $1B on engineering resources can't replicate the same exact feature that engineers elsewhere implemented.

If it’s simply an engineering feat I agree. Very, very, very often people tend to mix up science being done in technology spaces with engineering. Just because it’s being done in software and perhaps even without rigor doesn’t mean you’re not doing something novel and quite different at a fundamental perspective than someone else. Sometimes it could just be luck of the draw in an implementation approach that gets you there but that’s not always the case.

I’ve worked in scientific computing for awhile now and there are countless subtle decisions often done in the implementation phase that, from a theoretical perspective, aren’t definitively answered by the science working behind the scenes. There’s often a gap in knowledge and implementers either hit those gaps by trial and error, luck, or insight. That’s my opinion, at least. So I don’t think it’s always “just an implementation problem” some will claim, as if the science is well understood and solved. Perhaps it is, but from my experience that tends to not be the case.

Yep I agree with that. I will say though that people, teams, and even entire companies (like what happened at Inflection) get poached everyday so maintaining a moat that way is tough. Also, even though it could happen in the future, is OpenAI's lead due to a moat of scientists with ideas so novel that no other AI company can compete? Certainly not because even though ChatGPT took the world by storm, numerous other companies built LLMs in a very short time span that now perform at very similar levels, both subjectively and based on benchmarks.

This practice supercedes modern science and was originally called trade secrets.

And these are generally "defeated" through spionage. 1 billion $ let's you do a lot, including completely legal things such as poaching someone that has a basic understanding of the secret sauce and then copying that from the abstract description

"The secret of success in business is knowing something no one else knows." — Aristotle Onassis

aka insider trading

Or being extraordinarily observant, creative and skilled in predicting trends.

This 100%. And LLMs and the applications around them (even something as simple seeming as ChatGPT) have more subtle decisions than any other type of software that I've ever seen. Everyone claims there isn't a moat, I bet there is.

Really interesting perspective

Agreed, on another note I also struggle to see how no one could create a (better) CUDA implementation for e.g. $1B engineering budget

I purposefully used the word feature in my reply and I don't think CUDA is a feature. I see that as a massive ecosystem built on a proprietary platform. For that, it takes orders of magnitude more money and something else money can't buy.. time. Time to have the platform adopted by countless vendors and the ecosystem built up.

GPUs are scarce, expensive and in high demand. Devs are cheap compared to the expense of training a huge model. If Cerebras or some other hardware company develops a viable competitor to the H100, it doesn't matter if the ISA is 36-bit VLIW documented in Vedic Sanskrit, they'll have infinite demand.

AMD has a viable competitor to H100 but no one buys it because it doesn’t support CUDA.

that's surprising to me. Torch supports ROCm, why are startups wasting their money on H100s if they can get a better deal with AMD GPUs?

It’s actually better than the H100 by a mile in some inference workloads, but in others falls behind.

I used FAISS at work in the beginning of this year, and it was fantastic.

Company I worked for has massive private medical datasets and will never agree to non-local models or methods.

FAISS [0] is wonderful. Give it a try.

You can work with FAISS with LangChain, llamaindex and the like.

[0]: https://ai.meta.com/tools/faiss

Thank you! I’ve been wondering about something like this. Much appreciated.

They are outsourcing that to Qdrant, anyone can replicate it.

I'm sure OpenAI has some secret sauce that makes their RAG better than others, but LMStudio did ship the feature in 0.3.0 last week

https://lmstudio.ai/blog/lmstudio-v0.3.0

It’s not hard, we’re doing it with postgres and azure blob storage.

What do you mean, does it do anything other than the usual RAG?

Whatever it does, it’s indistinguishable from stuffing all the documents into the context window. Or at least I haven’t seen it fail yet.

Stuffing everything into the context window fails horribly every time i try it.

The model just doesn't seem to be able to really process the entire input.

I’m also confused what they’re talking about.. Does OpenAI have some feature I’m not aware of?

There's a way to upload documents that can be referred to in chat. I think they call it custom GPTs (which seems like a poor name): https://help.openai.com/en/articles/8554397-creating-a-gpt

Wait if this is what OP is referring to I am even more confused because the last time I built a custom GPT it was horribly slow and very inaccurate to look up uploaded information

also you certainly couldn't upload gigabytes of your own PDFs. Did anything change??

I’m referring to this specifically: https://platform.openai.com/docs/assistants/tools/file-searc...

there are third party vector databases though

Reply on news.ycombinator.com