Hacker News Reader

Comment by furyofantares | Hacker News Reader

Embeddings-based search is a nice improvement on search, but it's still search. Relative to ChatGPT answering on its training data, I find embeddings-based search to be severely lacking. The right comparison is to traditional search, where it becomes favorable.

It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to.

Maybe with more context I'll change my tune, but it's very much at the whim of the context retrieval finding everything you need to answer the query. That's easy for stuff that search is already good at, and so provides a better interface for search. But it's hard for stuff that search isn't good at, because, well: it's search.

View on HN

Agreed, GPT answering based on its own training data has been the best experience by far (aside from hallucinations) and comparing against that is difficult. Embeddings might not even be the long term solution. I think it's still early to really know for certain but models are already getting better at interpreting with less overall training data so there are bound to be some new ideas.

I’m sure many of you have tried generating epic conversations from history. With work and luck, I’ve read stuff way better than college.

But 90% of the time, it’s two barely distinct personalities chatting back and forth:

Me: Hey brian, what do you think of AI?

Brian: It’s great!

Me: I’m so glad we agree.

Brian: Great, this increases the training weight of Brian agreeing with Brian to a much more accurate level!

Me: Agree!

Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be).

But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline.

I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation.

Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.

I can understand why that framing would be attractive, but there is no real fundamental difference when considering JSONB/HSTORE in PostgreSQL, and now we have things like pgvector https://github.com/pgvector/pgvector to store and search over embeddings (including k-nn).

Yep. To be clear, that’s the exact approach I’ve been pursuing.

But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on.

To me that’s a fundamentally different approach to AI research at this moment. It seems to keep paying off in surprising ways.

> But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on.

Do you have any references to this? Seems really interesting if that can be a long term approach.

I’m considering the recent 64k token models as the most relevant examples.

More anecdotally, I couldn’t get anything to say more than a sentence locally at the beginning of 2023. I can get tons of useful results today.

Sure, this will plateau. But what if a model plateaus and it’s basically like a 10-year old?

But like, one of those 10-year-olds you hear about who gets his master’s degree at 13. At that point they’re just browsing the internet, reading books, and probably taking notes in a way that works for them.

Obviously this is wild speculation. Just laying out ideas that make me think in this direction.

OpenAI is rolling out access to a 32k context model. Mosaic ML just released a model trained on 65k inputs. https://www.mosaicml.com/blog/mpt-7b

My intuition is that it would work much better if the model could choose what to search for with something like langchain. The problem is that we don't know how to train such a system properly, we mainly do supervised finetuning on human examples of using the tools but this is fundamentally a reinforcement learning problem (RL is just hard).

That’s what I have been up to as well. I just think we’re so traumatized from the past 15 years of software engineering (kidding) that we’re stuck thinking “hey, let’s add memcached, that solved many problems like this before!”

Natural language is weird. It’s linear in the sense that you can’t take back what you said.

However, subjects and verbs and parentheticals and objects are like electron clouds. We only know statistically where it’s going to be. And damn, you’re reading this now, somehow, anyway.

Seems like the transformer model really is a new technology. But it’s just that we can actually run useful NLP neural networks now, which has opened the floodgate of innovation in various subfields.

(Personally bothers me because absolutely no one took it seriously when I was starting my career!

Lol. I had a meeting with my AI professor where they told me ANN’s were nonsense even if we had computers 1000x more powerful. Prune the search tree! That was not very long ago).

Vector search with move tos and move aways based on feedback is much more than attaching a database…

Encoder-decoder (attention) architectures still have a tough time with long-range dependencies, so even with longer context lengths, you'll still need a retrieval solution.

I agree that there's probably a better solution than pure embedding-based or mixed embedding/keyword search, but the "better" solution will still be based around semantics... aka embeddings.

> It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to.

I think the two could be paired up effectively. Context windows are getting bigger, but are still limited in the amount of information ChatGPT can sift through. This in turn limits the utility of current plugin based approaches.

Letting ChatGPT ask for relevant information, and sift through it based on its internal knowledge, seems valuable. If nothing else, it allows "learning" from recent development and effectively would augment its reasoning capability by having more information in working memory.

Is there any way to fine-tune GPT to make documentation a part of its training set, so you won't need embeddings? OpenAI lets you fine-tune GPT-3, but I don't know how well that works.

OpenAI doesn't let you fine-tune GPT-4 or GPT-3.5 yet (https://platform.openai.com/docs/guides/fine-tuning), but fine-tuning models on a set of documents is still an option but not really scalable if you want to keep feeding it more relevant information over time. I guess it could depend on the base model you are using and its size.

Reply on news.ycombinator.com