LlamaIndex: Unleash the power of LLMs over your data | Hacker News Reader

LlamaIndex: Unleash the power of LLMs over your data

llamaindex.ai

208 pointsdanboarder55 comments

If you're interested in the SQL component of this, we're building a product strictly focused on that at https://www.definite.app/. We let non-technical users ask questions of their SQL database. We do this by:

1. Pulling in your schema information and structuring it in a way LLM's can reason about it

2. Pulling in your prior query history against the database to understand how you actually use your data (e.g. what JOIN's are common, what tables are used most frequently, etc.)

3. Adding context from other tools you may be using (e.g. we can pull in metadata and tests from your dbt project)

We also have a Slackbot you can add to your #urgent-data-requests channel. If you @Definite in a thread, it'll parse out messages that can be converted to SQL tasks and return the answer from your database.

You could certainly build this yourself with (or without) LlamaIndex, but it's still quite a bit of work to set up.

Just to throw it out there, an alternative approach is to have the AI talk to a semantic layer. There are pros and cons to this and in practice you'd probably want an agent that can intelligently decide whether to call your semantic layer or go through some other process that may involve multiple queries, function calls, etc.

IMO a semantic layer makes it so easy to query that natural language querying is often not beneficial, even for business users. I take a hybrid approach of the semantic layer solving >95% of use cases and custom SQL as needed in backend processing.

https://github.com/totalhack/zillion

I like the concept of a semantic layer, but I haven't seen strong adoption of any of them (outside of LookML within Looker). Most of the companies I've talked to that are heavy dbt users don't use their semantic layer, but maybe that will change as the products mature!

We are seeing similar for Louie.ai . Data catalog yes, semantic no. We are also doing a lot of non-SQL like logs DBs (SIEMs) and graph DBs, which adds even more fun to that kind of question. My hypothesis here is that "semantic layer is to semantic web as LLM + vector search over activity is to Google/pagerank": easier to let the machines figure it out. We use knowledge graphs in some of our work, so definitely not disputing the value when people do put in the effort.

Wrt llmindex, the new tools like vector dbs & llmindex have been interesting. Fine-ish for our self-hosted, but most start getting tricky when a multi-tenant SaaS where we lower costs for teams and users by sharing infra yet need sharing boundaries.

I'm working on something that I believe has a similar to approach to what you're describing, particularly with the use of the semantic layer that AI talks to. Would love to connect sometime if you're open to trading notes? See profile for contact info, thx!

Do you have your source and model open? Its hard to use a AI service to give any type of access to prod DB without understanding the underlying code and models

Totally understand the concern. We're not open source yet, but working towards that as the open source LLMs improve.

We're SOC2 compliant which has been enough for many companies to get comfortable with security / data handling practices.

I’ve built a similar solution around BigQuery. I used temporary tables to let user iterate over the data instead of writing huge analytics queries.

I would open source it if I had decent experience with OSS.

If anyone with OSS maintainer experience wants to help with that - ping me.

That's really neat. I'll take you up on that. Will reach out to your HN profile email.

I built this in like one day for a pretty complex project, there’s no moat on this idea

Technically, one can say the same about any devops services out there, for instance. Makes more sense to opensource it.

Still, would you mind sharing the use case and the outcome? I have a couple of clients interested in similar solutions, but so far the potential outcome of this approach doesn't look promising.

With respect, it’s all in the quality of the execution. Care to share your version?

You should try building a bridge. Moats are a pitfall, literally.

thanks, giving up now.

Same is true for slack, since suc apparently does it in 5 lines.

Similar plug here: https://talktodata.ai/

We building a natural language chat interface for structured data. It can handle small csv files to large databases(500+ tables) seamlessly.

The name is misleading: this project is not based on LLaMA as released by Meta. It sends data to OpenAI by default.

It's not really. It was unfortunate timing. This used to be called GPT-Index, and then they changed names before Meta released their LLM. So their use predated it.

I feel sorry for the amazing team behind this great library. Changing names is hard.

Wow, that is some really bad luck.

17 feb 2023, https://twitter.com/llama_index/status/1626387226639888385

'Today, we’re kicking off a rebrand of @gpt_index to : LlamaIndex '

Meta announcing LLaMa 24 feb 2023 https://ai.facebook.com/blog/large-language-model-llama-meta...

third time's a charm

Was Llama a top-secret name at Meta? Could it have leaked?

It's a pretty easy pun to arrive at from the letters LLM, so I don't think we need a leak to explain the coincidence.

Oh, right. I always miss these puns.

Why is it hard? They should change it back.

s/Llama/GPT-/g

OpenAI has started sending out threatening letters to people who use GPT in their project name. If they don't want to be the test case for that (pending) trademark, I don't blame them.

Holy sh.t! I didn't know that.

Discussed here a few months back:

https://news.ycombinator.com/item?id=35973645

Thanks

Imma gonna go and grab gippity.com asap.

Still, they could have used a more general term other than "llama". They had it coming imo.

Sometimes developers come up with creative and interesting names for their products, but other times they fail miserably.

They being Meta or they being the developers of LlamaIndex? As noted in your sibling comment, LlamaIndex was public first.

the latter. Meta’s naming made sense cuz it was an acronym. LlamaIndex sounds like a joke name and not well thought out for such an ambitious product.

Meta's naming is clearly also a joke. It's a backronym, not an acronym— it's pretty obvious they started with Llama and came up with a cute acronym to justify it.

Large LAnguage Model Meta AI

Not saying they're not entitled to use the name too, but laying the blame on the developers of LlamaIndex when they had no idea that LLaMA was coming isn't fair to them.

Hey all! Jerry here (from LlamaIndex).

We love the feedback, and one main point especially seems to be around making the docs better: - Improve the organization to better expose both our basic and our advanced capabilities - Improve the documentation around customization (from LLM's to retrievers etc.) - Improve the clarity of our examples/notebooks.

Will have an update in a day or two :)

There once existed Google Desktop which was really useful.

Is this something similar, but with the added feature of being able to query the data with the help of a LLM?

Like: Find me all the text files which I've modified last month, there should be one containing a log snippet with a TODO I added to it.

I gave this a shot a while back and found plenty of examples but little documentation.

For instance, there is a tree structure for storing the embeddings and the library is able to construct it with a single line. However, I couldn’t find an clear explanation of how that tree is constructed and how to take advantage of it.

I wrote a short LangChain+LlamaIndex book that you can read free online: https://leanpub.com/langchain/read

Also, check out LlamaHub - lots of examples.

Yea, this was my experience too when I tried it out last week for my side project. It's easy to get started, but it's quite complex and disorganized and poorly documented. There are usually several ways to do things (which is by design, since it's meant to give you flexibility of either going with the default or customizing).

The main problem is the documentation is too disorganized, it's hard to figure out what even is the default and what are the configuration options, documentation is spread over a bunch of tutorials, reference pages, and blog posts by the founder. Sometimes the example code doesn't quite work because the library is changing so quickly.

We'll see if the community can figure out the best set of useful abstractions for this domain -- right now LlamaIndex is a mess and makes building things harder instead of easier and it's probably simpler to roll your own solution from scratch. However, the founders seem pretty smart, so hopefully with some time, they'll improve it and make it more usable.

If you’re doing legitimate retrieval rerank in the commercial enterprise setting, then I doubt this is a library that can support you beyond prototyping.

Retrieval involves complex integration (not just data connectors and open API wrappers), and meaningful rerank requires domain/context-specific trained models (that you can deploy performantly and cost effectively). If you’re doing these things, you’re well beyond the capability at platform scale vs what a python library provides

Is this a gripe with the verbiage used or is this a knock to the library itself?

I have been playing with langchain and llamaindex a bit.

I like the data loading abstraction and am very curious why you say it doesn't work? It uses ChatGPT for the reranking.

It’s just interesting to see the VC money pouring into these tools. My argument is serious integration/scale doesn’t involve a library like these (honestly prototyping doesn’t really need to either).

Id be more bullish on paradigm (platform/cloud level) shifts vs connectors, wrappers, and utility functions

Ymmv and to be fair I haven’t tried to scale these tools. I have worked on scaled platforms around embedding retrieval and rerank (including LLMs) so it’s just my take.

I would argue the level of abstraction it provides lowers the barrier to entry for most average programmers, myself included. LllamaIndex was my entrance to programatically utilizing LLMs. I have since moved to LangChain, with some documents loaded via LlamaIndex, but it has been a blast.

Right on— I should’ve and do recognize the utility of open source abstractions; esp with AI/ML.

Yes but this is just ETL - LlamaIndex and LangChain are re-inventing it - why use them when you have robust technology already?

1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.

2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.

Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN. Here’s our talk from Data+AI Summit:

Build a Generative AI App on Enterprise Data in 13 Minutes

https://www.youtube.com/watch?v=1exLfT-b-GM

Here’s a blog/demo

https://www.prophecy.io/blog/prophecy-generative-ai-platform...

Cool! Lets say I have thousands of documents that I want questions and answers for. Would your solution work for this? I wouldn’t know which documents to send with the prompts though as I want info on the aggregate (like trends and most mentioned phrases or words).

This sounds really cool. I predict it will consume a good chunk of my free time in the next week or so.

Is this an alternative/competitor to langchain? If so which one is easier to use?

It's an alternative, does a similar job, depends on/abstracts over langchain for some things. It's easier to use than langchain and you'll probably get moving much faster.

They've aimed to make a framework that starts concise and simple, has useful defaults, then lets you adjust or replace specific parts of the overall "answer questions based on a vectorized document collection" workflow as needed.

This works well overall, but some bits have kept me scratching my head for hours. Partly due to huge holes in the documentation when it comes to specifics ("plenty of examples but little documentation" another commenter wrote and I agree). Partly due to the frenetic release schedule, this project is highly active even by frothy LLM craze standards and interfaces change rapidly.

Overall recommend, LlamaIndex has helped me make good progress on my project

I just switched back to langchain when I realized that llamaindex doesn't support any kind of memory modules, at least not in a documented way...

That is what I am specifically going for with simpleaichat: https://news.ycombinator.com/item?id=36393782

It still needs a lot of work, but the goal is to decouple tools from LLM business logic (e.g. you would bring your on vector retrieval logic)

Complement

This stuff is like redis. Great for the infra folks

Langchain is like ruby on rails. Great for the AI app devs

There is overlap, where you might start and stay on langchain as a kitchen sink... But as you do full app dev, esp SaaS, the bottom starts falling out for more data infra centric tooling.

What's the difference vs embeddings on a vector database combined with GPT?

This is a layer of abstraction above that. So it handles ingestion of the data (possibly as embeddings), storing it (possibly in a vector database), and then querying of that data for LLM queries. You can swap in different embedding models, different storage databases, different retrieval methods, etc.

Not everything should persist, such as in a classic web architecture, analogous to redis vs postgres

Ex: For some of our use cases for more-text data sources and APIs we are helping folks index for Louie.ai, we do ingest-time embedding into persistent vector DBs for the raw, and are looking at llmindex and others for more ephemeral run-time caching of post-proccessed results. Ex: Pre-embed 1GB of text for fast search, but then only cache ad-hoc genAI task results on those, like classifications. We may still want to persist the embedded task results for other reasons, so not obvious.

If figuring out that kind of thing for use cases like analysts talking to DBs for cyber investigations, supply chain, emergencies, etc, we are actively hiring in backend engineering (remote): https://www.graphistry.com/careers, who build louie.ai

Reply on news.ycombinator.com