araghuvanshi

Wait this is actually pretty good! What interesting use cases have you seen so far?

Look man, Claude 3, GPT4 etc didn't work for my startup out of the box. I thought it would be helpful to tell others what I went through. Why hate on the truth?

Fair, valid point. I do admit that this is far from a perfect analysis. I do hope, though, that it helps people at least classify their problems into categories where they need to design around the flaw rather than just assuming that the thing “just works”. I appreciate the discussion though!

Why should I? If a person told you that they can multiply, divide, add and subtract, would you not also assume that they can at least count?

The point here is: the justifications from AI engineers for why counting vs math aren't the same task, while valid, are irrelevant because marketing never brings up the limitation in the first place. So any logical person who doesn't know a lot about AI will arrive at a logical, albeit practically incorrect conclusion.

Well LLMs are claimed to be good at math too, and yet they can't count. Same point with the long contexts. And our actual use case (insurance) does need it to do both.

My hope from this article is to help non-AI experts figure out when they need to design around a flaw versus believe what's marketed.

Well the same principle of false advertising re: context window sizes also applies to its inability to count, no? AI companies claim that their models can do math, so wouldn't a regular developer assume that they can also count?

And if I can't trust a so-called SOTA model to partially answer - say, recall each mention of the word "wizard" instead of just giving me the wrong answer - then why should I trust it to list out specific scenes? That's even harder to benchmark.

Direct quote from Anthropic's website: "Opus -Our most intelligent model, which can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks."

So you tell me: if a regular developer reads the above, how can they surmise that the model which can do higher-order math can't count?

I don't think that this is obvious at all. Yes, AI people who read papers on arxiv and know what "SOTA" stands for know it, but that is no longer the main user base of LLMs.

This is meant to be for the developer who doesn't fit the above profile and thinks a model that has a million token context window and "can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks" (direct quote from Anthropic's website), actually can do those things.

How much context? One sentence? Two? One paragraph? One page? It's very similar to the insurance policy problem - the text surrounding the information you're looking for, which could be surrounding it by one sentence or 10 pages, is just as important as the information itself

Totally agree there. And that's one of my points: you have to design around this flaw by doing things like what you proposed (or build an ontology like we did, which is also helpful). And the first step in this process is figuring out whether your task falls into a category like the ones I described.

The structured output element is really important too - subject for another post though!

I'm talking about the fact that they boast about their models having large context windows. And Anthropic says: "Opus - Our most intelligent model, which can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks." So if I were a non-AI expert, would I not infer that because it can do "higher order math tasks" it can also count?

That's true, but the problem of long context understanding (say, "summarize each of the situations where the word 'wizard' is mentioned") remains. And that gets much closer to the insurance policy thing.

Then why do the creators of this vacuum advertise the fact that it's really good at raking? And unlike your analogy, to actually figure out that it's bad at raking you have to read a bunch of academic papers?

Do most readers know that if you give a so-called million token context model that many tokens, it'll actually stop paying attention after the first ~30k tokens? And that if they were to try to use this product for anything serious, they would encounter hallucinations and incompleteness that could have material implications?

Not everything needs to be entertaining to be useful.

Please see my comment below, and the "Why should I care" section of the post. Yes you can count the number of times the word "wizard" is mentioned, but for tasks that aren't quite as cut-and-dry (say, listing out all of the core arguments of a 100-page legal case), you cannot just write a Python script.

The agentic approach falls apart because again, a self-querying mechanism or a multi-agent framework still needs to know where in the document to look for each subset of information. That's why I argue that you need an ontology. And at that point, agents are moot. A small 7b model with a simple prompt suffices, without any of the unreliability of agents. I suggest trying agents on an actually serious document, the problems are pretty evident. That said, I do hope that they get there one day because it will be cool.

Some counterarguments: 1. If an AI company promises that their LLM has a million token context window, but in practice it only pays attention to the first and last 30k tokens, and then hallucinates, that is a bad practice. And prompt construction does not help here - the issue is with the fundamentals of how LLMs actually work. Proof: https://arxiv.org/abs/2307.03172 2. Regarding writing the code snippet: as I described in my post, the main issue is that the model does not understand the relationships between information in the long document. So yes, it can write a script that counts the number of times the word "wizard" appears, but if I gave it a legal case of similar length, how would it write a script that extracts all of the core arguments that live across tens of pages?

Good share, thank you! Yeah I think Contextual AI has also been doing some interesting work in this area. Glossary is definitely interesting and an area we're looking into. Curious to see what work is being done with building knowledge graphs, that's another area where we've seen positive results.

The cost comparisons for the same model are interesting. I'm curious about why certain providers are a lot cheaper than others - for example, mistral-8-7b on OctoAI costs $0.2/1m tokens whereas it's $0.66 using mistral's own inference. My best guess is that one includes a cold start. Any thoughts there?

We use this product at pyq and I have to say, the speedup in our build times is amazing. Especially when HuggingFace is slow/backed up

That’s exactly right! We’re definitely more targeted towards non AI experts

Thank you! We've wondered the same. There are a few useful open-source models out there (doctr, TrOCR to name a couple) but our best guess is that it comes down to the relatively lower availability of good, public OCR datasets, especially for PDFs. A quick and dirty search on paperswithcode.com shows that there are 33 OCR datasets available, out of ~7800.

That said we've seen people have success with the ones I mentioned working out of the box, and I know of two folks who've fine-tuned a model to do what they need.

Of course, we're happy to help :) The pace of change, especially in AI right now, is pretty dizzying so I can certainly relate.

That's a super interesting use case. I'm curious to see if the model can achieve that out of the box or if it'll need to be fine tuned. Please keep us updated!

So I will admit that I don't know a lot about how elastic search works under the hood, but I can describe how ours does. Under the hood we use a large language model called BART which has 2 key benefits: 1. It has more "general knowledge", so it doesn't just search for semantic similarity between the text and tags you provide. This makes the tagging more accurate. 2. This model can be fine-tuned fairly easily for specific use-cases or on more recent datasets. For example, if you have your own taxonomy that you'd like to categorize certain text by - this is popular amongst advertisers who need to place ads on specific types of content, and e-commerce companies that need to categorize products on their site in an easily separable way - it's pretty straightforward to teach this model.

The synonyms question is a good one - so far, we've found that adding them here isn't really necessary as they'll all present similar scores.

Thank you! Totally hear you on the part about convincing people who know how to do it to use a tool. We've found that those folks respond well to the counterargument that they shouldn't have to do it, even if they know how to, unless it's mission critical for their application. And thank you for the advice and good wishes!