The Deep Research problem

I did a trial run with Deep Research this weekend to do a comparative analysis of the comp packages for Village Managers in suburbs around Chicagoland (it's election season, our VM's comp had become an issue).

I have a decent idea of where to look to find comp information for a given municipality. But there are a lot of Chicagoland suburbs and tracking documents down for all of them would have been a chore.

Deep Research was valuable. But it only did about 60% of the work (which, of course, it presented as if it was 100%). It found interesting sources I was unaware of, and assembled lots of easy-to-get public data that would have been annoying for me to collect that made spot-checking easier (for instance, basic stuff like the name of every suburban Village Manager). But I still had to spot check everything myself.

The premise of this post seems to be that material errors in Deep Research results negate the value of the product. I can't speak to how OpenAI is selling this; if the claim is "subscribe to Deep Research and it will generate reliable research reports for you", well, obviously, no. But as with most AI things, if you get paste the hype, it's plain to see the value it's actually generating.

Similar to what Derek Lowe found with a pharma example: https://www.science.org/content/blog-post/evaluation-deep-re...

> As with all LLM output, all of these things are presented in the same fluid, confident-sounding style: you have to know the material already to realize when your foot has gone through what was earlier solid flooring.

I think of it as the return to 10 blue links. It searches the web, finds stuff and summarizes it so I can decide which links to click. I ignore the narrative it constructs because it’s probably wrong. Which I forgive because it’s the hazmat suit for the internet I’ve always dreamed of.

It gets in the trenches, braves the cookie popups, email signups, inline ads and overly clever web design so I don’t have to. That’s enough to forgive its attempts to create research narratives. But, I hope we can figure out a way to train a heaping spoonful of humility into future models.

When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string together grammatically correct sentences. As a result, many people assumed that even GPT-3.5 was wildly more "intelligent" than it actually was.

I think Deep Research (and tools like it) offer an even stronger illustration of that same effect. Anything that can produce a well-formatted multiple page report with headings and citations surely must be of PhD-level intelligence, right?

(Clearly not.)

Research skills involve not just combining multiple pieces of data, but also being able to apply very subtle skills to determine whether a source is trustworthy, to cross-check numbers where their accuracy is important (and to determine when it's "important"), and to engage in some back and forth to determine which data actually applies to the research question being asked. In this sense, "deep research" is a misleading term, since the output is really more akin to a probabilistic "search" over the training data where the result may or may not be accurate and requires you to spot-check every fact. It is probably useful for surfacing new sources or making syntactic conjectures about how two pieces of data may fit together, but checking all of those sources for existence, let alone validity, still needs to be done by a person, and the output, as it stands in its polished form today, doesn't compel users to take sufficient responsibility for its factuality.

Deep Research is in its „ChatGPT 2.0“ phase. It will improve, dramatically. And to the naysayers: When OpenAI released its first models, many doubted that it will be good at coding. Now after two years look at Cursor, aider, and all the llms powering them, what you can do with a few prompts and iterations.

Deep research will dramatically improve as it’s a process that can be replicated and automated.

> Are you telling me that today’s model gets this table 85% right and the next version will get it 85.5 or 91% correct? That doesn’t help me. If there are mistakes in the table, it doesn’t matter how many there are - I can’t trust it. If, on the other hand, you think that these models will go to being 100% right, that would change everything, but that would also be a binary change in the nature of these systems, not a percentage change, and we don’t know if that’s even possible.

Of course, humans also make mistakes. There is a percentage, usually depending on the task but always below 100%, where the work is good enough to use, because that's how human labor works.

I urge anyone to do the following: take a subject you know really really well and then feed it into one of the deep research tools and check the results.

You might be amazed but most probably very shocked.

Everyone who has been working on RAG is aware of how important source control is. Simply directing your agent to fetch keyword matching documents will lead to inaccurate claims.

The reality is that for now it is not possible to leave the human out of research, so I think the best LLM can only help curate sources and synthesize them, but cannot reliably write sound conclusions.

Edit: this is something elicit.com recognized quite early. But even when I was using it, I was wishing I had more control over the space over which the tool was conducting search.

Not to take away from the main point of the article, which is true but:

It seems to be at intern level according to the author - not bad if you ask me, no?

Did he try to proceed as with an intern? ie. was it a dialogue? did he try to drop in this article into prompt and see what comes out?

For skeptics my best advise is – do your usual work and at the end drop in your whole work with prompt to find issues etc. – it will be net positive result, I promise.

And yes they do get better and it shouldn't get dismissed – the most fascinating part is precisely just that – not even their current state, but how fast they keep improving.

One part which always bothers me a bit with this type of arguments – why on earth are we assuming that human does it 100% correctly? Aren't humans also making similar mistakes?

IMHO there is some similarity with young geniuses – they get tons of stuff right and it's impressive however total, unexpected failures occur which feel weird – in my opinion it's a matter of focused training similar to how you'd teach a genius.

It's worth taking step back and recognizing in how many diverse contexts we're using (like now, today, not in 5 years) models like grok3 or claude3.7 – the goalpost seem to have moved to "beyond any human expert on all subjects".

I always wondered, if deep research has an X% chance of producing errors in it's report and you have to double check everything + visit every source or potentially correct it yourself. Then does it really save time in helping you get research done (outside of coding and marketing)? .

This article covers something early on that makes the question of “will models get to zero mistakes” pretty easy to answer: No.

Even if they do the math right and find the data you ask for and never make any “facts” up, the sources of the data themselves carry a lot of context and connotation about how the data is gathered and what weight you can put on it.

If anything, as LLMs become a more common way of ingesting the Internet, the sources of data themselves will start being SEOed to get chosen more often by the LLM purveyors. Add in paid sponsorship, and if anything, trust in the data from these sorts of Deep Research models will only get worse over time.

"Deep research" is super impressive, but so far is more "search the web and surf pages autonomously to aggregate relevant data"

It is in many ways a workaround to Google's SEO poisoning.

Doing very deep research requires a lot of context, cross-checking data, resourcefulness in sourcing and taste. Much of that context is industry specific and intuition plays a role in selecting different avenues to pursue and prioritisation. The error rates will go down but for the most difficult research it will be one tool among many rather than a replacement for the stack.

Watched recent Viva la dirt league videos on how trailers lie and do false promises. Now I see LLM as that marketing guy. Even if he knows everything, he can't help with lying. You can't trust anything he says no matter how authoritative he sounds, even if he is telling the truth you have know way of knowing.

These deep research things are a waste of time if you can't trust the output. Code you can run and verify. How do you verify this.

These days I'm feeling like GenAi is basically an accuracy rate of 95% maybe 96%. Great at boilerplate, great at stuff you want an intern to do or maybe to outsource... but it really struggles with the valuable stuff. The errors are almost always in the most inconvenient places and they are hard to see... So I agree with Ben Evans on this one, what is one to do? the further you lean on it the worse your skills and specializations get. It is invaluable for some kinds of work greatly speeding you up, but then some of the things you would have caught take you down a rabbit hole that waste so much time. The tradeoffs here aren't great.

I'll share my recipe for using these products on the off chance it helps someone.

1. Only do searches that result in easily verifiable results from non-AI sources.

2. Always perform the search in multiple products (Gemini 1.5 Deep Research, Gemini 2.0 Pro, ChatGPT o3-mini-high, Claude 3.7 w/ extended thinking, Perplexity)

With these two rules I have found the current round of LLMs useful for "researchy" queries. Collecting the results across tools and then throwing out the 65-75% slop results in genuinely useful information that would have taken me much longer to find.

Now the above could be seen as a harsh critique of these tools, as in the kiddie pool is great as long as you're wearing full hazmat gear, but I still derive regular and increasing value from them.

Indeed, the main drawback of the various Deep Research implementation is the quality of sources is determined by SEO, which is often sketchy. Often the user has insight on what the right sources are and they may even be off-line on your computer.

We built an alternative to do Deep Research (https://radpod.ai) on data you provide, instead of relying on Web results. We found this works a lot better in terms of quality of answers as the user can control the source quality.

Deep Research, as it currently stands, is a jack of all trades but a master of none. Could this problem be mitigated by building domain-specific profiles curated by experts? For example, specifying which sources to prioritize, what to avoid, and so on. You would select a profile, and Deep Research would operate within its specific constraints, supplemented by expert knowledge.

The problem with tools like deep research is that they imply good reasoning skills of the underlying technology. Artificial reasoning clearly exists, but is not refined enough to build these kind of technology on top of it. The reasoning is the fundamental of this system and all on top of it gets very unstable.

"I can say that these systems are amazing, but get things wrong all the time in ways that matter, and so the best uses cases so far are things where the error rate doesn’t matter or where it’s easy to see."

That's probably how we should all be using LLMs.

This is such embarrassing marketing from an organization (OpenAI) which presents itself to the world as a "research" entity.. They could have at least photoshopped the screenshot to make it look like it emitted correct information.

> they don’t really have products either, just text boxes - and APIs for other people to build products.

Isn't this a very valuable product in itself? Whatever happened to the phrase "When there is a gold rush, sell shovels"?

Two factors to consider: human performance and cost.

Plenty of humans regularly make similar mistakes to the one in the Deep Research marketing, with more overhead than an LLM.

The thig is, if you look at all the "Deep Research" benchmark scores, they never claim to be perfect. The problem was plain to see.

Yes the confidence tbh is getting a bit out of hand. I see the same thing with coding with our SAAS, once the problems get bigger I find myself more often than not starting to code the old way, even over "fixing ai’s code", because the issues are often too much.

Ithink more certainty communication could help. Especially when they talk about docs or 3rd party packages etc. Regularly even Sonnet 3-7 just invents stuff...

What a beautiful website

I, for one, have it in my prompt that GPT should end every message with a message about how sure it is about the answer, and a rating of "Extremely sure", "moderately sure", etc.

It works surprisingly well. It also provides its reasoning about the quality of the sources, etc. (This is using GPT-4o of course, as it's the only mature GPT with web access)

I highly recommend adding this to your default prompt.

One other existential question is Simpson's paradox, which I believe is exploited by politicians to support different policies from the same underlying data. I see this as a problem for government especially if we have liberal or conservative trained LLMs. We expect the computer to give us the correct answer, but when the underlying model is trained one way by RLHF or by systemic/weighted bias in its source documents -- Imagine training a libertarian AI on Cato papers -- you have could have highly confident pseudo-intellectual junk. Economists already deal with this problem daily since their field was heavily politicized. Law as well is another one.

I am currently in India in a big city doing yoga for the first time as a westerner.

I dont Google anything. Google maps, yeah. Google, no.

Everything I want to know is much better answered by ChatGpt Deep Research.

Ask a Question, Drink a Chai, Get a Great, Prioritised, structured Answer without spam or sifting through ad ridden SEO pages.

It is a game changer, and at one point the will get rid of the "drink a chai" wait and it will kill the Google we know now.

I used deep research with o1-pro to try to fact/sanity check a current events thing a friend was talking about, read the results and followed the links it provided to get further info, and ended up on the verge of going down a rabbit hole that now looks more like a leftist(!) conspiracy theory.