Comment by mentalgear | Hacker News Reader

Comment by mentalgear | Hacker News Reader

> The author of Bend has completely missed that this is the current standard in the field of formal verification, if they even know that this field exists at all. They have instead come up with this whole system requiring verbose specifications and even more verbose proofs. A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.

> This example matters beyond Bend, vibe-coding makes it makes it far too easy to implement a design that’s horribly broken or decades behind the current state of the art because you can immediately get a result without ever having to do any research. If you ask a LLM for a language where it’s possible to prove that a function is formally correct by building up a proof from basic principles then it will happily do so, it will never stop to suggest to you that computers can already build complex proofs without the need for a LLM and eliminate 99% of the work. It will never tell you that what you’re building already mostly exists as work that you can build on.

---

That's why all your LLM requests to build something substantial should start with "run prior work research first". Of course, at some point everything converges (if we share our outputs open-source) and then we may have solid standard patterns and libraries and do not need to waste trillions of tokens globally to rebuild the same minor, fundamental things, each one in their silent little silo.

IF we share, it will be of course to the monetary detriment of LLM providers who will have less income overall, and of course now they can't repackage anymore all our collective input, thoughts, human 'thinking traces' that they collect in their meta-data, as their new 'innovations' any more to inflate IPOs / stock prices.

Although, I have known using an LLM to do your prior research to itself be a trap.

The searches it runs, and the summaries it provides, are all incredibly sensitive to your choice of words. Words you chose from a state of minimal knowledge. So it’s like a particularly perverse version of the anchoring bias: information that could have led you to a better solution is often actively filtered out of the agent’s response precisely because it leads down a different path from your first idea.

In short, if you ask an agent what’s the best hammer for driving screws, it’s liable not to mention that screwdrivers exist.

Yes this is precisely why experts drive LLMs so much better than novices. It’s also why I don’t dismiss pure LLM output as slop and uninteresting; even if we both are using the same LLM, I probably can’t make it output what you can make it output.

Beautifully written. Hammers for driving screws + sycophancy are horrible together.

Is it?

As in, are you sure, and can you provide concrete examples?

I just tried it with chatGPT asking for whats the best hammer to drive in a nail, and it gave me a list of hammer suggestions :p . I drove it a bit further and specified wood screws and while it did point out a screw driver would be better it still kept recommending more specific hammers for driving the nails instead!

Depending on what I’m doing I’ll dedicate a few deep research sessions to building a framework. It will generate some grounding docs that go into the repo and get consumed as we go. Said docs establish terminology, widely known formulas and methods, etc.

And yes the output of these researchers are highly sensitive to prompting. Left to their own devices the LLM will often ship some very biased prompts to its deep research agents loaded with pre-conceived ideas rather than letting the agents uncover things themselves. Then all the agents do is confirm what the prompt told them to rather then “think independently”. (Very similar to open ended interview questions rather than asking yes/no questions)

It’s is far better to spend a session writing writing the research prompt itself.

All of this takes time and tokens of course…

Or maybe it should be a flag to indicate that there might be more to think about before handing the task of to an LLM.

> That's why all your LLM requests to build something substantial should start with "run prior work research first".

Yes, but I think there are incentives to not do this for many LLM providers. Doing prior-work research is slow (web searches aren't fast, LLMs are rate-limited or blocked from plenty of pages, etc.), and sometimes contradictory which annoys LLM users, many of whom like faster gratification cycles from the agent slot machine handle.

Also, writing a bunch of bespoke code instead of leveraging prior art makes a lot of users feel like they own something novel/big/important, and also poses a larger maintenance surface for the LLM to make future changes (which costs tokens).

I don't think there's, like, a conspiracy at LLM providers to set up system prompts/RAG/etc. to discourage research-and-use-prior-art-by-default approaches. Rather, OpenAI/Anthropic/Google/etc. are optimizing for real but sometimes misleading success metrics which often lead away from a research-first approach.

- That should nowadays be very obvious. You start any substantial work by doing research. LLMs are [perhaps surprisingly to some] good at that. They are also pretty good at doing the plumbing. That is a godsent in the hands of a good researcher and architect. And I don't have any qualms if tokens are "wasted" on research or on the precursor work. That doesn't replace reading good books on a subject but it DOES replace finding and filling knowledge gaps.

- But that shouldn't be confused with getting the LLMs to make the decisions. I believe that would quickly ruin a good design, unless the decisions are about truly inconsequential aspects, which are very rare in language and API design.

- I don't believe that (sharing) is to the detriment of LLM providers either. Not realistically. We would build faster and the questions / research directed at LLMs would be more sophisticated. Believe it or not, they can't cache questions as easily as websearch queries. If anything, I believe the more people learn to use LLMs effectively (rather than just to generate slop), the more their usage will be ingrained in daily life. Some of that will be redirected towards current LLM providers. But perhaps more of the economic share will increasingly go to hardware providers, as more and more people will be interested to run their own models.

> "run prior work research first".

As effective as “make no mistakes.”

It is trying to please you, and it always determines that the way to please you is to fulfill the original, core request. Any caveats or first steps will always be secondary to the ultimate goal of “this person wants to do X, so I will do X.”

The only first step I have found somewhat consistently useful, because as we know LLMs do not behave consistently, is when doing tech troubleshooting I will go “look at documentation for X before answering” so that it will search manuals and such. Helps avoid speculation. But even then, it’s still not full proof.

Sidebar: this is one of the core problems of LLM’s currently. You are basically arguing with them to get them to behave a certain way all the time and it’s not always clear if they’re doing what they’re being told to do. Then add the compounding layer that the longer the conversation goes on, the more likely it is to misunderstand or just ignore things as it descends into context-length-induced madness

> As effective as “make no mistakes.”

Those aren't comparable instructions. Providing useful, related context to improve outcomes is a basic best practice, and asking LLMs to do research first is an important source of that.

I’ll admit I was being overly tongue-in-cheek with that. You’re right it is more useful. But it also can’t be trusted to do that reliably, it’s primary goal is to give you the core thing you asked for - and often this is regardless of the parameters you put around it.

Fair enough, knowing when to be and not be prescriptive about implementation details is critical too!

Humans dont let LLMs raw dog internet web searches.

Reply on news.ycombinator.com