Who is building LLM Chatbots, and what issues are you running into? | Hacker News Reader

Who is building LLM Chatbots, and what issues are you running into?

16 pointspetervandijck11 comments

Heya, like probably everyone, we are building some internals LLM chatbots for customers of ours. I'd love to hear hands-on insights for what people are doing, why, what's working for them/not working, etc.

View on HN

Chain of thought is underutilized. It almost never makes sense to show the user the "bare" response of the LLM. It's so easy to have LLMs self-critique, think through user intent, etc. to drastically improve the final output

Improve probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience becomes just really slow.

You don't make multiple calls, you ask for a structured output and use some of the keys as the "chain links" in the chain of thought

Is there an example on how to do this ?

Contrived example, but say I'm summarizing an arbitrary document, I might ask for JSON with the following keys and instructions:

- purpose: one sentence on why would someone want to summarize this document

- effect: one sentence on how this affects the strategy we should use to summarize it vs a naive approach

- summary: one paragraph with the summary of the document based on purpose and effect

The LLM will change its summary based on the keys that come before summary in a meaningful way

Yeah you better be 99.9% perfect if you are going to take 60 seconds of plain spinner time to come back to me.

Which model are you using in production that's so slow you'd be waiting for 10 seconds, let alone 60?

For example, for us, we are building an LLM chatbot that pulls in the data of a technical book publisher. They have 20 years of technical books, and 20 years of videotaped conference talks.

Hard:

- We're using LangChain, which isn't always great

- The data pipeline was trickier than I had initially thought

- Indexing embeddings (in PostGres) is just hard (requires tons of ram)

But the hardest thing has been working on conversation quality. We've started to use LangSmith, which was a godsend for tracing and observability, and came out fairly recently. But it's not perfect and I wish there were better tools out there.

What do you find lacking in LangSmith?

I have been using it since the week it was in private beta, albeit a lot less recently, and thought it was good, though with some confusing UX and a handful of bugs.

Just generally a lot of abstractions that seem sometimes overwrought, and seem to hide details.

I would say that's langchain's ethos as a whole! haha

Reply on news.ycombinator.com