Ask HN: What is the right way to fuse my enterprise data with an LLM? | Hacker News Reader

Ask HN: What is the right way to fuse my enterprise data with an LLM?

4 pointsuptownfunk12 comments

Some stuff it into the prompt

Some fine tune llama

Some rag and then ask LLM to answer questions

What are the others?

What is the best?

I think an important baseline question is how much you care when the user of the LLM forces your model to divulge sensitive details or large chunks of that data.

Imagine that the LLM portion is running as client-side browser JavaScript application: Nothing in the training data or prompt is reliably-secret, and a determined user can get it to emit almost anything they like to whatever is downstream.

That is a valid constraint. For us we don’t really mind if it emits anything sensitive we are more concerned that it is correct.

Unless you can define and measure expected business value and risk, walk away and let someone else spend money and take the risk on a mostly hype- and FOMO-driven fad. If LLMs ever actually deliver real business value we'll all have time to integrate them.

the problem with your position is the absolutness of it to such a wide set of input parameters.

It's kinda like the folks saying Google is useless when there are clearly cases where it is useful.

In the same way LLMs have enormous value and utility, but equally there are places it is not as useful. Simply dismissing it as "mostly hype" is to miss the potential value.

For example, I've found it to be good at writing SQL queries. I can see how a human interface to a sql backend would make reading data much easier for my users.

Equally I have a well documented data schema, and lots of canonical docs for the product. I'd like to integrate this specialised knowledge with the existing SQL knowledge. In that context I'm interested in actual answers to the original question.

Is it even possible?

If you mean Google search, that had an immediate and obvious value from the day it went live. I didn’t call Google or anything else useless. I suggested defining and measuring expected benefits, costs, and risks. I would suggest that for any unproven technology, not just LLMs.

We can revisit the graveyard of over-hyped technologies if you want, and catalog the failures and wasted money. Ride the wave if you like. Show the success stories of LLMs in terms of business value or innovation. Don’t just repeat the press releases and anecdotes. You won’t find many successes, though maybe they will come in time.

> I’ve found [an LLM] good at writing SQL queries.

I don’t mean snark, but how do you know it writes good queries? I will grant that it can write queries that execute, but do those give correct results? Will they look good on a large production database where performance starts to matter?

When I played with LLMs writing SQL they worked well enough on toy schemas that resemble those found in tutorials. But faced with a real schema that requires joining tables and applying complex conditions they failed to produce usable SQL. However they did (with prodding) produce SQL that executed. It just gave the wrong results. That’s not a gain along any axis and could cost a lot of money for an organization blindly trusting such tools.

LLMs can write “good” code for some constrained definition of “good.” If you also want correct, testable, maintainable, secure, etc. they don’t perform well enough. Not just my opinion, we’ve had time to study the output of coding LLMs and it’s not looking good.

For me they provide a good jumping off point. I do find I still need to read it and tweak it. But it’s much easier than having to write the damn thing myself. I am not trying to be an engineer, I am trying to build a business and get an mvp off the ground. And if we get anywhere, then maybe we can have budget for real engineers.

Good luck. Several of my clients have tried the same thing, using an LLM to build an MVP or something simple. In every case they’ve had to hire programmers and either start over or spend more money getting the AI-generated code to work than it would have cost to hire it out.

Look out for security problems, because the bots certainly will.

Please report your experience.

If LLMs ever actually deliver real business value

Your typical LLM now days makes a lot of applications humans programmed thus far look silly.

There will definitely be a denial phase, but entire companies and products just look literally silly compared to an llm.

Can you point to or describe any of those? The silly apps and the LLM-based non-silly versions?

Almost unanimously CEOs/CTOs report no big gains in business value or measured productivity due to LLM adoption. The majority opinion seems to land at "We don't want to miss out, but we're not sure how to get any real benefit."

LLM (chatGPT) has mostly replaced 90% of Google search for me at this point. (Except browser, email, image, YouTube, google docs)

Until ChatGPT and related get inevitably enshittified with ads and other crap, gamed and polluted, as Google search has.

Prompt stuffing: Quick, dirty, and like cramming for an exam, works until it doesn’t.

Fine tuning: Great if you have static data and deep pockets for compute.

RAG inclusive Vector DB: The gold standard. Think of it as having your data whisper the answers to the LLM.

With AI Squared, you can keep your data fresh, dynamic, and external because nobody wants to retrain a model every time the boss changes their mind. :D

Reply on news.ycombinator.com