Hacker News Reader

Comment by oxfordmale | Hacker News Reader

I have also seen ChatGPT going berserk yesterday, but in a different way. I have successfully used ChatGPT to convert an ORM query into an actual SQL query for performance trouble shooting. It mostly worked until yesterday when it start outputting garbage table names that weren't even present in the code.

ChatGPT seemed to think the code is literature and was trying to write the sequel to it. The code style matches the original one so it took some head scratching to find out why those tables didn't exist.

View on HN

Okay, so I don’t really _get_ ChatGPT, but I’m particularly baffled by this usecase; why don’t you simply have your ORM tell you the query it is generating, rather than what a black box guesses it might be generating? Depends on the ORM, but generally you’ll just want to raise the log level.

Agree. Bizarre to use an LLM to do that. I wouldn’t be surprised if the LLM output wasn’t identical to the ORM-generated SQL.

I'd be very surprised if the LLM output is anything _like_ the ORM's, tbh, based on (at this point about a decade old; maybe things have improved) experience. ORMs cannot be trusted.

Maybe didn't have the environment set up locally and did initial investigative work?

So, if I wanted to investigate ORM output and didn't have an appropriate environment set up, I would simply set one up. If you just want to see SQL output this should be trivial; clone the repo, install any dependencies, modify an integration test. What I would not do is ask a machine noted for its confident incorrectness to imagine what the ORM's output might be.

Like, this is not doing investigative work. That’s not what ‘investigative’ means.

So imagine there is an urgent performance issue in production and you have a hunch that this SQL code may be the culprit. However before doing all of what you mentioned you want to verify it before following down a bad path. Maybe the environment setup could take few hours, maybe it is not a repo or codebase you are even familiar with. Typical in a large org. But if you know the SQL you will be able to run it raw to see if this causes it. Then maybe you can page the correct team to wake them up etc and fix it themselves.

But _you do not_ know the SQL. To be clear, ChatGPT will not be able to tell you what the ORM will generate. At best, it may tell you something that an ORM might plausibly generate.

(If it's a _production_ issue, then you should talk to whoever runs your databases and ask them to look at their diagnostics; most DBMs will have a slow query log, for a start. You could also enable logging for a sample of traffic. There are all sorts of approaches likely to be more productive than _guessing_.)

So I don't know what use-case exactly OP had, but all of your suggestions can potentially take hour or more and might depend on other people or systems you might not have access to.

While with GPT you can get an answer in 10 seconds, and then potentially try out the query in the database yourself to see if it works or not. If it worked for him so far, it must've worked accurately enough.

I would see this some sort of niche solution although OP seemed to indicate it's a recurrent thing they do.

I have used ChatGPT for thousands of things, which are on the scale of like this, although I would mostly use if it's potentially an ORM I don't know anything about in a language I don't have experience with, e.g. to see if does some sort of JOIN underneath or does an IN query.

If there was a performance issue to debug, then best case is that the query was problematic, and then when I run the GPTs generated query I will see that it was slow, so that's a signal to investigate it further.

The answer you get in 10 seconds is worthless, though, because you need to know what SQL the ORM is actually generating, not what it might reasonably generate.

You are thinking in a too binary way. It's about getting insights/signals. Life is full of uncertainties in everything. Nothing is for sure. You must incorporate probabilities in your decisions to be able to be as successful as you can be, instead of thinking either 100% or 0%. Nothing is 100%.

But it is a meaningless signal! It does not tell you anything new about your problem, it is not evidence!

I mean, I could consult my Tarot cards for insight on how to proceed with debugging the problem, that would not be useless. Same for Oblique Strategies. But in this case, I already know how to debug the problem, which is to change the logging settings on the ORM.

Well, based on my experience, it does really, really well with SQL or things like that. I've been using it basically for most complicated SQL queries which in the past I remember having to Google 5-15min, or even longer, browsing different approaches in stack overflow, and possibly just finding something that is not even an optimal solution.

But now it's so easy with GPT to get the queries exactly as my use-case needs them. And it's not just SQL queries, it's anything data querying related, like Google Sheets, Excel formulas or otherwise. There are so many niche use-cases there which it can handle so well.

And I use different SQL implementations like Postgres and MySQL and it's even able to decipher so well between the nuances of those. I could never reproduce productivity like that. Because there's many nuances between MySQL and Postgres in certain cases.

So I have quite good trust for it to understand SQL, and I can immediately verify that the SQL query works as I expect it to work, and I can intuitively also understand if it's wrong or not. But I actually haven't seen it be really wrong in terms of SQL, it's always been me putting in a bad prompt.

Previously when I had a more complicated query I used to remember a typical experience where

1. I tried to Google some examples others have done.

2. Found some answers/solutions, but they just had one bit missing what I needed, or some bit was a bit different and I couldn't extrapolate for my case.

3. I ended up doing many bad queries, bad logic, bad performing logic because I couldn't figure out a way how to solve it with SQL. I ended up making more queries and using more code.

This is for a performance issue and the Laravel code base is straightforward to map to SQL. It is to get a rough idea of the joins and the filters to see if there is potentially an index missing.

This is low hanging fruit. ChatGPT can do this, and also easy to verify it got it right.

I am a Data Engineer. I can spin up the service locally, but it would probably take me half a day to install the right versions etc and change the log level. The code is there, ChatGPT does a good enough job at it, or at least did. It is super easy to verify it did a decent job at it.

EVen if your ORM doesn't support this, you can always just turn on profiler on SQL and capture the actual query.

SSMS has SQL Server PRofiler, i'm sure others have similar.

No, it's very close to useless. This is exactly the kind of thing that experienced developers talk about when they warn that inexperienced developers using ChatGPT could easily be a disaster. It's the attempt to use a LLM as a crystal ball to retrieve any information they could possibly want - including things it literally couldn't know or good recommendations for which direction to take an architecture. I'm certain there will be people who do stuff exactly like this and will have 'unsolvable' performance issues because of it and massive amounts of useless work as ChatGPT loves suggesting rewrites to convert good code to certain OO patterns (which don't necessarily suit projects) as a response to being asked what it thinks a good solution to a minor issue might be.

I am a Data Engineer and it would take me ages to spin up the service with the right log level to grab the query. It is much easier to grab it from the codebase. I used to do this manually from time to time, now I use ChatGPT.

Well, wouldn't the sequel be version 2?

Reply on news.ycombinator.com