Ask questions of SQLite databases and CSV/JSON files in your terminal
simonwillison.net
simonwillison.net
I believe the attempts to have LLMs "write the code" or "reason" are wrong. What we need is to move to more formal languages to replace the ambiguity of natural language.
Another concern with tools like this is query performance and not accidentally overloading your database with some stupidly expensive query.
A tool that returns SQL from human language sounds great, but not one that runs the query unchecked. I speak four (human) languages, and sometimes I'll mistranslate something. Not to mention that this tool won't even bubble up the first two syntax errors that it generates - not very confidence inspiring.
The tool seems to show you the query that it executed, so people that do speak SQL can easily check it.
I do find it much easier to read/validate than to write though, which makes it an excellent application for LLM usage, in my experience.
Same applies for JavaScript and Python and Bash and AppleScript and Go and jq and dozens of other programming languages.
If you're going to use LLMs as a productivity boost you need to get good at quickly verifying that they've done the right thing. Effectively that means investing more in QA skills and habits, which is generally valuable anyway.
So why use an LLM when you can use sql queries directly and know the query is correct?
> If the SQL query fails to execute (due to a syntax error of some kind) it
> passes that error back to the model for corrections and retries up to three
> times before giving up.One minor footgun I've seen with that approach is that while the model is guaranteed to produce syntactically valid outputs, it can still "trap" the model into outputting something both wrong and low-probability if you design your schema badly in specific ways (contrived example: if you're doing sentiment analysis of reviews and you have the model pick from the enumeration ["very negative", "negative", "slightly negative", "neutral", "positive"], then the model might encounter a glowing review and write "very", intending to follow it up with " positive", but since " positive" isn't a valid continuation it ends up writing "very negative" instead).
Based on anecdotal experience (we should make a study) LLMs don't make more errors than human analysts IF given the same information.
The thing with humans is that you can build trust. I know exactly who to ask if I have a question about music, or medicine, or a myriad of other topics. I know those people will know the answers and be able to assess their level of confidence in them. If they don’t know, they can figure it out. If they are mistaken, they’ll come back and correct themselves without me having to do anything.
Comparing LLMs to random humans is the wrong methodology. Of course, Upton Sinclair had a point so I don’t expect to convince someone who is monetarily invested in having this broken assumption succeed.
There are plenty of use cases where this is a great starting place to be able to jump in and start to ask questions.
I have worked on a variety of acquisitions where I didn’t need to have 100% certainty, I just needed a starting place to make sense of some bizarre home grown financials and something like this would’ve been great to be able to quickly probe and come back with thoughtful questions.
Instead I spent my time either tediously figuring out a schema or waiting for someone in finance to come back with ad-hoc analysis that also had a bunch of errors.
You seem confused. My argument is not at all concerned with outcomes and is not binary in the slightest. My point is clearly spelled out:
> Comparing LLMs to random humans is the wrong methodology.
I’m not saying LLMs are never useful or anything of the sort. What I am saying is that defending LLMs on the basis of “some random undefined human would do the same” is a poor argument which shows a deep misunderstanding of human collaboration.
Which wouldn’t be so problematic if people didn’t just turn off their brains when interacting with them.
Either way, everything you’re suggesting are possibilities for the future, which may or may not pan out. The bad comparisons to humans are happening today.
Even at work, I can’t hog a single data engineer’s attention for hours on end without at least trying myself first.
At work, the situation should not be so different. If your manager cannot provide you with the means to maintain the database according to the business needs, you / your boss / your team / your business / your company have a problem and should better choose technologies that are manageable and/or for which you have enough resources.
> If you have created a hobby project database, you have the skills to learn how to query it - it is probably part of your hobby and fun.
Not at all in many cases. Many existing open source projects these days involve a database for better or worse, and I wouldn't enjoy porting their storage layer to something other than whatever database they already use.
I really don't need more engineering rabbit holes to go down in my life, neither at work nor in my free time. I do so by defining loose boundaries labeled "here be time sinks" and only go there if I'm really curious or it seems enjoyable, but not when I'm trying to get something else done.
LLMs have moved these boundaries somewhat, and I believe for the better.
The correct usage here is of course not to just run a natural language query and blindly trust the results. Sanity checking both the query and results is essential.
I still find LLMs incredibly useful in exposing new SQL functionality to me or refactoring larger existing queries to a very different approach (as SQL is unfortunately does not allow defining query components in a modular way that would let me avoid that).
Huh, that's really interesting. I've found LLMs (mostly Claude) to be pretty bad at writing SQL (they love cross joins for some reason), so it's interesting that others are getting good results. What models are you using, do you do any particular prompt engineering or anything different?
Another pattern I've found pretty useful: "This isn't working. Build a small sample dataset to exercise your query, and I'll paste the results back to you so we can both see what might be going wrong."
Basically, I treat it as an intern, not as an oracle of truth.
But soon... Won't be able to do anything without interacting with an LLM!
If I'm not toying around I'm not going to send information about it to some corporation in a jurisdiction without decent data protection laws.
Dumping some stuff into Ollama is close to trivial. It's also rather slow, except on quite expensive hardware.
Full list of plugins here: https://llm.datasette.io/en/stable/plugins/directory.html
So now there are tools like this to help automate this simple work, but I wonder why the LLM can’t already handle it… “create a tool to automate querying a SQLite database using an LLM”
I'd say it's just the opposite:
I'm telling the LLM what outcome I want (and, currently, how I want it to go about doing that, since it often can't figure it out itself), and it handles the low level details of getting the SQL syntax right, tapering over minor details in available utility functions and syntactic sugar across databases.
Would be cool to be able to hook this up with ollama or something local.
Ollama is supported in LLM via this plugin: https://github.com/simonw/llm-ollama
I just assumed LLM was a en var manager for openai stuff. But makes sense.
Our bet with a similar tool (getdot.ai) is that the shift goes into existing communication tools (Slack, etc.), but CLI is an interface I personally enjoy.