Average is all you need | Hacker News Reader

Average is all you need

rawquery.dev

113 pointsAlexC04102 comments

> You did not write a single line of SQL. You did not set up an attribution model. You asked a question, in English, and got a table.

But nobody bothered to check if it was correct. It might seem correct, but I've been burned by queries exactly like these many, many times. What can often happen is that you end up with multiplied rows, and the answer isn't "let's just add a DISTINCT somewhere".

The answer is to look at the base table and the joins. You're joining customers to two (implied) one-to-many tables, charges and email_events. If there are multiple charges rows per customer, or an email can match multiple email_events rows, it can lead to a Cartesian multiplication of the rows since any combination of matches from the base table to the joined tables will be included.

If that's the case, the transactions and revenue values are likely to be inflated, and therefore the pretty pictures you passed along to your boss are wrong.

Further reading, and a terrific resource:

https://kb.databasedesignbook.com/posts/sql-joins/#understan...

Ok but… nobody said you didn’t had to check either(?).

How do you check if you don't have any other view into the data but SQL and you don't know SQL?

Same way you do today; you trust whoever wrote the query.

I do not sell a wrapper on top of some LLM; you can absolutely write your SQL directly. There is an engine, there are iceberg tables. You can just live your best life doing your own SQL by hand.

Now if you couldnt do it before and you have a sensible understanding, you can likely do a bit more with the CLI tooling. And if you know a lot more, you can still do that. The queries are not hidden, or abstracted, If you need them they will be saved - transparently in SQL.

So I dont know what is the answer to the question "how do people do things they don't know how to do" ?

> So I dont know what is the answer to the question "how do people do things they don't know how to do" ?

The statue quo had been to learn SQL or ask a human you trust to check their own work, which hopefully you can reuse.

Now it's ask AIs that are intentionally a bit random, and less likely to (or incapable of) check(ing) their work. Perhaps without seeing the SQL at all, requiring to trust it for every interaction. And in a culture that moves so fast that there is no checking by any(one|thing).

If you think a language model can't check their work, then you are using the tools wrong. Plain and simple.

Modern models are quite capable at surfacing and validating their assumptions and checking correctness of solutions.

Oversight helps you build confidence in the solutions. Is it perfect, no.. but way better then most engineers I also ask to check things.

No they don't. To be able to "check one's work", implies that they can be held accountable, that they can tell apart right from wrong, when in reality they're merely text predictors.

If you think an LLMs can check their work, then you are doing a terrible job at writing software. Plain and simple.

They even go as far as "cheating", so tests fail, writing incorrect tests, or straight out leaking code (lol) like the latest Claude Code blunder. Is this the tool the original comment "is using wrong, plain and simple"? Or do you have access to some other model that works in a wildly different way than generating text predictions?

you can have it write test cases though.

in this case to make a local copy of the db, fill it with a set of records with an expected output of the query, then check to see of the query produces what you want.

you could then have it make queries that check the various assumptions that went into that artificial set of data. if it can find the assumptions broken, add records like that to the test set.

same old agentic programming techniques as ever. use your engineering skill to set up feedback loops. stuff that was painful to do as an engineer for checking your work is now straightforward

The point is that you have verify it yourself. Like you wrote: "check to see of the query produces what you want"

Otherwise the LLM can just write tests against whatever it wrote and not what is expected. This happens often with the top models too.

Someone needs to check the tests work, review they cover edge cases etc.

Feedback loops require a deterministic metric for success. You are doing the equivalent of using a slot machine to decide whether something is right or wrong.

Sql takes at most an afternoon to learn enough of to navigate a database with

Yes. The correct answer is to ask an SQL expert to write the query. An LLM will make the expert much faster at writing the script, maybe.

It doesn't.

I can write that script faster than I can write the text asking the AI to write the script as SQL is concise and my IDE has auto-complete.

In fact auto-complete in VS code with copilot works surprisingly well. Let’s say there are patterns in how classes in a framework are constructed, and I already added changes to the schema, it will often emit the correct implementation code when going there. That is really speeding me up even if I know the code base very well.

SQL is not hard enough to require an LLM to think about for you

I will never understand Engineers who struggle with SQL lookups. The vast majority of queries are extremely basic set theory

The harder part is understanding the nature of the data you're working with. There's always some catch ("oh that field `foo` was never backfilled, so for queries before 2020 you have to recompute it by joining with legacyBar instead")

> SQL is not hard enough to require an LLM to think about for you

As someone who's seen queries that are hundreds of lines long, involve a bunch of CTEs, nested SELECTs as well, upwards of a dozen joined tables with OTLT and EAV patterns all over the place (especially the kind of polymorphic links where you get "type" not "table_name" so you also need to look at the app code to understand it), I'd say that SQL can be too hard for people to reason about well.

Bonus points for having to manually keep like 5 Oracle package contents in your working memory cause that's where the other devs on the 10 year old project stored some of the logic, while the remainder is sort-of-dynamic codegen in the app.

Same as with most app code, it shouldn't be like that, but you sometimes get stuff that is really badly developed and the cognitive load (both to inherent and accidental complexity) will increase until people will just miss things and not have the full picture.

I think the author should be introduced to (or reminded of) the tale of the average from the US Air Force [1]. Social reality is high-dimensional and the "normal" thing is actually to be average in some dimensions, but strongly non-average in many others. So a "perfectly average" family would paradoxically be an outlier themselves.

I think this is important, because if his hypothesis is right, then LLMs behave differently here: They really are average in all dimensions. They are the pilots the Air Force thought they had before Daniels made the study.

So if he is right, we'd be changing from a mostly-non-average to a mostly-average society, which would really be a massive change - and probably not a good one IMO.

[1] https://noblestatman.com/uploads/6/6/7/3/66731677/cockpit.fl...

Wow incredibly interesting read, got me thinking about design principles and the "average user"

Interesting link! What was the solution? How did they end up redesigning the cockpit?

From what I understood, it was the beginning of user-adjustable design. Instead of mounting the seat in a fixed height with fixed distance to the pedals, etc, they designed the cockpit so the pilot could adjust everything himself. Basically, what is standard in every car today.

In high dimensional spaces, almost all of the volume of a sphere is in a thin shell close to the surface.

If average is all we need, then anyone can do it. What value do I add? How does an employee differentiate themselves?

Why didn’t the boss ask the AI for the charts to begin with?

Everyone’s income is going to be below average, because they got fired.

Not everyone can be average. Half of people will be below average.

I might not agree with the point, but I can see that idea that many things just need to be "good enough" (which we might define as "average") and we save our real expertise for the things that really matter.

> Half of people will be below average.

s/average/median

I don’t believe this is a meaningful distinction when we’re not going to agree on how to judge performance of software engineers. If this were solely about income, it might be an important distinction.

The article assumes a normal distribution, making the distinction moot

But it is useful to question whether that is true in all cases. The cases that aren't normal-distributed might be exactly the cases where it pays off to be neither average or median

It is almost never true. If you filter people you're going to get a Pareto distribution.

there is a major shortcoming in this assumption; everything we've seen related to the internet and technology in general suggests there is rarely a normal distribution. I think it's way more valuable ato frame the questions as a long tail (pareto) distribution and a "good enough" cut-off point.

Median is a type of average.

Though usually "average" implies arithmetic mean.

Reducing the amount of time I spend on the average code has meant I'm spending more time adding my above-average contributions to the code base. Amdahl's law, basically. Reducing the amount of time spent on one task means the percentage of time spent on the others increases.

How stable that is on the long term, I don't know any more than the next guy, but it is where I'm contributing now.

The power saw makes average cuts, it didn't disemploy carpenters, we just made better homes.

We make more homes, but I would say the construction of the average home is worse after the invention of the power saw than before it.

Good gosh no.

That's like saying 'cars were better made in the 1950's because they used tons of steel'. Like they were 'heavier and more robust' - but that doesn't mean better.

Foundations are way better, more robust, especially weatherized. Windows today are like magic compared to windows 100 years ago.

What we do more poorly now is we don't use wood everywhere, aka doors, and certain kinds of workmanship are not there - like winding staircases, mouldings - but you can easily have that if you want to pay for it. That's a choice.

AI is power and leverage, it will make better things as long as it's directed by skilled operators.

Yes, houses got better because materials got better. Windows are better. But the construction of the houses is worse.

The precision of how the wood or material meets is worse (when cut at the site). There is a huge amount of sloppy work in modern construction.

I'm interested in how one would prove that one way or another.

It seems to me that in the past there probably was lots of shoddy workmanship and just no-one paid attention to it.

But I have no proof of that.

Fortunately, there are millions of buildings that remain standing as evidence of what was done in the past. So at least there's that!

Massive selection bias - only the good quality ones remain standing, the low quality ones are not.

You have to get a representative sample, that's the tricky part.

So there's that!

Buildings don't get taken down because 'they were built poorly', it's cheaper to rebuild than refurbish.

And we can accommodate for 'selection bias'.

We have all of the historical evidence we could ever want for 'how things were built', basically 'infinity examples'.

I think some things were more robust, particularly some of the old framing, like in Europe, with non load-bearing walls etc. Those will stand for 1K years, but arguably unnecessary.

this is not true in my experience. prefab kits of all sizes (from sheds to houses to barns, like were once possible to order from a Sears catalog) have worse tolerances than a carpenter working on site. you can measure 3 times and cut perfectly, and still end up with a few mm gap (or sometimes worse) after tiny errors accumulate as you assemble piece after piece. it _requires_ measuring as you go and cutting on site to handle this small amount of drift and to really produce something of high quality. it doesn't come in a box

Correct about large scale kits. I had meant to head off the fact that preassembled pieces like windows have improved a lot, things that used to be assembled on-site but are now delivered as a unit or small kit.

No, it doesn't. The power saw makes perfect cuts. That's why carpenters use them.

That isn't a sane starting point; if a corporation's strategy is to only hire above average employees they're going to fail. Enron springs to mind. Corporations generally take average people and give them a reasonably well defined scopes of simple work to complete that adds value. The bigger the corporation the more difficulty they have handling even the standard deviation above average differently to the one below; almost everyone just becomes a human resource to be swapped around based on social factors.

The people who need to be above average and exceptionally are senior management and maybe a few bright sparks in middle management. Most of the value-add happens there that builds social machines that then do the work.

> If average is all we need, then anyone can do it.

Pretty much, yes. That is why the range of salaries on offer is pretty compressed compared to the range of returns capitalists get.

> The people who need to be above average and exceptionally are senior management and maybe a few bright sparks in middle management. Most of the value-add happens there that builds social machines that then do the work.

That is the dream. Upper management can get software made without talent.

But is seems to be the greatest ideas in the last 30 years didn’t start in board rooms. They started with a couple coders creating a new idea.

No boardroom could have invented Google. It was so fundamentally different than what other search engines were doing.

We have this myth that upper management is so important. It is as the business grows in size, they are excellent for coordination. But ideas come from people closer to the problems.

> No boardroom could have invented Google. It was so fundamentally different than what other search engines were doing.

You might want to try a different example, that one rather undermines the point you're trying to make. PageRank [0] was developed by Page & Brin as original research/based on the work of other people who weren't employees.

[0] https://en.wikipedia.org/wiki/PageRank#History

The average of quality isn't always available in all people.

For that matter, how does a business differentiate themselves, if people can write their own software? While we're busy trying to replace our employees with AI, our customers are trying to replace our products with AI.

support quality is an obvious one,

the other being how well the ai can use it, and how ai SEO-ed you are.

vercel and next.js for isntance are absolutely loved by claude

At any tech company with leveling guidelines that I have seen, promotions above mid level have never been based on “I codez real gud”. It’s always been based on scope, impact and dealing with ambiguity. It’s stated differently in different companies.

No one has ever differentiated themselves based on how good of a ticket taker they are. Coding especially on the enterprise dev side where most developers work has been being commoditized since 2016 at least and compensation has stagnated since then and hasn’t come near keeping up with inflation.

In 2016, a good solid full stack, mobile or web developer working in the enterprise could make $135K working in a second tier city. That’s $185K inflation adjusted today. Those same companies aren’t paying $185K for the same position.

My one anecdote is that the same company I worked for back then making $125K and some of my coworkers were making $135K just posted a position on LinkedIn with the same requirements (SQL Server + C#) offering $145K fully remote.

> At any tech company with leveling guidelines that I have seen, promotions above mid level have never been based on “I codez real gud”. It’s always been based on scope, impact and dealing with ambiguity. It’s stated differently in different companies.

I 100% agree here.

AI has been a huge boon for me personally, because I stopped spending most of my writing code years ago. I was reviewing code, writing procedures, handling incidents, and generally just looking for pain points across the entire company and solving them before they became critical.

Those skills have transferred directly to working with AI.

Tbh I dont really agree with your statements. Especially with working with data, intention is key. By using an llm, by definition, you are loosing intention. And Thai puts you in a position where you have to 1) think of exactly what you look for. 2) able to understand what the llm generated.

You might say it "still less work" and that's true, perhaps, only for the first few times. After a while you _learn_ how to do it, and understand how to _think_ with the language of your data. With LLMs, you never get this benefit, and also loose your ability to judge the LLM's output properly.

But again, that might be enough on your case, or, you simply don't _know_.

It seems like person receiving the report is already in that position. Should everyone make reports themselves?

Let's say you start with a report someone else wrote. It seems like you still need to read it and understand what it's telling you. Sometimes plotting all the points helps, or drilling down and looking at the raw data.

The real risk is not that "LLM SQL" is wrong, it is in fact that a man who asked for it can't recognize when it is wrong. Writing a query and knowing when JSON silently doubled your rows are different skills and LLMs are able to give you only one of them (I'm not sure if they really can)

Average is only a tombstone of someone having failed to do better. And settling for average means pulling down.

When it comes to bs dashboard where "average is all you need", maybe the "better than average" result would be asking yourself if it's even worth doing in the first place?

Average is all you need, if your needs are average.

But, you see, our needs are above average because we target above average exists so we only hire from the top 1% of software engineers, blah, blah, yadda, yadda, etc.

The Business simply cannot admit that it’s really doing nothing above average. If they did, investment dries up.

That is correct. And if you need more you can get it as well.

I liken it to the Ikeaficiation of furniture. To a great majority, such as my college self, it was preferable and desirable. As I've made more money, I've wanted something better.

There's a market for both, but the furniture slop of Ikea is dominant.

how do you know those queries are actually correct without domain knowledge?

Do you know enough about JOINs and how they work to be able to break those big queries down and figure out whether they are doing exactly what you're asking for in English?

You don't, and if businesses start using vibed reports for regulated reporting then I guess we'll see soon what the courts say about that

Litigation aside for a moment - I'm not sure vide-coded reporting could be much worse than what I've seen from early-career analysts in past companies.

You don’t. But you can still check ? ¯\_(ツ)_/¯

I always find it a bit weird to see posts on the front page where all the comments disagree with the central premise of the article. In this case the post is an ad advocating for executing code you didn't write and handing the results to your manager.

It makes me wonder if Hacker News has a silent majority of people who would actually use AI in this way without wanting to admit it, and a vocal minority of people who wouldn't.

I'll admit that there are definitely times where I decide it's fine to roll with it blind. It's not often, not for critical paths, and definitely not where I don't have a good understanding of the blast radius if it fails spectacularly - but you'd be surprised how often it's easier and faster to fix it if it breaks than it would be to make sure it's not broken.

This seems like a nice context to mention Sturgeon's law:

> ninety percent of everything is crud

https://en.wikipedia.org/wiki/Sturgeon%27s_law

> But this is a pain, first because, if you do anything that is not selling a product online that people can buy right when they click a button, it is a drag to create those attribution models effectively: is it last click, first click, weighted attribution... who knows. Nobody knows. Everybody gives up and just adds it to a dashboard and pretends it makes sense.

Yes, thinking about your data and how to check it is so annoying. Much better to do something average, see if the result puts you in a good light, and share that insight into your company's working with ~~everyone on the internet~~ your boss.

Rarely have I seen "we help you create meaningless slop more easily" advertised so explicitly. Or is this also average?

Why average? I've always taken pride in my work and developed things that went beyond the expectations of the management and of the final users. Now I'm using LLMs a lot and I've been able to do much more than I used to- I find them great coworkers, technically very knowledgeable, patient and fast. I provide the big picture, keep an eye on the architectural soundness and code quality, and design the features. The LLM does the rest. The results are way above average.

Nobody cares

Just want to add this is my experience as well. Just solid coworkers. Of course they mess up sometimes, but easier to fix up than with humans and their politic and egos. I find I can actually reason for once instead of always fighting and deferring to whomever has The Biggest Opinion and not rarely just the loudest voice.

I think many people here work at nice, large places with reasonable and knowledgeable colleagues that are cooperative and mostly rational and try to do the right thing. In my experience that is not a common or widespread thing. Of course I only have small to medium business experience, but that's still a pretty good chunk of the economy. LLMs are an absurd, ridiculous win in those kinds of environments.

analytics/data engineer here. The approach described here falls apart on most datasets I've seen, because the source data has folds that are almost always out of context of the data itself. Even for a typical simple question a founder might have, like "what's the revenue for product X last month". Perhaps some orders don't have a Stripe record associated, and we receive money through a separate invoicing process. Perhaps there's a high revenue breakage rate between when a purchase is originally placed and when the payment goes through, and so a naive query for point-in-time revenue will almost certainly over-count revenue. The SQL the agent generate might not even yield directionally correct answers.

And that's when the agent even manages to construct a reasonable naive query. I've seen even Opus 4.6 ignore a `is_demo` column in the schema it was given when asked to construct a query for the number of active users.

Where I've seen text-to-SQL work well enough is when you're pointing it at data that's already been well-modeled for analytics such that the naive query a LLM will construct is correct by default. The data is either structured as a wide table such that no joins are necessary, or all the joins are 1:1 fact <-> dimension joins. All metrics are additive and so can be aggregated without asterisks. Columns follow a consistent naming convention, using the business domain terms a user would use in their prompt to the agent.

But that's a much thinner niche that what rawquery is proposing. You can't get around the analytics engineering effort involved in constructing a quality analytics dataset; the LLM will be a best a fuzzy fronted to your data warehouse, coextent with your BI tool.

Note: I do see value in value in rawquery's CLI-first approach to accessing data. In the right hands agents are very helpful at rapidly exploring datasets and validating assumptions on source data; but all the cloud data warehouse products I've interacted are all somewhat fiddly to access locally.

This is one of the best comments around.

This is yet another ad, it's tiring.

It's a post claiming average AI is useful... by a for-profit "data platform with a CLI that LLM agents can use directly". What are they going to do? Criticize the whole industry they are selling to?

I did not post it. I did not intend for it to be posted here - really. It just happened someone did see it and posted it. So I did not advertise anything :-)

The majority of devs are average. What a shocker.

The majority of any filtered group are below average. Imposter syndrome isn't a thing, 80% of people really did just barely make the cutoff.

This tracks. Tasks that used to be a day or two of grunt work are now an hour with Claude.

And there is a lot of that type of work to do if you're trying to grow a business. But, something in there should be trying to be exceptional or else you have no moat. Claude will probably not be able to breeze through that part with the same amount of ease...

Being average is a just stage LLMs pass through as AI makes its way towards 'expert' and 'super human' levels.

LLMs are trained to predict tokens on highly mediocre code though. How will it exceed its training data?

Humans can decide to write above-average code by putting in more effort, writing comprehensive tests, iteratively refactoring, profile-informed optimization, etc.

I think you can have LLMs do that too, and then generate synthetic training data for "high-effort code".

Well state of the art LLMs sure can't consistently produce high quality code outside of small greenfield projects or tiny demos, which is a domain that was always easy even for humans as there are very few constraints to consider, and the context is very small.

Part of the problem is that better code is almost always less code. Where a skilled programmer will introduce a surgical 1-3 LOC diff, an incompetent programmer will introduce 100 LOC. So you'll almost always have a case where the bad code outnumbers the good.

Current LLMs do tend to explode complexity if left to their own devices but I don't think that's an inherent limitation. Mediocre programmers can write good code if they try hard enough and spend enough time on it.

Thats because humans have "understanding" they can use to assess quality, without understanding "trying harder" just means spending more "effort" distilling an average result, at best over a larger sample size.

Who are you to question our faith? /s

Because you ask it to improve things and so it produces slightly better than average results - the average person can find things wrong with something, and fix it as well. Then you feed that improved result back in and generate a model where the average is better.

/end extreme over optimism.

Probably the same way other models learned to surpass human ability while being bootstrapped from human-level data - using reinforcement learning.

The question is, do we have good enough feedback loops for that, and if not, are we going to find them? I would bet they will be found for a lot of use cases.

This says "Editorial" at the top but has no authorship information. Who wrote this?

Nobody wrote this.

i fail to understand how text2sql on quite simple data sources is anything to write home about 3 years after it came onto the market? can someone elaborate?

where it gets interesting is when you have a custom system that your LLM surely never saw (custom ERP) that has 50 sometime cryptic tables, unclear look up tables and unexplained flags.

something no text2sql solution solved for us.

we built a second mcp that lets the agent look up business logic (generated from source code) and then does better queries. that i think is something i never read in a blog post about a text2sql solution.

There is a missundertanding. This is not text2sql. This is CLI first; ideally the tool being operated by Claude Code and alike.

You could use claude code for the "text2sql" kind of part, but this is not why this tool exists. Nor what the article advocates.

Dunno mate. Recently wanted to extend life of my old average laptop and need 4GB of RAM. I mean how much 4GB of RAM can cost? checks online... 55EUR, shipping in 2 weeks...

    This is not only average. This is actual magic.

    So let's be real: the SQL is average. The joins are average. The chart is average. And that took us less than 5 minutes and that was amazing, that is the entire point.

    You did not need a data engineer to model your HubSpot data, or a meeting to agree on whether it should be last-click or first-click or linear or time-decay or whatever.

    You needed a query, written fast, on data you already own. Your LLM wrote it. You confirmed it made sense. Your manager got a link.


    Honestly, average is clearly magic; prove me wrong.

I'll give it a go. This is generated slop, and the poor, factory-made quality of the writing undercuts every aspect of the argument.

It is like nails on a chalkboard.

Author here; I suppose the... side eye awkward monkey meme was a bit lost on you; it was written on purpose. Funnily enough. Everything is slop if you want it to be slop. This however, was written by hand my little hands. Now I might be a bad writter - that is indeed another subject.

This is all fun and games when you work with toy data samples. But most organizations are more complex, they have to match invoices from SAP with opportunities in Hubspot; or they have to consider that little sales territory exception for the sales guy in Munich to calculate the proper commission projection; or they have custom tables in Salesforce with 0 documentation; or... you get my point.

Not all context is documented, and some context has to even be changed because it doesn't make sense.

I find AI very useful, but I think a lot of this AI SQL products are misleading.

yes. Most people are upset and fear losing their job because they feel their job is sub-par. In reality, that's for most of them impostor syndrome, for some could be a wake up call.

Average is all we need ! I mean, working 50% is enough, right ?

A car that starts 50% of the time ?

A plane that stops on 50% of the flights ?

A pacemaker that beats only 50% of the time ?

David Goodenought said that average is enough ..

A car that starts 50% of the time isn't "average". The average new car starts more or less every time. (And if you said 'modal average', I'd say the modal average new car starts every time).

It is not average today because people in the past tried to do better, not average things

I think that's maybe the point of the article:

"Whereas before, average was expensive in terms of both time and effort, average became cheap."

adding LLMs to the incompetent doesn’t transform them

if anything it makes the world more dangerous

a reckoning is coming

the top decile will be janitors for the rest

> Claude doesn't ask you to define an attribution model. It doesn't open a whiteboard. It runs [query]"

Why is that a good thing? Claude didn't ask any obvious follow up questions, like what determined whether a user got an email or not? It is using the ab test terminology in Step 3 without any kind of confirmation that this is, you know, a valid test.

I flat out disagree. If we want things that are better than average, we will need to be making things that are not average. The machine doesn't learn yet.

Another writer trying to redefine a common english word to mean whatever they want it to mean at the time.

Pass.

Reply on news.ycombinator.com