Hacker News Reader

Comment by oblio | Hacker News Reader

The problem with this is that software engineering is a very unorganized and fashion/emotion driven domain.

We don't have reliable productivity numbers for basically... anything.

I <feel> that I'm more productive with statically typed languages but I haven't seen large scale, reliable studies. Same with unit tests, integration tests, etc.

And then there are all the types of software engineering: web frontend, web API, mobile frontend, command line frontend, Windows GUI, MacOS GUI, Linux backend (10 million different stacks), Windows backend (1 million different stacks), throwaway projects, WordPress webpages, etc, etc.

View on HN

Yeah I agree.

A controlled experiment done with a representative sample would be lovely. In the long-run it comes down to the financial impact that occurs incrementally because of LLMs.

In the short-run, from what I see, firms are trying to play-up the operational efficiency gains they have achieved. Which then signals promise to investors in the stock market, for which, investors then translate this promise into expectations about the future which are then reflected in the present value of equity.

But in reality it seems to be reducing head-count because they over-hired before the hype and furore of LLMs.

> In the short-run, from what I see, firms are trying to play-up the operational efficiency gains they have achieved.

The thing is all of this is getting priced in, and will be table stakes for any business, so I don't see it as a big factor in future success.

As I've said here, LinkedIn, and one a few other places, the businesses that will succeed with AI will be those who can use it to add/create value. They will outcompete and out-succeed businesses that can't move beyond cost cutting with AI[0].

[0] Which might not last forever anyway. Granted there are a decent number of players in the market, thankfully, but this wouldn't be the first time tech companies had hooked large numbers of individuals and businesses on a service and then jacked up the prices once they'd captured enough of the market. It's still very much in the SV and PE playbook. SolarWinds is a recent example of the latter.

I wanted to point you at https://neverworkintheory.org/ which attempted to bridge the gap between academia and software engineering. Turns out the site shut down, because (quoting their retrospective)

> Twelve years after It Will Never Work in Theory launched, the real challenge in software engineering research is not what to do about ChatGPT or whatever else Silicon Valley is gushing about at the moment. Rather, it is how to get researchers to focus on problems that practitioners care about and practitioners to pay attention to what researchers discover. This was true when we started, it was true 10 years ago, and it remains true today.

The entire retrospective [1] is well worth a read, and unfortunately reinforcing your exact point about software development being fashion/emotion driven.

[1] https://www.computer.org/csdl/magazine/so/2024/03/10424425/1...

The other problem is the perennial, how much of what we do actually has value?

Churning out 5x (or whatever - I’m deliberately being a bit hyperbolic) as much code sounds great on the face of it but what does it matter if little to none of it is actually valuable?

You correctly identify that software development is often driven by fashion and emotion but the much much bigger problem is that product and portfolio management is driven by fashion and emotion. How much stuff is built based on the whims of CEOs or other senior stakeholders without any real evidence to back it up?

I suppose the big advantage of being more “productive” is that you can churn through more wrong ideas more quickly and thus perhaps improve your chances of stumbling across something that is valuable.

But, of course, as I’ve just said: if that’s to work it’s absolutely predicated on real (and very substantial) productivity gains.

Perhaps I’m thinking about this wrong though: it’s not about production where standards, and the need to be vigilant, are naturally high, but really the gains should be seen mostly in terms of prototyping and validating multiple/many solutions and ideas.

"I suppose the big advantage of being more “productive” is that you can churn through more wrong ideas more quickly and thus perhaps improve your chances of stumbling across something that is valuable."

But I think there is a very big danger here - you build in the action but completely neglect the deep thinking behind a vision, strategy etc.

So yes you produce more stuff. But that stuff means more money spent - which is generally a sunk cost too.

In a bizarre way, I predict we will see the failure rate of software firms rise. Despite the fact these 'productivity' tools exist.

Yeah, I mean, you might be right. As others have commented, I think it's simply very hard to say what gains we're really going to see from AI-assisted software development at present.

And then of course there's the question of how many businesses have their key value proposition rendered obsolete, and to what extent it's rendered obsolete, by AI: doesn't have to be completely nullified for them to fail (which obviously applies to some software companies, but goes far beyond that sector).

Reply on news.ycombinator.com