Hacker News Reader

There's a simple arithmetic to this: Computer Science is a growth industry (still), and has been on an exponential curve for decades. It doubles roughly every 5 years, which means that every 5 years half of all programmers have less than 5 years of experience, by definition.

More importantly, the absolute number of these inexperienced programmers (and architects, and admins, and ops people, etc...) is growing, so the "total ignorance" is increasing.

I've just gotten involved in a project to "modernise" a "legacy" ASP.NET project. By legacy, they mean ASP.NET 4 running on .NET Framework 4.8 with SQL Server 2017. Not exactly ancient! This apparently is no longer the hot new thing and "has to be" rewritten using cloud microservices or something.

The app actually runs just fine in its current state. But none of the programmers involved know the pre-core version of ASP.NET or anything but Azure. So it's time to revamp it.

> Why is the modern software industry in such a constant state of flux?

I see huge software frameworks sort of like political parties. The community around each of them promotes the benefits of using their framework over someone else’s, and we “vote” by hitching our products to these tools.

If you didn’t choose wisely, or the framework didn’t develop in a way that benefited you or your org, you’re probably keeping an eye out for what’s next.

The other component to all of this is figuring out how to stay relevant in the industry, and I see more and more frameworks making design decisions around this fact.

Take Gatsby, for example. I still can’t figure out why they use graphql to populate templates other than developers wanting graphql experience, ergo let’s use gatsby. The whole point of graphql was to minimize data transfer from server to client, but somebody thought this was somehow more beneficial to generating a static page than a vanilla JS object.

> Every time Slack redesigns its interface, users need to relearn where all the buttons are. Nothing can possibly justify rolling out ten million redesigns;

I think this partly falls on engineering orgs with development teams that need to justify their existence by constantly rolling out these kinds of changes. Because of Slack decides to freeze the UI, how many front end devs are going to need to be reallocated?

This rant was somewhat aimless, but I think ultimately that many of your rightful criticisms are the product of the politics of engineering companies and communities.

> I think ultimately that many of your rightful criticisms are the product of the politics of engineering companies and communities.

I don't doubt that. I would, however, like to see some people in those organizations try to enact changes, particularly those nearer the top. The business benefits of doing less unnecessary work are relatively obvious.

Separately, a lot of the attitudes towards legacy software use ought to change. Most techies regard organizations still running Windows 7 in 2020 with a mix of amusement and scorn. "Windows 7 came out X years ago, why haven't they upgraded by now?"

Well, why haven't they? Maybe because they already paid for the development of Custom Software X, and aren't particularly keen in redoing all of that work just because Microsoft decided to change things. I don't feel great about Microsoft holding security patches hostage just because more Windows 10 customers would improve their brand or some such. I understand needing to charge for said patches past a certain period, but why not continue providing it for a reasonable price, as long as their are paying customers?

It might help to think of the entire field of software development as a single living organism that's clumsily, through trial and error, trying to grow into maturity, to find the best way for software to go from concept to realization.

It's very young. Thousands of years passed between the earliest mathematics and the first calculus. Software has had, at most, a hundred years.

Things like microservices, SOLID, dependency hell, npm are all examples of the software organism trying to figure out what works in reality and what doesn't. None of the individual cells -- us developers -- can get a big enough view of the whole thing to understand ahead of time exactly what will work and what won't in large scales and over long timelines. We might argue from some ideology or personal experience, but we don't really know.

Software wants to be inexpensive, both in initial development and in ongoing maintenance. But what does inexpensive really mean? Does it mean that it starts with a formal proof that takes weeks to develop and then is cast into code and never modified again, or does it mean a function that's banged out in minutes and then has an entire suite of test cases written against it and is maintained in minutes and hours here and there for years?

The software organism has learned some things it shouldn't do, but that required lots of developers trying it out, and other developers wailing and gnashing their teeth about it and altogether saying, "this is bad and it doesn't work and we must do it another way!"

We aren't going to see software development reach maturity in our lifetimes. I don't think it will happen for another hundred years at least. I think, eventually, software might settle down to a handful of purpose-built languages, and somehow all the problems of code reuse and distribution and documentation will gradually get solved, probably over some massive and absurdly complex network. Developers might become more like immune system cells, wandering around the network and fixing up the little bits here and there that they specialize in. I dunno. In any case, before any fo that happens, we first have to find out why every other approach to software development is worse.

Besides the fact that software is a relatively young field, it also aims to exploit hardware devices which have been evolving ridiculously quickly.

AI will be writing the code by the end of the century. You'll long for these debates when writing code is relegated to what hobbyists do in their garages with wood. There will be few admirers and even fewer buyers for your carpentry then.

Without taking the time to make a data-driven argument, I'll just hypothesize that, it's a natural result of lowering the barriers/costs of entry, to making changes.

30 years ago, if you worked in software, then you were almost certainly building shrink-wrapped products, that had to "go gold" before being printed onto 1000s of CDs at a factory. How quickly/easily could you change anything? Not for years, typically. Hence, you were very careful to try and build things right the first time, but the ironic thing is, with such long feedback cycles (i.e. you never really got feedback), that all this waterfall-style effort to plan everything and do everything right the first, rarely ever materialized as perfect/seamless UX for the end user anyways. It turns out, you need to iterate to improve.

Nowadays, the pendulum may have swung too far in the opposite direction... it could legitimately be said that, some things are too easy, tinkering with changes is too easy, and thus nobody expects to have to maintain something for a long period of time. If the cost of rewriting everything is low enough, people will be tempted to just do that whenever they change their minds about something, rather than forcing themselves to predict the future upfront and make durable decisions that will survive for 5-10 years out in the wild with no changes.

IMO, there's no single right answer for all scenarios, all products, all forms of engineering, etc. But it's explainable why such trends may emerge over time.

> it could legitimately be said that, some things are too easy, tinkering with changes is too easy

This is certainly the case in modern game development. It's now routine for a game to receive a major patch within 2 weeks of release.

> there's no single right answer for all scenarios, all products, all forms of engineering, etc.

Sure. A search engine for cat photos can go ahead with a 'minimum viable product' release, and improve incrementally. Not so for avionics.

To be fair, the minimum viable product for avionics is more complex than anything most of us have written, so it's not implicitly wrong.

I don't see how avionics' complexity is relevant. Game engines are complex too.

Avionics cannot be developed iteratively, for domain-specific reasons. The costs of re-testing, re-certifying, and deploying, would be crushing. Besides that, how can an aircraft be delivered if its avionics are unfinished? What's the gap between 'viable' and 'finished' here? Games are of course a different domain.

I think that frequent changes and experimentation are great qualities for new software, which is still figuring things out. However, this should be a temporary state, and it should be clearly communicated to users with some sort of extended "beta" moniker. After some number of years, the software's creators should recognize that their product has matured, and the pace of change should slow down drastically.

I'd say 50 years is the magic number.

Just to be clear, by "some number of years" I was thinking closer to 3–5 years, although I of course understand where you were going with that. :)

> Why is the modern software industry in such a constant state of flux?

Not all of it is. C is remarkably stable, for instance. C++ is much quicker to add language features, but has a real emphasis on backward-compatibility. I think Java and C# have done a pretty good job there too. Web front-end frameworks have a higher churn rate, yes, but for now, Angular and React are king, and have been for a while. Maybe they'll be toppled by some other near-identical framework in time.

> Is there an end point where we all actually figure out how to develop software properly?

Depends on the domain I suppose. Embedded code, operating systems, and drivers, have been written mostly in C for decades now. Perhaps languages like Rust will make some inroads, but the 'normal way' has been pretty stable. I imagine other non-software fields have seen more radical changes than that over the same interval.

> Every time Slack redesigns its interface, users need to relearn where all the buttons are. Nothing can possibly justify rolling out ten million redesigns

I once read a rather cynical take on this same question: these companies hire a UI team, and that team isn't about to admit that they have no meaningful work to do.

> Why is the modern software industry in such a constant state of flux?

Is it though? Or is it barely trying to catch up with development of hardware and new fields of application?

Or should we just do AI, web and realtime distributed databsesa in Fortran and Cobol?

Maybe people who feel the excessive flux just lack the capacity to notice how fast environment changes? And that's why they feel flux excessive not borderline inadequate?

This might be cynical but maybe because most individual pieces of software don't matter too much.

Software is much easier to change than other things that humans build. It's easier to replace, sometimes with success, than what other disciplines have to work with. A lot of software isn't used in safety critical areas either so failing is acceptable. Lots of companies on the web have lots of downtime and still make money.

Even poorly written software can absolutely print money with huge margins. And old software that's still around and kicking is often the most successful!! A form of survivorship bias.

Sure, _products_ often matter a lot. But within them and as long as they still solve their problems, it doesn't really matter what's inside. If a product I use as an end user has a lot of internal churn, but still solves my problem adequately, I don't really care for the most part.

Sure, if it's medical software, or aviation software, or software to replace an elevator panel, I might care a lot. But your average web app or desktop program?

I think software engineering is depressingly, suboptimally stable. The space of all possible ways to write programs is so, so large and almost completely unexplored. Lisps, Hindley-Milner type systems, Cox, neural networks with autodifferentiation ... whenever we substantially depart from the local optima around C-like programming, we find cool shit.

However, the vast majority of programming is done to build nearly indistinguishable CRUD SaaS apps. If we could just spend half our time doing real, interesting exploration of programming-space to satisfy our curiosity, maybe we could all agree to build apps with a more boring tech stack. However, that would require a large scale change in incentives that is hard to imagine shy of a Space Race style event.

... or Jonathan Blow was right and C was more or less the global max, and everything else was a mistake. I really hope that isn’t the case.

It's because not even the foundations are not properly engineered, mostly due to lack of funding (because they can't be sold, and anyway the market doesn't reward proper engineering beyond a basic level, and governments don't massively fund open source), as well as lack of accountability for broken and insecure software.

For instance, look at programming languages: to write provably correct code you need dependent types, but none of the dependently typed languages produce optimal code or have a viable ecosystem. This problem has been unsolved for more than 30 years. We also didn't even have a barely decent production language until Rust in 2015.

Or databases: the best database is PostgreSQL and it doesn't even have things as basic as unsigned integers in core (!!!) and doesn't support multi-master in core (and of course, written in an horrible language, which is C). Again, problem unsolved for more than 30 years.

Or operating systems: all OSes are fundamentally insecure, because they are not written in a proper programming language, and at least the desktop ones don't generally use a hierarchical architecture, but rather have a single gigantic kernel-user boundary (plus they have horrible APIs, written in horrible languages, etc.).

Most other programs have no chance of even being provably correct to a given spec since the foundations are missing, and thus much less of provably correctly implementing the provably optimal solution to a problem. They are not even engineered properly with reasonable expectation of correctness, because there is no liability if you release an incorrect program.

Of course it's not clear if this is fixable without developing a general purpose AI to write the software or massive societal changes.

> Why is the modern software industry in such a constant state of flux?

This is true of both language design and software practices. It's painfully obvious. There are no standards for what constitutes "better" or even "good". Is strict typing better? To what degree? What's it worth as a tradeoff? There are almost no quantitative analyses (https://quorumlanguage.com/ - an attempt was made), which is lamented https://vimeo.com/9270320 by some.

Even ROLES are not well defined (https://www.youtube.com/watch?v=GhfVK_ubk8U) because roles are dependent on known concerns, which are also fuzzy in this industry. Saying development is immature is so understated, it's laughable.

For python 2/3, it was because the BDFL made a wrong call. But we are past that now, and we collectively learnt something from the debacle, including the BDFL himself:

> As for "Machiavellian Python maintainers", that's real. There have been explicit attempts to apply pain to Python 2.7 users. This has decreased since von Rossum got stuck maintaining Python 2.7 code at Dropbox as his day job.

(https://news.ycombinator.com/item?id=13020680)

For java 8/9, it was because Brian Goetz still has a boss. We can just compare https://www.youtube.com/watch?v=2y5Pv4yN0b0 to https://www.youtube.com/watch?v=wHoRBvt3U6o. Hopefully some lessons were learnt as well.

Uncle Bob used to blame it on the doubling of developer counts every 5 years. Which means half of us have less than 5 years experience. Which makes democratic decisions a little awkward.

Arrogance. Younger programmers or people just getting into the industry assume they can automatically just do it better without taking the time to understand what is already out there. Reading and maintaining existing code is harder that writing your own.

Laziness. Rather than work from the lowest level denominator (what can vanilla javascript do for you?), do a quick google, see some random comment somewhere on the internet that uses library X and just add it to the lumbering Frankenstein's monster of dependencies without a care for the future.

React + Redux + Typescript is nice in terms of frontend development, but the shear number of packages required for a small list of direct dependencies is staggering. Really it is absurd. It was created by people who have the above two tendencies I would say.

While I think there is plenty of people not learning the pasts problems, I think it's unfair to say it's all laziness and way over complicated. Business solutions are hard, as they're designed by people through a game of telephone until you get to the engineer intended to make it. Maybe the company put the burden on a junior engineer to save money. Should they tell their boss they need to spend a couple months researching? Okay so the company hired a senior engineer, who has been using X tech for 15 years that has seen a large decline in usage. Senior builds product, but gets a better offer and leaves? Who now maintains this? Hiring for X tech is hard, you bring someone on to learn it but they kinda lied at how much they knew and since no one in your org knew much about it they passed through. Now they're making a mess of the beautiful architecture the senior laid out.

Stop blaming this all on junior programmers and javascript. That's just saying those damn kids and their lack of respect for their elders is what's causing it. Truth is this is a relatively young industry, that's being used in just about every single aspect of life, where the cost to make a change is minimal, where the ability to learn is one of the most accessible skills, where business demands change on a dime.

Stop your own arrogance and laziness in understanding the landscape and the people in it.

Apparently you didn't need any answers after all as you have it all sorted out for yourself already!

Human factors. Developers ship a new OSS framework Z because it makes an extremely subjective improvement on framework X in their opinion. And the process of writing Z satisfies an intellectual curiosity to write something from scratch and it also looks great on your resume to have written your own framework. Z framework actually gets popular, other developers, kind of bored at work from using the same tools for ages try out framework Z. And it's nice to try out something new, it makes your work day feel fresh and different and you learn something. Framework Z slowly gets more popular and now it's the hot thing to have on your resume because the smart bleeding edge developers are using Framework Z. So more devs start using Framework Z, try it out on a side project, or a new project. Eventually even the skeptics about Framework Z have at least tried it out and put out a blog post with their thoughts on Framework Z.

Framework Z feels legitimized, it's a boon to a lot of careers. Lots more people use it. Recruiters start to ask for it as a requirement.

I don't want this to sound cynical, because I think it can sound cynical. I suspect the industry starts from good first principles of wanting to learn new things and try new things and make improvements, and grow themselves. But there is also serious capitalist pressure on the whole system. And a lot of real money on the line, and careers and Twitter followers and conference speaking engagements.

So between the good reasons and the bad reasons, in web development in particular, things move way way way too fast for little gain.