The growing complexity of modern software systems | Hacker News Reader

The growing complexity of modern software systems

infoworld.com

133 pointstut-urut-utut57 comments

Microservices and serverless were the "we've jumped the shark" point for me.

Both can be a great pattern applied properly (nasty scaling problems), but they became a religion or a replacement for entire stacks. I see so many devs these days who are clearly confused but have no idea what they're doing is overly complex—they just think it's normal.

AWS and the like are great, but the move to make everything cloud-based is going to lead to a lot of unmaintainable (or in extreme cases, unsalvageable) messes later. It will be a sad day when devs only understand how to build or deploy something if it's using a heavily-abstracted corpo stack, or even worse, "no code."

Conversely, I've seen overly complex but "normal" monoliths. Thousands (millions?) of lines of dead code. Three copies of a file, all 80% the same, but only one in use.

We say one or the other is overly complex, but I think both are knife throwing just with slightly different details. If you can grok system design, microservices are easier to reason about. If you maintain your monolith well, it's easier to reason about.

> Three copies of a file, all 80% the same, but only one in use.

I too have seen this. The one benefit of monolith in this case is the three files are in the same repo so at least it's (theoretically) easier to standardise on just one.

If they're cross-repo duplicates, then it's a little trickier.

That complexity is due to developer laziness, lack of discipline, and inexperience—not the pattern. And again, it's not monolith vs. microservices. It's "when is each technique best applied and why?"

The future is SharePoint workflows!

As someone forced to use them on a daily basis, they are at least open to being used in an overly restrictive corporate environment and insultingly easy to use. But yes.

I had to fix someones workflow where they had it kicking off other workflows endlessly stuck in a loop until someone changed something. They could take a week to follow up.

So no, I think only trained teammembers should be allowed to touch them. Even simple coding with workflows is difficult for non-programming oriented people.

no code will increase alongside ai, developers won't have jobs in the future. it'll be interfaces where the system builds it's solution okay after you tell it what you want. basically jit on steroids

all of the no code creations I have seen posted on this site in the past month are incredibly slow and sluggish. Sometimes even straight up buggy, for instance that font compare picker thing, would sometimes load no font (null) on the left side. As a paradigm, it has a long road to go.

If people really think that no code is acceptable, then I am going to cry. It won't be an improvement.

Yes. It's not about improvement, it's about ease of use and reduction of work, or: convenience. That convenience is going to bite an incredible number of people from behind and send a lot of businesses spiraling.

Will there really be a no code, or is that just moving the level of abstraction one more level higher... While still needing occasional coding to solve those bits that aren't available...

I will have to see it to believe it. 4GL promised the same thing. I actually look forward to the day happening when I become a farrier in the car age. A lot of people smarter than me will turn from code hopefully to hard science and we can start advancing again.

Do you think science is stagnating because React jobs are sucking the firehose of smart hard-working people?

Fortran also promised the same thing.

The bad developers won't, the good ones will. It's going to be very lucrative to know your stuff when nobody understands what the no code approach is doing under the hood.

> Microservices and serverless were the "we've jumped the shark" point for me.

I'm not really familiar with micro-services.

Could you give me a concrete example of a system that people would built with a micro-services approach but would be simpler as a monolith?

> a system that people would built with a micro-services approach but would be simpler as a monolith

It's not a binary decision. Micro-services are a tool/technique best applied piecemeal for specific things. For example, "we started out with our API in the monolith but it's got more traffic than we expected so we should split it off to its own service."

The point I was making is that people take this wholesale approach and add unnecessary complexity (i.e., split the API, the payments system, the DB/ORM layer, etc. off into services from day one).

I have found that both Microservices and serverless work well for "batch-type" writing applications where data is processed asynchronously.

It does not work so well for interactive reading applications like a REST-api.

A lot of nostalgia for an era that didn't exist: “It has never been more difficult to be a software developer than it is today...” and “Everything was so much simpler before..."

SW dev has always been overly complex, long before AWS and the web existed. It's just how we do things in software engineering. Too many choices and lack of physical constraints leads us right to the edge of where things totally fall apart.

Just because it's always been complex doesn't mean the complexity isn't increasing. Software development is ridiculously complex, increasingly so, and one often doesn't have any tools to deal with it.

i think it's done for job security. kinda like antivirus software always had new "threats", but that's probably not true, it just feels sometimes like we are reinventing the wheel to keep tech jobs relevant, at least in certain fields

If this were the case, you'd see at least some people/orgs doing better and spending less than the current "best practice". I don't think it's reasonable to say software developers are just fooling everyone.

VB 6/Delphi in a 3-tier application versus HTML/CSS/JS npm spaghetti pretending to be a GUI framework instead of interactive documents.

The biggest problem seem to be microservices. Basically, an army of cargo culters missed the small print that you should not even be thinking about them unless you work with at least 100 programmers. The talking heads that popularized microservices are also guilty of this. They occasionally mention that microservices are only for large teams but not nearly often enough.

Cargo culting is just what you have to learn to accept and work around, because there will always be developers who fall for it. A decade ago, everyone was writing dependency injection frameworks because that was the hotness and debugging became a nightmare, but, hey!, we completely decouple EVERYTHING! Then there was no sql (mongo etc) and if you were still using sql, your system was shit. Oh and way before that there was GoF and you ended up with UserBillingFacadeFactoryFactory abstractions.

None of these technologies are bad. They're just bad in the hands of mediocre developers.

Yes and no. There are legitimate reasons to build microservices with small teams (such as requiring differing runtime environments due to hardware/architectural constraints). They should never be the default at a team below that scale though.

modern software has become complex.

but people, especially those involved in this complexity, insist that this is all normal. simplicity (or whatever it will be) is coming for modern software. we are seeing the early stages of that upcoming change: denial & resistance.

denial is when we hear engineers say that software is simpler or no more complex than it was. resistance is when engineers try to convince us that "we need that complexity".

following the well-known psychological curve, next up is acceptance. and i have started to see it already. kubernetes, for example, continue to deprecate poorly designed or overly complex features. it is hard because of backward compatibility.

all that to say that change is coming. maybe we will simplify and start the cycle all over again :)

I am not a fan of denial & resistance arguments. Do bad thing X. Try to convince others that X is good and its failure of acceptance is just people's denial. Next step is the well-known psychological curve that you will accept X eventually. This can be applied indiscriminately to anything one wants to push without having to defend the specifics. We should instead question all assertions - there is no law out there that denial always precedes acceptance.

> there is no law out there that denial always precedes acceptance.

Yeah. "Denial, resistance, acceptance" -- isn't that from the six (or was it seven?) stages of grief? But grief is for something inevitable and irresistible, a fait accompli: When a loved one has died that has already happened and there is nothing you can do about it.

Here, though, if enough of us think that our industry is going the wrong way, maybe we can still stop it.

One thing I've seen is that forced prioritization can help reduce complexity. The idea is that if you don't have enough resources to re-invent the wheel you will be forced to think harder about what you choose to work on. This has to be cultural and supported all the way to the top of the organization. Counterintuitively too many resources can slow you down.

This is true. Someone has to be the architecture person who can separate the actual complexity created by your requirements as they exist from manufactured complexity created by using architecture techniques that don't follow from requirements.

As other posters have said, I think there is somewhat too much nostalgia about how "easy" it was in the old days. I was reading about software dev in the 70s and it took like 4 years to write what we could do in a day today.

Also, I think that relatively, we are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better, but after a bit longer, we will have more experience of what works and doesn't work and we will consign certain things to history.

Think about what medicine was like in the first few centuries? Leeches, trepanning, toxic "medicines", they presumably had the same issues until they established a much better foundation of what works and what doesn't which has made medicine move on.

We will keep some of our cool things but others (XML anyone!) will just die off as we realise they are a solution to a problem that most of us don't have.

>>Think about what medicine was like in the first few centuries ...

I spend thinking about medicine quite a lot (health problems in my family run deep) and this does not give me great deal of hope for our industry at all. There is so much quackery everywhere - most of diagnostics is based on some magic guess work. If symptoms are non-specific than You are out of luck - You have to spend years visiting random doctors hoping that one of them will have fresh enough memory of some book that he read years ago to guess right.

And even when the diagnosis is easy the therapy can be not that far away from leeches.

Few years ago I had wart on my hand and dermatologist offered me three options:

- freeze it to death

- burn it to death with acid

- cut it out

If You look at this options closely they require some high tech to be possible but from medical point of view they are as primitive as leeches. I honestly belive that in 100 years time when nanorobots will be available to everyone the doctor will prescrabie therapy that is basically nanorobots attacking the wart with little flamethrowers.

> We are still in the early days of software where lots of people are clamouring to invent the silver bullet to make it all work better [...]

I have a feeling that (especially in devOps/kubernetes) often there is a wish to have a magic button/command that does everything. Like in a rube-goldberg machine.

What is forgotten then is the cognitive overhead of templating YAML inside templating YAML which is needed for the magic-button to be configured.

And if the buttons behaviour needs to be changed very few weeks, not much is gained.

Using packages, frameworks, and libraries has made it easier to build complex applications and systems faster, but it has made maintaining and debugging them much more difficult. If you are using 10 libraries, each with 100,000 to 1,000,000 lines of code, interacting with an operating system and drivers that are another million lines of code, you can't possibly understand what it is all doing, even if you have all the source code. You don't have time to even read it all. Trying to debug something you don't understand turns into trail and error to find clues as to what is going wrong. Sometimes you never figure it out. Hopefully you find a workaround. My first experience of this was in writing a 3D visualization for evaluating an electronic warfare system which used a 3D library from a third party. My aircraft would always disappear after a few minutes. The people who wrote the 3D library couldn't help me. I couldn't figure it out since I didn't know what the 3D package was doing internally. However I found that popping up a helicopter at 2000 feet for a half second fixed the problem. So from that point on all demos would be disrupted once in a while due to a helicopter popping onto and off the screen. I eventually found a way to make the helicopter invisible. It was still being popped up, but nobody could see it. I suspect there is tons of software today that works because of hacks like this, which makes them very fragile and prone to break if you make changes, and makes them almost impossible to troubleshoot (in addition to the underlying complexity).

Every few years, some version of this article comes out. The version of 30 years ago lamented how programmers don't really understand the hardware anymore because a compiler abstracts the machine.

20 years ago, it was about how virtual machines (like the JVM / CLR) abstract memory management and developers write bloated software.

We simply have more layers of abstraction that we depend on, and the interesting problems move up the stack. That trend will continue until the machines take over and write software.

Until then, we'll continue to see this article show up, s/kubernetes/next-new-hotness.

The stack also improves. Early versions of the JVM were very noticeably slower than C++, and the first CLR GC was so poor that my team at Microsoft had to resort to arrays of structs for keeping a service’s state. These days they’re great, and it’s hard to do better without putting a lot more effort into something like Rust.

The JVM is great, but I think it's not too difficult to do better in a language that doesn't force so much pointer-chasing. Cultural differences probably also play a role. Java systems tend to be engineered such that more pointer chasing happens even in places where the language doesn't force it.

Does the CLR improve on that much, with its stack-allocated structs?

> Early versions of the JVM were very noticeably slower than C++,

The fastest java software I use feels much slower than the slowest c++ one (talking about UI lag)

There's a popular saying about hardware vs software performance gains: "What Intel giveth, Microsoft taketh away".

I think we can have another in anoter level: "What the language giveth, complex patterns and libraries taketh away". This works not only on performance levels, but also in terms of complexity.

* https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law

* https://en.wikipedia.org/wiki/Wirth%27s_law

For what it's worth, one of the QEMU (the open source emulator and virtualizer) maintainers, Paolo Bonzini, recently gave a talk about this topic. Here's an extended summary of that on LWN:

"A QEMU case study in grappling with software complexity" — https://lwn.net/Articles/872321/

Although the article uses QEMU as an example, the lessons discussed in it are equally applicable to many other projects.

(If you're short on time, read the intro, "Sources of complexity", "Ways to fight back", and the short conclusion. The QEMU command-line case study section might be interesting for those who use a lot of QEMU, or to those devs who use QEMU to write higher level tools.)

I'm not sure I agree with this. In the past, software development and how things were deployed etc was the wild west. Every company and team had completely different standards and methods etc and lots of it was manual and little to no testing etc etc.

Now, with things like AWS, Terraform, increasingly converging popular programming languages, frameworks and best practices like tests, code review etc, in some ways things are less complex.

I agree with your disagreement. I've been in involved in web dev for 30 years. There is more moving parts, but IMO they better align with the actual complexity. This is likely only true for professional, production grade apps where the full range of concerns are in play such as security, maintenance, testing, migrations, scaling, etc. Deploying a simple app is more complex. For the apps I built I don't have to plan server purchases months out. I don't have to coordinate multiple apps on one machine. I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity.

Beyond that, in some instances even for simple apps there is less complexity once you know the technology. That's not complexity though. That's a buffet of useful options.

Good architecture decision are even more critical today.

> I don't have to worry about swapping out LBs and Firewalls as a nightmare connectivity issue. I don't have to scale file storage regularly. I can scale up service DBs as needed. All of that reduces total complexity.

You're wrong. Complexity has increased by a huge magnitude so much so that it's actually unmanageable by a single person.

What's going on in your case is that the complexity of all of this has been abstracted away from you by amazon. They have entire teams of people making sure you don't have to deal with the actual complexity. That's the difference. This isn't an actual abstraction in the same way a high level language abstracts the details away from you. This is an abstraction where a team of people take care of you. It is not a true abstraction because you cannot wield this abstraction yourself in the same way you can wield a high level programming language.

Make no mistake, the actual complexity of it all has increased by huge orders of magnitude.

I think they mean that it reduces the complexity that a single developer has to deal with.

Right, but this redefines "developer" to "consumer of a cloud platform's services".

Complexity I don't have to deal with doesn't exist. Only if I really don't have to deal with it. We use BigQuery to report on 100s of terabytes of data. Behind the scenes it is massively complex. I don't have to deal with it. It's not meaningfully complex for me.

Wrong. Just because you don't deal with it doesn't mean someone else is. A true abstraction abstracts complexity away so no one deals with it.

Amazon abstracts complexity away by having someone else deal with it. Big difference. One would imagine software progressing towards abstractions that are simpler and allow one person to do more, what we are getting is instead abstractions that only make it seem this way when in reality it's going in another direction.

Complexity is increasing to the point where one person can do less, but companies like amazon don't let you realize this by hiding the true complexity behind other people who deal with it.

I agree, but I think you are overselling how standard deployments are. And, underselling how well older java frameworks actually solved most problems folks would have. The standard "war" file goes a long way for what folks actually need for web applications.

And that completely ignores how well PHP works for standard web sites.

So, if anything, I would say that things peaked with very understandable deployments about a decade or so ago. Sure, we now have "web scale" orchestration frameworks for the giants out there, but... I question how well this actually solves problems that folks have.

Worse, I also challenge that anyone really has a handle on scale. Walmart's site, as an example, was completely garbage today when they released there silly supply of PS5s. In all sorts of ways that showed that eventual consistency is the unnamed assumption in how all "web scale" things are created.

I worked in 5 companies so far that used AWS, and consulted for a handful more. None of them did things the same way than the others.

Even inside AWS there's no standardised way of doing deployments. Are we using EC2, Lightsail, Lambda, Elastic Beanstalk, EC2 with ECS, Kubernetes? For databases, are you hosting yourself or with EC2 or using RDS? Or are you using containers? Or Kubernetes? Too many options.

Pre-AWS, however, things were much more standardized: Linux servers. Or Windows for 5-10% of cases.

Is it more complicated though ? I've been developing for over 30 years, and I don't feel like its got any worse, I think it's the nature of the medium.

These days I fight with npm or nuget dependencies problems, 20 years ago I was trying to write Delphi code to do simple REST requests, the complexity moves around but is still there.

Out of the Tar Pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) was written 15 years ago, and it's all about curbing complexity

I think this is an example of simple vs easy. Writing request code 20 years ago was harder, because you needed to read docs and write a lot by hand. But it was simpler because it didn't add any additional hidden complexity from libraries or whatnot. At least in most languages it got better: we only need something simple and easy, like Javascript's fetch. Does complexity we don't have to maintain count?

About Out of the Tar Pit, complexity manifests itself in various shapes. The essay itself lists some that have gotten better: state is one, with people using more functional code. The "Power corrupts" part has also gotten better, IMO, with languages becoming simpler (Go) or more restrictive (Rust).

I think the "Code Volume" part is still a big one. One thing I'd like to see curbed to avoid code volume are the proliferation of code for handling edge cases. Having libraries and code able to handle tens of edge cases is both a blessing a curse. However most developers and business people can only see the blessing part.

I don't quite agree with much of this article. It paints the industry with a broad brush and not everyone has had the same experience - least of which me. The article starts:

> The shift from building applications in a monolithic architecture hosted on a server you could go and touch, to breaking them down into multiple microservices, packaged up into containers, orchestrated with Kubernetes, and hosted in a distributed cloud environment, marks a clear jump in the level of complexity of our software. Add to that expectations of feature-rich, consumer-grade experiences, which are secure and resilient by design, and never has more been asked of developers.

Comparing the most complicated way to build and run an app with something that isn't as complicated is just a straw man. Today you can still build a simple, monolithic app and deploy it into a completely managed service. All the points raised address certain problems:

  - microservices - sliced out of a monolith when complexity or scalability becomes difficult
  - containerisation - deploy an app with all of its dependencies down right down to the OS
  - orchestration - ensure reliability when pods/tasks fail, rolling updates, commoditize hardware 
  - distributed cloud environment - remove the need for on-prem, or have the ability to elastically scale without monthly VPS commitments
  - secure & resilient by design - apps that make money should have good uptime, not leak data

None of those are mandatory for all apps - it's pick and choose as and when you get the need. But also what is the alternative provided from "years ago" when things were seemingly simpler?

  - microservices - ever tried keeping velocity when building XMLSOAP based apps?
  - containerisation - write a bunch of ansible scripts or maintain disk images that can be sent to server fleets with bash scripts, else more scripts to apt-get/apk install stuff. On windows before package managers? enjoy shipping zips and exes with your code.
  - orchestration - scripts?
  - distributed cloud environment - build your own datacentre if you're big enough, lease out some racks if your not and have a systems team maintain them, patch them, RAID stripe disks and swap out dying infrastructure; or sign up to managed VPSs and raise a requisition order each time you want a new server that'll take a week to go through.
  - secure & resilient by design - a lot to do for a 6-9s uptime; so just lock down your release process with a ton of bureaucracy and approval gates by different teams. not releasing is great to improve uptime SLAs.

I don't have a good impression of the "good old days". The complexity was always there, it was just managed by different teams.

The real question is should a software developer need to know and work across all of these things, or are people who no longer have "systems management" job move into DevSecOps and augment a software team?

Perhaps this suits todays technology focused developer. There has been no progress on how to effectively determine and model complex business domain requirements into code, so just focus on the technological requirements and plumbing.

some mention it about microservices but no one mention about the emerge of cloud?

when we are still using monolith and using old deploy to one server, that would be easy. Then there is scaling issue and that's where the cloud came because really, when you have to manage a lot of things at once, it's better to let someone manage it for you and you can focus on your business logic.

Obviously, the practice wasn't perfect. A lot of "micronization" comes up instead domain driven problem but that's the problem on mostly people not the tech

Reply on news.ycombinator.com