Hacker News Reader

Comment by aaronbrethorst | Hacker News Reader

The architecture of our systems has gotten a lot more sophisticated and complex over the past 20 years. We’re not running monoliths on a few beefy servers these days.

I am running a monolith, and troubleshooting is still pretty easy.

We’re operating distributed microservice ecosystems on top of a deep stack of frameworks, abstractions and runtimes that are all running on other people’s servers (aka 'the cloud').

Yikes, good luck.

Snark aside, I just want to remind most people out there: you do not work for Google or Facebook, and your systems do not have to be architected as if you are the next Google or Facebook. Neither of these companies started existence with microservice architectures or whatever. Save yourself the pain. Build a product. Find some customers. Worry about the architecture astronautery later.

View on HN

The problem is that if a company advertises a job that is all about maintaing one or two monolithic applications without K8s, distributed system topics, tracing, microservices, AWS/GCP, etc., well, then not many engineers will apply. That's actually a paradox in itself: my bet is that only a few group of (good) engineers would apply to such a job. Management otherwise usually thinks that the bigger the pool of applicants, the better for the company. Also, a lot of engineers out there are resume-driven: they do want to work with microservices, the cloud and k8s.

It's a vicious circle.

> then not many engineers will apply

I would. I don't run server farms and don't need to provision entire environments so K8s would be extreme overkill for anything I do. Docker makes sense to formally define system requirements but even that I use sparingly.

I develop embedded systems though and usually only trace coffee machines.

I do host stuff on AWS and gladly use all the services they provide. Neat, easy and less maintenance, but the most important thing for me personally is that I work with an account that doesn't use my own credit card.

I can understand the applicants though, there is a fear to commit to solutions where any experience might not increase their market value. But from my experience that fear is highly exaggerated. There are people that reject candidates on these accounts but usually not the good one that know they need to invest in engineers. There are also quite a few bosses that are only interested in the newest tech. They have absolutely no idea what that means, but they will buy it. And a lot of cloud software hits the sweet spot for them.

> > then not many engineers will apply

I would.

Like you said ; one point is probably market value and another future relevance. I have had discussions with hiring managers lately and almost none that I talked to actually use most or any of the buzzwords they put in their ads. They just say they do to get more applications.

How broadly is this true?

Some of us developers do want to focus on things that are useful to people, that we can describe to non-tech people we meet in public. Are those of us who run bootstrapped, all-remote SaaS companies going to have trouble hiring?

Based on my experience:

- many frontend engineers would run away from companies that hire for jQuery skills. Frontend engineers nowadays usually want to write React and use Vercel and what not (even though the product may only require jQuery)

- many backend engineers would run away from companies that hire for PHP. Backende engineers nowadays usually want to write Python/Go/Kotlin/Java, well, anything except PHP, even though modern PHP is well-suited for most SaaS/web-products

- many infrastructure engineers would only apply for companies that use K8s, modern CI/CD pipelines (e.g., GitHub/GitLab) and cloud providers (e.g., AWS, GCP) and run away from anything that smells like on-prem installations using Ansible/Chef/custom bash scripts, even though the vast majority of web products out there do not need K8s/AWS/Terraform/etc.

I think in general, engineers do want to build useful stuff for people but only if they get to play with the coolest tech.

All right, you got me. I've eschewed PHP for almost 20 years. And I don't use jQuery in new code either, though I do sometimes use plain, framework-less JavaScript, which in modern browsers provides a lot of the functionality of jQuery but with no runtime overhead. I wouldn't disqualify a candidate for being primarily experienced with jQuery, and I'd be curious to see their modern PHP.

> even though modern PHP is well-suited for most SaaS/web-products

Maybe, but is it pleasant to use?

My memories of PHP are not fond at all, it is a really miserable language to write. Maybe it has the features it needs but is it actually nice to use?

Same with JQuery. There's a lot of rose colored glasses these days for JQuery, imo. It is not fun to write.

Also the complexity lift to use React versus using JQuery is minor compared to the complexity lift distributed K8s microservices have over an NGinx server with a custom bash script.

The extra complexity of the scalable microservices architecture might be sufficient to actually sink your startup. The extra complexity of using React over JQuery costs... Essentially nothing.

Edit: I guess what I'm really saying is that choosing a different backend technology or frontend framework isn't remotely the same as choosing to build a scalable distributed microservice architecture over a static bare metal server.

Choosing a technology is a choice based on what you want to work with and what you think you can hire other people to work with.

Choosing K8s terraform distributed microservices for scalability early on is the most premature optimization you can possibly make. Get a customer before thinking about scaling.

A Python/Go/Kotlin/Java framework has the same amount of complexity as a PHP one (you are using one, right? It's not <?php tags all over the source, right?) and are nicer to use. (Not much nicer, mind you, PHP frameworks have evolved too, but still nicer.)

Isn't that what we expect from evolving technologies? (And well, if you are doing <?php all over, bare Python with print statements has the same complexity, with about the same downsides. Nobody will want to touch it, for very good reasons.)

jQuery is a very predictive mark for decades old leggacy cruft that nobody is allowed to improve. The modern equivalent is vanilla JS, that is simpler, about as powerful, and perfectly viable to hire for. There's actually no technical problem with jQuery, it just correlates very well with bad companies.

The infrastructure world has a real problem. A large part of it is that Puppet, Ansible and Chef are all badly managed centralized, complex pieces of software, and custom bash scripts can be anything. Also, cloud offerings tend to minimize on-call incidents, and companies normally have very bad policies about on-call time (kept secret until the candidate has no other option).

A purely community oriented ops tool (Apache style) would do wonders here. As would better labor laws (on the US mostly).

(And edit here, the trend on newer ecosystems, like Go, Rust and Haskell - and old language where the ecosystem is mostly new, is to make the backend frameworks simpler. So you get a nicer to use system with less complexity to stop you from solving issues.)

You really don't want to hire those engineers though? We really explicitly hire for judgement, and while those all might be a good judgement for the business some day, they aren't today, and an engineer who doesn't get that is way more of a liability for the business.

What I find more worrying is that incubators and VCs now often push it. One of a disturbing metric for a known incubator (cannot name them) is how fast you burn through your free $100k aws budget; if you don't do it fast enough, it is a metric for failure as far as they are concerned. And I see this a lot; this push to pump massive amounts in stuff that's not needed and probably never will be needed.

Good! Let the people who want to over-engineer their systems self-select into companies with over-engineered systems.

"The problem is that if a company advertises a job that is all about maintaing one or two monolithic applications without K8s, distributed system topics, tracing, microservices, AWS/GCP, etc., well, then not many engineers will apply. "

Also: if you are an engineer that has managed to run successful systems for years but didn't use microservices, K8s, AWS and all the other stuff you shouldn't even bother to apply for many jobs because you are "not up-to-date". Heck, reading the job ads from my company even they probably wouldn't hire me or a lot of my colleagues if we applied.

Neither Facebook nor Google are built on microservices in this fashion. Amazon is (but it’s on its own servers).

Doctor, doctor, every time I touch my left knee to my right pinky I get a stabbing pain in my heart!

Don’t worry, I know a physical therapy regime that will fix you right up: no medicines or scans required

Wonderful! what is it?

Every day before bed, don’t touch your left knee to your right pinky

Even without architecture astronautics you still run on a very deep stack of frameworks and abstractions that make troubleshooting difficult.

My turtle stack is pretty well documented and 95%+ of problems have already been solved by other people.

For everything else, I'm quite happy with https://sentry.io and https://www.skylight.io for my troubleshooting needs.

How do you know that?

Because even if you run bare metal on a raspberry pi you rely on fairly complex binary blobs. Most software is too complicated for such a simple stack.

You mean the Linux kernel and device drivers? Most of the time these things cause fewer problems than other things.

That seems a fairly stupid line to draw, to be honest. Do you know how the silicon your code runs on is made? Do you know how the fibreglass the CPU chip is stuck to is made? How do they get the copper to stick?

Yes, but the sooner you build to scale the better. More costly later on.

Is it? You trade retrofit costs later for maintenance costs and feature development friction every day until scale is needed.

You make a very good point.

Scalability is a two way street and those services don't scale small.

That makes business opportunities for more flexible engineers.

Chances are quite high you will never need scale. If you knew 100% sure you needed it when building, you would build for scale from day 1. But most/all startups and also for many projects in large corps, they don't know if they need scale. And startup statistics definitely show it is a very high chance you don't need it ever because you will be out of business long before that. It's mostly a waste of effort, time and money.

i recall reading ken thompson's reply to what he thought of virtual machines. something like "they are such a good idea that i wrote unix".

> astronautery

gonna borrow that phrase.

Surely "astronautics" is the right word though?

Reply on news.ycombinator.com