Back to the '70s with Serverless | Hacker News Reader

Back to the '70s with Serverless

evrl.com

518 pointspjmlp298 comments

One thing that surprised me as a latecomer to software development coming from a visual arts background is how much choice of technology and working practices are purely fashion driven. The thing about fashion is that the way if develops is largely arbitrary. Changes in fashion resemble a drunken walk through possible design space with the drunkard receiving regular shoves from "influencers" who are usually trying to sell you something. Occasionally you have a fashion "revival" where someone takes an idea from the past, gives it a new spin, and then sells it back to newcomers as the next big thing.

This seems especially true in the types of startups and companies many HN readers work at or aspire to join / build - that is ones which are low stake / high reward.

I think when you combine the low stakes nature of the VC driven startup world with its cult of youth and the in group conformity of young people this is what you get.

[1] by low stakes I mean no one will die and you won't be prosecuted if your single page app startup goes tits up. Indeed you're supposed to "fail fast" precisely because the cost of failure is so low. Even if a VC or angel has invested a few million in you, to them that's still low stakes because they exist on an entirely different plane of wealth and you are just one of multiple bets.

[2] We're going to rebel by all dressing the same but not the same as our dad!

I agree with pretty much your entire point.

In addition it sets up a situation of constant tension because many developers refuse to follow the fashion when they can (justifiably) say that the way they currently do things is perfectly fine.

The people on the edge of the fashion then regard them as absolute dinosaurs because they refuse to follow the trends.

Then there are people like me - I watch what is going on but I don't follow the fashions, I maintain a list of "that looks cool, check back in 18-24mths and see what they did" and then make a decision on whether it's worth spending more time actually looking at something.

https://i2.wp.com/www.business-to-you.com/wp-content/uploads... on that graph I'm a pragmatist.

Been bitten by the shiny a bit too much at this point in my career.

This. Devs nowadays seem to be just NPCs, they jump on every bandwagon, it's very depressing. But this is the same with rest of society, groupthink is becoming a real problem. Unwanted opinions, thoughts are scrubbed everywhere.

For a latecomer, you've managed to see through a lot of bullshit, and the psychological/sociological nature of it. What you're maybe missing is that's not always been like that. Cloud, SaaS is only a thing since around 2006 when AWS hit the scene big time. Before that, we had a relatively conservative, standards-driven perspective on progress. Maybe it's also the time when the generation having learned computing from the ground up (assembler, C, hardware hacking) was starting to fade from the job market, or rather the then-new devs became eager to carve out their niche with arbitrary abstractions (but not entirely - I remember OOP bullshit from early 1990s and WS-* SOA crap from 2000s).

The abstractionists, philosophers, theoreticists, had their days before too (60s-70s Lisp, 70s-80s Logo for teaching children). So in fact the practicalists would only have ruled in and around the 90s.

If we add all the "everything is a XXX" languages (XXX in [list, string, object, function ...]) and gather all simplificationists together, then they almost always were there, with fluctuant success at being prominent.

A lot of those "abstractionists" you mention were very heavily practical, too.

1980s and 1990s gave us "decision made by senior management reading shiny magazine during flight", we just look back with sentiment now combined with survivor bias.

What I find frustrating is that if you are in a high stake environment it’s increasingly fashion driven too. I am in one of those environments where we can get prosecuted, big time. I tend to opt for a conservative engineering approach with maturity and security considerations being a key part of the design process.

But no, fuck that, someone went to a conference, bought everything shiny and started a death-march of throwing any old hacked up shit and containers from random joes in a limping kubernetes cluster that no one really understand and calling it a success.

That’s reality in 2020 unfortunately. I don’t sleep very well these days.

I dont understand why business leaders cannot see this for what it is and take back some control. Investors need to start asking more probing questions about how organizations develop and deploy applications. Perhaps if the money becomes contingent, people will start to give a shit about the engineering quality.

At no point should "fun" be a line item when determining what technology to select in a high stake environment.

We are in finance and we picked the most boring technologies we could find. Most of our stuff doesn't even talk to the network stack. I can't imagine running our transactions, with their piping hot PII, through the labyrinthine monstrosity that is modern "best practices".

Problem with investors is that they pay third parties to do audits and third parties are filled with box tickers who ask a load of questions. You can strategically answer any question. So the game becomes how to answer the audit questions effectively rather than an honest appraisal of the situation.

Also it's not really fun. It's a frustration ridden shit show. You spend all day solving complex problems instead of building business value.

A fine example of where the tradeoff ends is spending three weeks solving fundamental networking issues due to stack complexity and immaturity.

> You can strategically answer any question.

Good luck with that.

We do exactly that for a living here, we're essentially 5 people who are all CTO grade and have run our own companies. If you can fool us for a whole day you've deserved your investment, but I highly doubt anybody ever got away with that.

There’s at least two layers of indirection and yes I probably could fool you or just blatantly lie.

Good for you, it's been tried but so far without success. Maybe you're the exception but I highly doubt that. Arrogance on my part, for sure, but the record on my end speaks for itself (170+ companies looked at to date) and you are an AC.

My house was paid for by someone who was that arrogant :)

Oh, so you're a fraudster. No wonder you keep your account anonymous here. Well, good for you. I prefer to go the long way though.

Absolutely not. I was merely an engineer telling truth to the management staff. This taught me a rather large lesson that it’s more about the person’s integrity passing the facts on than the convenience of the facts or the reality of the facts themselves.

I hold myself to a far higher standard than my former employers.

Then your comment above doesn't make any sense.

Companies are lying to each other and to customers on a regular basis. Most of those lies are framed in a certain way and are not fraud from a legal point of view (the one where you go to jail).

The context we're talking about here: acquisition or investment there is a duty to research and a duty to inform, the two should theoretically meet in the middle, there is always some give and take but that's the basic idea. Failing your duty to inform is very close to fraud and in many cases crosses right over into fraud. This can get you in very hot water depending on the jurisdiction and the willingness of the counterparty to sue. Typically such cases are settled out of court, the settlements are usually not small and could easily cost you control of the company.

Most parties are aware of this and will do their very best to avoid going down that road, the few idiots that don't usually end up in a bad way though some may get lucky.

>"Failing your duty to inform is very close to fraud and in many cases crosses right over into fraud."

Cutting down the fluff you've just confirmed what I said. Basically when it does not cross into legal criminal fraud as defined by law you can't call it a fraud. Same as with taxes, especially corporate. One can be and idiot, creative, fraudster. The middle one is subject to interpretation and the amount of money involved. "Doing very best" does not necessarily mean not lying.

I can call it whatever I want. Whether a judge will agree with that is another matter, but the common layperson's definition of fraud is not the same as the legal one that could lead to a criminal conviction, if only because they both have a different standard for evidence. Splitting legal hairs is what may make you decide to pursue a fraud case or letting one go, doesn't mean that it wasn't fraud.

You can see this happen every day, plenty of cases that are fairly obviously - to a layperson - fraud end up not being prosecuted because it is hard to prove the facts and then there is the matter of intent. Still, nobody is fooled.

Because all the shiny new things make promises. From larger talent pools in node.js, to cheaper scaling with kubernetes. They all talk to business leaders in terms they understand.

Also, this is a fundamental property of innovation, innovation usually is made possible by the new and shiny things. I run the software side of a hardware startup, and I wouldn't have been able to run such a complex software system with such a small team if it wasn't for all the shiny stuff. Yes a lot of it is wobbly, but we're also comfortably a year ahead of the established players.

> From larger talent pools in node.js, to cheaper scaling with kubernetes.

Larger pools of people, and cheaper infrastructure.

Not necessarily talent and cheaper scaling in my opinion.

> Yes a lot of it is wobbly, but we're also comfortably a year ahead of the established players.

It's likely that you would be ahead of established players with old, not shiny technology as well. Established players have an inherently harder time progressing in most circumstances and an entirely different maintenance burden in pretty much every layer of their product. Shiny, new technology accounts for much less than people think it does.

> Investors need to start asking more probing questions about how organizations develop and deploy applications.

Does this even matter though? Shitty technology choices don't always translate to poor business outcomes.

You are totally correct. Beyond a certain level of due diligence it doesn’t matter.

This is absolutely true. But shitty technology tends to have hard upper limits to scalability and shitty security tends to put hard upper limits on the company life span. It's interesting how established companies can have one security issue after another but the customers are so locked in they can't leave and so those companies are not nearly as affected as they should be (Equifax anybody?). But your run-of-the-mill start-up would be mortally wounded by such an affair because they still need to sign the bulk of their customers.

Of course it doesn't always play out like that (to every rule there are plenty of exceptions), but investors don't like to be associated with companies that fail or that blow up their reputations either because it makes their exit that much harder or even impossible.

I've seen investors that took the responsible route and made fixing major security issues or other undesirable elements of the target companies a condition to investment and typically these investments succeed in the longer term, not in the least because they cleaned up their act and could then turn that into a USP relative to the rest of the field.

Even so: better invest in a company with a mediocre product and a stellar sales team than to invest in a company with a fantastic product and a mediocre sales team. The first one will get you a better ROI, and tech can be fixed post deal.

What company has been harmed by shitty security on any sort of worrying timescale?

I worked for an online gambling startup that got owned on launch day and the investors pulled out the day after.

Thank fuck I was just a contractor and not a founder or shareholder.

“ Investors need to start asking more probing questions about how organizations develop and deploy applications. Perhaps if the money becomes contingent, people will start to give a shit about the engineering quality”

Why? Most companies fail due to a lack of product market fit not because they picked the wrong software stack. Hence investors time is better spent on company outcomes and not micromanaging whether the backend team is building using Java or Rust or whatever’s trendy.

I see this idea often, and it feels somewhat shortsighted for me. Company outcomes are primarily determined by product-market fit, yes, and: the tighter your feature release feedback loop, the more product-market-fit experiments you can run before you’re out of runway.

Many technology and process choices are just style and preference, absolutely. Others have a material impact on the effort and time to ship a feature. (I’ll leave examples as an exercise to the reader; I imagine any example I used could turn into discussion of “ah you’re doing $FOO wrong.” I suspect this tendency entangles with why these choices seemingly frequently impact business outcomes, from my perspective.)

Well I imagine it’s a function of the stage of companies the VC’s are investing in.

If you are an early stage investor and you are writing lots of checks to cast a wide net do you really have the time to do that level of due diligence? Presumably a quality founding team and a decent demo is enough to tick that box.

If you are a series A or above investor, does it really do anything to provide this advise after the fact?

I image that the assumption is that if the company is successful you can afford to bring in more seasoned people to fix things. At least that’s what I’ve seen on my travels in the Bay Area.

True story: last "AWS Solutions Architect" I met told me "AWS is a sect and I am part of it", I'm still not convinced it was meant as a joke!

There are a lot of religious followers of the one true AWS way of thinking. I dared bitch too loudly about SES delivery policy once and said to look elsewhere and earned the ire of the architectural elite.

This is insane because I'm certified too but a non believer, a concept which does not compute for a lot of people.

You don’t believe in AWS? Or not in SES, because I feel like they still win in general ‘want to recommend’.

I don't believe in anything past 100% portable IaaS platforms. Within those bounds, careful application of AWS is fine.

This is something that annoys me, exhausts or worse. The amount of time wasted on bikeshedding and chapel wars is draining. I appreciate craft a little more because tooling and practices are more stable. You focus on work and not how to work.

> One thing that surprised me as a latecomer to software development coming from a visual arts background is how much choice of technology and working practices are purely fashion driven.

It's not “purely fashion driven”, it's just that (1) as solutions radiate out from their point of origin understanding of the problem, how the solution addresses it, and what caveats the user should be aware of gets (on average) foggier, and definitely more unequally distributed, and (2) there are real network effects in software development where being popular genuinely makes things better, all other things being equal (from a business perspective, a technology for which you can get people that are proficient and comfortable is better than one you can't, other factors being equal.)

> a technology for which you can get people that are proficient and comfortable is better than one you can't,

Hence why PHP is still the leading language for eCommerce, where fashion gives way to tight margins and hiring talent for a decade long lived piece of software is a major concern.

When cost is the driving factor for most decisions, and ROI is a very real and measured metric, convincing someome to rewrite their money making machine to keep up with the latest in tech is a hard sell.

I got some downvotes for this, but look at the numbers. I'm not saying it's the best language for eCommerce, but it is the workhorse. It's the old beige cash register in the corner bodega that's still the heart of thousands of businesses. These businesses don't have the margins to replace their Magento or WooCommerce shop, hence the need for pragmatism. If you're thinking about Shopify or AWS this or that you are already thinking about the wrong part of the industry.

> Hence why PHP is still the leading language for eCommerce, where fashion gives way to tight margins and hiring talent for a decade long lived piece of software is a major concern.

I don't think the forces that resemble fashion are any less applicable to e-commerce.

> When cost is the driving factor for most decisions, and ROI is a very real and measured metric, convincing someome to rewrite their money making machine to keep up with the latest in tech is a hard sell.

Convincing people to rewrite systems without (and even often with) changing needs that the old system seems to brittle to accommodate is hard everywhere, even when ROI isn't clearly measurable; that's why so much established software in government (as well as other places) still runs on COBOL.

The shifting preferences in development that look like fashion are evident, even in those domains where there is lots of established software that is sticky, in greenfield development.

> The shifting preferences in development that look like fashion are evident, even in those domains where there is lots of established software that is sticky, in greenfield development.

Definitely agree, greenfields projects are where you get an opportunity to suggest new technology.

What I find in eCommerce though is the people who make the decisions on this stuff move around from company to company like pollinating bees, and they always want what they had before, which is often WooCommerce. It's a matter of user interface and ecosystem. They don't care about the merits of the technology underneath, it's truly not a concern for them, what they want is to be able to load up the Yoast SEO plugin and check their copywriters work, or have their in house guy make popups with whatever popup maker they had last time.

I think that's why it's so hard to kill it off.

Keen observation. I enjoyed Halt and Catch Fire because of its parallels to today. A lot of previous startups had good ideas just the tech wasn’t quite right or v1 at the time.

As far as technology goes, I think what we describe as "fashion" is often mostly caused by a desire for safety in numbers, and to a large extent that's rational.

If I pick the same language or framework or whatever as everyone else, there's a better chance it won't be abandoned, and when I want to integrate with some other project, someone else will already have done a great deal of the work.

Too true. On my job we have frameworks from the Architects, and stuff that drips in from outside.

The Architects typically buy some Framework from a Trustworthy Enterprise, as approved by the Gods of Gartner. Quality varies wildly, but if you move half a mm from the beaten path you're on your own. I gain a thumbleweed badge on stackoverflow every time I deal with their stuff.

Compare it to outside world stuff. Quality also varies wildly, but 5 minutes on Google generally delivers someone who already suffered trough hell for me, and forged some unholy pact with the technology in question.

Unfortunately, neither camp can choose a boring, proven, working technology that exists already a few years and is in common usage.

If you found yourself using a boring, proven, working technology ...how vividly would you remember that experience?

You would have a feeling of trust.

As an example: I click on a browser icon and I get a working browser with a predictable behaviour.

One day, chromium edge took over, messed up a ton of settings,and kept on blathering about my experience. 5 minutes of filling out a work sheet changed to 30 minutes of whack a browser.

My amicable brotherly feelings of good that day did not extend to the average Microsoft employee.

My experience using PHP at my previous company stands out because it was so relatively glorious.

By the same token, people should stop expecting exceptional results if they used the same tools and processes as everyone else.

It's a fine line to use recent frameworks and at the same time to be productive and actually solve problems. That's ultimately why good frameworks are developed. When React came out many called it a fad but it solved really nasty problems in Web development, that's why it stayed. Also I'm quite sure K8S is here to stay because it removed this whole VM complexity layer and makes integration and testing of services a breeze. On the other hand there are fancy frameworks that don't solve such problems or the understanding is not good enough (yet) to solve them reliably. IMHO a lot of web frameworks that promised in the past to remove the boundary of frontend and backend suffered from that. Also the early NodeJS ecosystem to some degree.

It's fun to use a framework, get to it's boundary preventing you from getting things done and then discover a framework that actually crosses that boundary. Speaking of server-less, I never came into the situation where it would have made things easier for me so I never used it but YMMV...

> Also I'm quite sure K8S is here to stay because it removed this whole VM complexity layer and makes integration and testing of services a breeze.

Kubernetes does basically nothing to assist in integration or testing of services, and adds more complexity than the “VM complexity layer” ever had.

I don't think so. In the average company running VMs themselves, the person managing the VM is a different person that runs stuff on it. Even in < 5 people tech teams. And Enterprise-grade VM management software is completely out-of-scope for any dev person for that matter. Chances are I will never even know any details about the real hardware it runs on.

K8S is a different story, it can be managed by a devops team (or a really engaged dev ;)). VMs virtualize your network interface and can take care of some IP routing, K8S can route HTTP traffic and TLS-terminate it. It facilitates being aware of other services and multiple instances. Basically it's possible to get rid of nginx. And most importantly it runs the containerized applications basically on bare metal. Spinning up a test instance and destroying it 1 minute later is cheap, any dev can do it.

I'm going to play devil's advocate here. A sleepy engineer wakes from their slumber when a new technology takes hold their mind. They find new ideas exciting and intoxicating. And sometimes that energy can be infectious and makes the entire development process more enjoyable for product owners and QA.

There needs to be a balance. For example, upgrade to the latest version of Rails. Upgrade previous file storage gems to active storage. Upgrade the deployment process.

There are many exciting things to improve in a monolith.

It's an astute observation, especially if you have not heard Alan Kay's thoughts on the subject. He calls our industry a Cargo Cult and explicitly compares it to fashion and pop culture.

Keeping with the anology I'm sure there are software engineering equivalents to Jeans and T-Shirts and it would be an interesting study to compare organizations that opt for these fashions to those that do not.

If you're reading PG, it seems like a great topic for an essay.

It's not the cost of failure that is so low, it is the cost of hanging on but not ever taking off that is so high.

And just like fashion, what's "in" is driven by marketing.

We don’t teach people a technical aesthetic. So, lacking a strong internal sense of taste, they cargo cult it from the latest FAANG scraps thrown at them. This compounds with the conceptual disaster that is web apps.

The good news is there’s a lot more to software dev than webdev.

As a software developer since my early days, I wholeheartedly agree with you. It's honestly ridiculous to see how people invent "revolutionary new ways" of doing the exact same thing that could be done in a dozen ways already. And they invent those by piling more and more unnecessary complexity on top of existing technology stacks.

Electron is a revolutionary new way of making desktop apps. Except they look like they're something else and take up all the RAM you have and ask for more. But nah, you don't understand, it's the future, native desktop apps are dead.

Kotlin is a revolutionary new JVM language that will solve all the problems you had with Java... Except I haven't had any. Java is simple and predictable. Kotlin is not. To me, Kotlin feels like an abstraction layer that gets in the way. But people will be defending it relentlessly because that's what Google says is The Future™ for Android development.

Web development... That's just nasty. "Revolutions" happen every day, so now your news article takes 5 seconds to load on a 100-megabit connection just because someone absolutely had to make it a react app instead of a server-rendered page with just text and maybe some pictures.

This overall trend of abstracting the platform (the operating system, the web browser) away is abysmal. It does no good. And lowering the barrier to entry to programming? Honestly, I'd rather prefer it to remain high. Programming is meant to be an engineering job, not the kind where you throw stuff at the wall and see what sticks and just assume that browsers and operating systems are made of unicorn jizz.

Half the threads on hn mention that software engineering was never engineering, so which is it? Or maybe you're the exception to the rule?

Sarcasm aside, that ship has sailed You can't have internet as widespread as it is today while keeping programming in the hands of the very few "true engineers". So either you're arguing the past 20 years of internet progress was a mistake (fair enough point of view) or you're deluding yourself.

I'm not sure I want the internet to be as widespread as it is today. I certainly don't want it as commercialized and as centralized as it is today.

I also take issue with this whole dumbing down of technology that's been happening over at least the last 10 years and shows no signs of stopping. It's understandable that there are certain business entities that would prefer to be in possession of as much money as technically possible, and to achieve that, they need to give their products into as many hands as possible, which means said products have to be "easy to use", aka don't require reading manuals and understanding what the damn thing is actually doing.

Computers and other computer-like devices like smartphones are, ultimately, tools. As are the programs they run. And tools are meant to serve and empower their end user. 99% of the IT industry, unfortunately, would rather like them not, because that clashes with their own interests they for some reason are allowed by the society to have. That's all I have to say.

You made some bold statements without any details.

Kotlin to me solves most of my Java pain points. In what sense is it not predictable?

Electron allows a company with web developers to be able to build cross platform desktop clients without hiring desktop developers. That's incredibly compelling and powerful regardless of the compromises with the final product for the end user. (which are not guaranteed. Something like VSCode is indistinguishable from a rich desktop client)

Sadly yes, it's become a very fashion-driven industry.

Much of it is driven by the near-complete lack of historical knowledge of the field. I advocate that a computer science or software eng curriculum strongly needs a semester or two of the history of the practice. What roads have been taken in the most important areas and examine all the dead ends and why they were abandoned.

So much of the new-shiny is just a badly implemented rehash of ideas that we already know didn't work out in the past, if we only knew. Because we mostly don't, as an industry, it seems like an exciting new idea.

> So what have we now? A “mono repo” codebase, because clearly a Git repository per function in your system would be too much, a large deployment descriptor per fine-grained component, which Spring maybe called “Controller” but is now called “Function”, and instead of combining them all on your desktop, you send them off to someone else’s mega-mainframe [AKA "the cloud"]. You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow. At least we’re not having to walk to the printer on every try, but that pretty much sums up the progress of the last half century. Oh, and a “Function” will be able to handle a single request concurrently, so here’s your AWS hosting bill, we needed a lot of instances, I hope you won’t have a heart attack. Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise.

I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.

Surely others here on HN have run into similarly minded individuals too. They're not hard to find.

Plus ça change, plus c'est la même chose.

Everyone's chasing "scalable", mostly for no reason at all.

There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place. A few minutes per month is the most devops any startup would need to do in reality running their own VMs or servers, after an hour or two of setup. And that hour or two is often less than faffing around getting all the labyrinth moving parts of AWS/Azure/etc. working.

And the whole thing becomes a transparent farce when you look at providers like Azure, where their own admin interface is so slow and prone to random breaks you have to ask:

How is this scalable?

It's clearly not scalable for themselves, so how is it "scalable" for everyone else?

There are benefits, mainly for me I'll use them for hassle free deploys[1], but the whole thing sometimes feels like a massive con to sell the performance of a potato by advertising it as a jet engine.

[1] Like a month back I deployed a demo prototype web app I wrote in 3 hours onto Azure straight from github with like two clicks, and got the contract, can't really beat that.

Makes me think of those old "Webscale" skits on YouTube.

https://www.youtube.com/watch?v=b2F-DItXtZs

https://www.youtube.com/watch?v=bzkRVzciAZg

> A few minutes per month is the most devops any startup would need to do in reality running their own VMs or servers, after an hour or two of setup.

Right up until something crashes or some hardware fails.

> Right up until something crashes or some hardware fails.

Unlike the public cloud where Nothing Ever Goes Wrong (tm).

There are never brownouts.

There are never latency spikes.

No undocumented quota limits.

Documented but confusing quota limits.

Unexpected performance issues caused by other customers.

Forced upgrades.

Forced patching.

Shock bills.

No siree Bob, that kind of thing just doesn't go on in the public cloud! It's all rainbows and unicorns....

You have all that DIY too, except maybe shock bills, but it's your problem to fix. Like with everything, extremes are bad and all managed or all unmanaged are both bad. My personal guideline is use off-the-shelf services if they are cheaper than DIY, including labor, or if DIY provides no competitive advantage and off-the-shelf is cheap enough. But if DIY is much cheaper or gives a competitive advantage, do it yourself.

I fail to see how this supports your claim that you can get by with "a few minutes per month" of devops. In fact, it's a classic red herring.

Your original claim is quite unrealistic and you'd do well to retract it. It's bad advice both for public cloud and for self-managed efforts alike.

When I used to work for an MSP, we would sell Dell servers to rural banking groups.

We supported disaster recovery and high availability deployments.

Any failed component, Dell would send us a replacement in less than 24 hours.

24? I guess it's that long because it is rural? We use Dell servers and I sleep a bit better having seen their 4hr support replace a motherboard with a bad lifecycle controller within 2 hours of the initial call.

Try calling Amazon and getting that kind of response for your problems.

Yeah, my closest Target store and major airport is over 2 hours away.

But, yeah, that is worse-case scenario.

But it is a few minutes per month. A properly architected application won’t depend on a single server so a hardware failure will be as unimportant as it is the serverless world.

Architecting a system properly is one of the key functions of devops. You don’t get it by investing a few minutes per month, whether public cloud or bare metal.

I always have standby copy deployed elsewhere. And I make it a point not deploying production until a single script could rebuild new clean system by using backed up data, configs and software packages. So for me it is no a big deal at all. Well if the hardware died I have to pay for replacement which is not fun but it is a very rare occurrence.

You do not notice that either with the right setup. I also rather hire a good SA and pay 10-100x less monthly (for which I can hire an army of devops here in the eu by the way) with the risk of all hardware dying at the same time (which does not happen, not now, not in the 90s when I started hosting).

Sure serverless etc has it's upsides and in that case I do use it, I just encounter close to 0 companies per year that need it or benefit from it. Aws ways benefits.

The choice is not only serverless or hosting on bare metal - there are a lot of options in between. You can hire VMs and have the vendor transparently deal with hardware faults for example without downtime. Running your own VMs doesn’t mean running your own hardware.

The original claim is not at all unrealistic, I’ve been running a small shop with about 20 VMs without issues for years, uptime seems to me about comparable with serverless or hosted services from big players who regularly have small outages and there really is very little work involved.

Its like a well stocked grocery store with thousands of brands, I think the biggest draw of "the cloud" is the illusion of infinity plenty, all you need to do is press a button (even though the reality really doesn't measure up, at least not in ways people naively assume)

> There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place.

I dunno, the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me.

Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.

Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it

> Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.

Not using serverless doesn’t mean there can’t be a CICD pipeline.

Yeah of course. I’m not saying that serverless is the only ci-cd solution, many businesses build ci-cd pipelines on top of serverless apis. However, most serverless platforms take care of a lot of the deployment process out of the box, which would have been enough for the business I was describing in my comment

You don't even need serverless for those things. Any of the automated deployment/orchestration systems in the last ten years solve those problem (ansible, terraform, etc).

Yeah it’s true but it requires a lot more config. The job I mentioned in my comment involved thousands of lines of anisble to perform operations you wouldn’t even need to consider/could perform in a few clicks when using a serverless platform.

Don’t get me wrong—serverless platforms are not a ci-cd silver bullet. Many organizations build pipelines on top of serverless deployment apis, but it certainly does take care of a lot of things for you—not to mention other features mentioned elsewhere in my comments

You're supposed to encapsulate those in reusable roles, see an example methodology here: https://yourlabs.org/posts/2020-02-08-bigsudo-extreme-devops...

And since we have containers it's become really much easier actually ... Deploy a Traefik container which will watch dockerd socket and self-configure HTTPS and all when your other script starts container. Really, it's much easier than back in the days: https://github.com/jpic/bashworks/tree/master/vps

Wasn’t the original post complaining about avoiding a gazillion lines of config in the first place ?

Lines of config are like lines of code, refactoring them regularity is basic hygiene like brushing your teeth. You can "have no config lines" if you go ahead and click your config but then you don't have "infra as code" anymore so that's also "going back to the 70's" I guess.

All you needed was PaaS, which has existed for a long time. Or the even older name: managed services, offered by pretty much any decent hosting provider for decades.

I remember using companies like MediaTemplate and FullControl 10 years ago for clients and not having to worry about OS updates.

Serverless is a PaaS. And yeah, those services could work too. Serverless isn’t the only solution that works.

I don’t think your past experience reflects current practice.

My reality at a 20 person shop with a few devs is much closer to the OP - a few hours setup (we save the cloud config and can remake servers in a few minutes), servers run for months without intervention, uptime is on par with cloud services, deploys take seconds. Managing servers or deploys just are not major problems for us.

Serverless is not a solution to deploy problems - mostly because they’re just not significant problems for us, if anything the dev deploy story is worse than renting VMs. It’s an interesting take on serving requests and I do believe some variant of it might be the future (a variant more like cloudflare workers IMO), but at present there are serious drawbacks and it is not clearly better, particularly when provided by a predatory company like Amazon.

When your infra is in AWS, you need an "AWS Certified DevOps"[0] instead of a "typical DevOps", I mean, the typical OSS hacker - which is kind of sad really, and makes me wonder: is this how we're going to build an Idiocracy[1] ? Anyway, things like backups, monitoring, alerting, debugging, is going to be a requirement anyway, and for me it's harder when locked in proprietary stuff such as AWS, I'd rather have OSS I have freedom to hack ;)

[0] https://aws.amazon.com/certification/certified-devops-engine... [1] https://en.wikipedia.org/wiki/Idiocracy#Plot

Yes, "the cloud" is bunk from any technical point of view.

It exists to shift CapEx to OpEx for accounting. Any technical considerations are a distant third place on the list.

That can happen without serverless - renting VMs is OpEx enough.

Yes, if you manage to shift the responsibility of maintaining these VMs on the developer positions you already have.

Leasing programs did that well before the cloud ever came along

"The cloud" is a leasing program except even more optimized for OpEx - pay-as-you-go monthly billing and you now don't even need to hire ops people because it's "serverless".

(Of course you have to force your devs to do the ops stuff now, but that's not accounting and HR's problem now.)

Well, if you mean CapEx because one needs fewer programmers and a higher AWS bill, then yes I agree with you. But if you mean CapEx because one needs fewer physical computers that they own and a higher AWS bill for computers that they rent, then no I do not agree with you.

First, the markup in just sheer ops terms is enormous. Minimum 4x, and in some cases 20x. Sure, you get a couple more nines of reliability, but 99% of startups don't need that much reliability.

Second, in countries like Canada where capital expenditure on computer systems can be heavily marked down[0] people still use AWS and DigitalOcean by default.

The main reason people don't do colocating is that it is a waste of time better spent. DigitalOcean is cheap enough. AWS is cheap enough and if the team doesn't have experience running Postgres, fine, they use RDS. Sometimes it's just sheer laziness at first that later blossomed into "holy shit, we're growing too fast to even think about migrating off of these giant AWS bills" which was essentially Dropbox's story if I recall correctly.

[0] 55% per year, and in practice they allow full expensing in the first year for petty amounts of outlay. For example, if your business is a data centre, sure it's 55% per year, but if you run a flower farm and fully expense a new laptop for sales the CRA is generally fine with it.

I'd guess that the vast majority of "cloud" customers are boring enterprises doing boring enterprise stuff, probably not even Internet-related stuff at all.

Does anyone know of a resource showing me some parameters for estimating how far different managed servers reach? How many concurrent users, bandwidth, etc.? Have a hard time getting a mapping for expected concurrent users -> hardware requirements. How quickly can you adjust the HW resources in a non-cloud setup if your startup product is more successful than expected?

Impossible question, it depends what your app does and what your users do. Even within the same company, 1000 concurrent users of Google Maps will use completely different resources than 1000 concurrent users of gmail. It'll even change over time for the same app and same users, as their behaviour or the world around them changes.

Scalability is just one reason for using serverless. There are many others. Most of which have to do with being able to focus on the part of your code that actually does something instead of on infrastructure and integration. For example, not having to configure and maintain VMs, containers, or Kubernetes, off-loading a part of security onto the cloud provider, and built-in integration with supporting services like logging and metrics.

Serverless, like every design decision, comes with its own set of trade-offs. Whether those trade-offs make sense for your environment is something you have to determine. At my company we have seen a productivity increase and shorter development times using serverless. Of course, that was after the learning curve which is certainly not trivial.

> by not having "devops"

Apparently, you don't even know what that word means. You can not "have" DevOps. It's a methodology and not a thing. Besides, DevOps means exactly the opposite of what you are implying: it is the practice of not having a dedicated operations team and letting devs handle as much of the operations for their own applications as they can.

> few minutes per month

I can tell you my team spends a good 30% of its time on operations. Some of that (for example adminstrating K8S clusters) could be offloaded to cloud providers for money.

I sometimes think of how far we still are from utility-quality computing services. Every company needs electricity with near-perfect uptime, but almost no company has a full-time electrician in-house.

> Everyone's chasing "scalable", mostly for no reason at all.

No technical reason, maybe. There is a social reason though: if you point out that there is no need to build scale (yet), you are pointing out that the business will not achieve overnight success. Even if it's true, it kills excitement and is therefore not something that people want to hear. Who wants to realize that what they're spending large amounts of time on (and possibly even neglecting other things like health, sleep and social contacts) will not even make all their financial dreams come true? Much easier to not think too hard about it and keep on doing what everyone is doing.

So, building out scalable infra is signaling that you too believe in the company and are preparing for the moment (any time now!) that "webscale" is needed. It's showing that you are part of the team.

I think what’s important to remember is that these kinds of “truth” are not the full story.

Yes, a technology probably won’t live up to its marketing.

Yes, what works is probably good enough.

Yes, changing has switching costs and sacrifices institutional knowledge.

Yes, we’ve seen these technologies before and switched away from them from various reasons.

That’s all fine and sometimes frustrating as an engineer. But it’s not the only reason decisions are made.

Sometimes pitching your exec team on The New Hotness gives cover for changes that actually do need to happen.

Sometimes adopting The New Hotness is marketing to recruit engineers who want to try something new.

Sometimes adopting The New Hotness is because some people are bored and uninspired in their jobs and just want to believe in something good again.

The success of things like MongoDB and blockchain are not because they are better or even very good at most things people use them for. They’ll never live up to the hype. But people need things to get excited about. It’s psychological. Attacking them purely on technical points can lead one to miss the actual point.

I don't think burning in sum hundreds of billions of Dollars (or even more) because "it's OK being irrational, because emotions" makes sense.

The business case is that you won't be able to retain good engineers if you refuse to rewrite your AS400 COBOL app.

I’ve wondered whether these good engineers are really a net benefit. They crank out something spiffy, move your big systems to it, and then disappear to work on the next hotness, leaving an already burdened tech team to support yet one more spiffy stack that is not quite as reliable or easy to use as we would like.

I think you need to either talk to your system architecture function or get a system architecture function in your organisation.

Managing this stuff is a job, and it requires discipline and organisation. It's one of the few functions of an IT department that is left once you go to cloud (as dealing with Oracle's bullshit is then gone). The younglin's doing development in the business functions have to play by the rules - and the business functions need to agree that there are rules or they will :

a) spend much monies on the issue you describe

b) get cracked open like fresh eggs by any passing hacker who is keen on sports cars

I'm not sure I would call people that would rewrite something just because it gets boring otherwise as "engineers". I think the proper term would be "children".

Yet the best engineers in the field aren’t working on legacy banking apps. The problem is that once a product is put into production and the hard problems are solved, the engineering becomes significantly less challenging (unless you’re in an ultra rare scenario of exponential growth).

The problem we have in software is that we have engineers build bridges and then somehow expect them to stay around to paint the thing, change light bulbs, etc and stay fulfilled. A person who is an expert at making new things won’t like being in maintenance. It’s that simple.

Sure, that's absolutely true. But it doesn't mean that you need to rebuild already working stuff over and over.

It's much more fun to improve the status quo by inventing something nice as to repeat the work of the past generation (only painting things in a different color).

Thanks for putting this into words. People want a reason to log on, a reason to show up to work.

I wrote a blog post on Monday about a relatively old and not-hot language. The post nevertheless trended on Medium and received 50k views. A lot of the feedback was from people who were just happy to see someone senior say something positive about the not-hot language.

Anyway, working with The New Hotness also gives people something to put on their resumé: “I built system X on top of The New Hotness. Nobody had built such a system before for The New Hotness”.

Is being paid hundreds of thousands of dollars plus stocks and bonuses not enough of a reason to show up at work?

You would think so, but as it turns out, it’s not. First world problems^2.

Not when that becomes table stakes, no. If you have five different companies offering you that kind of money, eventually you'll start thinking about more than just the money.

Well the funny thing is that once you crack FAANG interviews you can usually get offers from multiple of them and then money no longer becomes the decision factor. The choice becomes, “work on X and make $400k or work on Y and make $400k”. You bet the tie breaker often ends up being the prospects for working on cool shit.

I suspect the majority of readers are being paid less than $100k with no stock or bonuses, and correctly intuit that new-hotness things on their résumé will help them climb the salary ladder towards that magic $100k threshold.

I think it's sad that the vast majority of people in our field (at least in my experience) is so damn excited about building new stuff all the time. There's no appreciation for maintaining, optimizing, and improving existing systems. There's only talk of "adding value" but very little of "keeping value."

So people keep moving on to the next project, leaving the old ones running live but to rot, and at best every once in a while someone will be assigned the dreadful task of having to figure out why the "legacy" (even though it's still fully in operation and servicing customers) system had an outage and can't recover.

Our field suffers from a serious lack of focus. People don't learn to work under constraints. Instead, they prefer to build a whole new thing, full of complexity and hidden issues, just to get around a few nuisances that, with the right mindset, actually force one to be creative (which should make work fun).

I've recently commented here on HN about how I feel like I'm outside the norm on these things [0]. I was happy to learn I'm not alone. I do think those of us who don't mind working on "legacy" systems and perhaps with limited tools are the minority though.

Sometimes it's not even about working on old systems. You work with current tech, build something, and as soon as it's barely deployable, you're expected to move on to the next thing. "It's a great opportunity for your career," they say, "you're gonna get to build something new and make key decisions."

Why does it have to be decided for me? It's my current predicament. I'm working on a project that took me two months to get in the right mindset for and get all the context I needed. I had to try things out, sort out ideas, until I got a clear picture of what to build. Now I've built the first iteration of it. It works, and I'm super excited about it. But it's not done by any means. It needs more features. It needs to be polished so it doesn't make operations miserable running the thing. And yet... I have to move on, and hand it over to someone who doesn't have the same excitement for it that I have, and has maybe 10% of the context.

All because building new things (and adopting The New Hotness, as you say) is what "everyone" wants.

[0] https://news.ycombinator.com/item?id=25404039

It doesn't help if your career path in a company is tied towards 'visibility,' where certain roles will never see a promotion no matter how critical they are.

I've chosen to take on more than one heavy maintenance job, no client facing visibility in terms of fancy new features. Feedback in performance reviews would always be "it's hard for people to see what you're doing." In the end you have to find your own way to game the system to get appropriately recognised, but of course that kind of office politics isn't really what you want to spend time either.

I'm doing more 'new feature' style work now but I'm happy to not be tumbling through the waves of hype. We get a little bit of new hotness, because of course you need to embrace some aspect of that to avoid stagnating and getting too stuck in your ways, but not so much of it that the engineering team is incentivised to churn through half-assed features.

It's great for those engineers that they are so far up Maslow's hierarchy that they're willing to bring significant risk to their employers in the pursuit of self actualization. However, there are a lot of engineers in the world (outside SF and NYC) that are still concerned about lower levels like physiological and safety needs. I wonder if the world getting more accustomed to remote work will change the cost benefit analysis on hiring those engineers instead.

>"Invariably, their response has been to tell me that I just don't get it"

I cured this one when my native server running on not so expensive multicore CPU with enough RAM left their cloudy setup in a dust performing more than a 100 times faster and still having huge potential reserve in vertical scalability. They were just astonished at what a single piece of modern hardware in combination with the native code could do.

We need to meme c++ back into popularity

I do not think there is any real need. To me it is doing just fine. It does not have to beat every other language into submission.

Congratulations, from now on you are on call 24/7 in case anything happens to that machine. It is now your own pet. When RAM fails in unpredictable random ways, you'll spend hours reading the syslogs. When your SSDs fail, you'll spend hours ordering new ones, setting raid up and migrating the backups you've had to provision, automate, supervise and secure yourself.

When the server gets even remotely suspected of having been hacked into, you must trash it overnight and never use any of it again.

You talk like using a physical server somehow prevents building a highly available system or from using automation. Why would the hardware be any more of a pet than any random VM in the cloud? You could also just run a hypervisor on the host and get the benefits of virtualization along with full control of hardware.

Sure, if a drive breaks you'd probably want to replace it instead of just replacing the entire machine; it's a thing you don't need to do with VMs, but that doesn't mean it's difficult. It definitely doesn't take "hours".

Managing physical setups is different from managing fleets of cloud instances, but it's not necessarily automatically inferior.

>"You talk like using a physical server somehow prevents building a highly available system or from using automation. Why would the hardware be any more of a pet than any random VM in the cloud?"

It is pure scaremongering and making not very nice assumptions about people's abilities without any real substance. If this is the idea of attracting customers or advocating the approach it seems like a very poor job.

I always have up to date standby for a fraction of money they spend on cloud. In the last 10 years or so I do not remember RAM failing in an unpredictable ways. When/if my SSD fails It'll take me a whole 10 minutes ordering another one from amazon. Oh and I always have spares lying around. My backups are automated.

Anyways nice job advertising cloud and trying to scare people into using it. You happy with it - good for you. No need to convince me as I have my own reasons and do not have to buy into yours.

> "...I do not remember RAM failing in an unpredictable ways."

I'm about halfway through my career and I have seen something like that happen exactly once; the erratic behavior of the machine had my team scratching our heads for a fair bit until we ran memtest86 on it after running out of ideas. So I would say that it's not impossible, just extraordinarily unlikely.

The rest of your post I entirely agree with.

Things did happen with my stuff as well. I had graphics card semi failed in an unpredictable way so I just replaced it. I also had one system randomly crashing because thermal paste dried up on one ancient CPU. It is just a little annoyance that happens extremely rarely. Anything is possible and I'll deal with it when/if. Nothing world shattering here. Cloud can and does fail as well. The list of reported outages affecting large chunks of customer beats my accidental cases with the healthy margin.

I find your quote here and the follow up seriously unconvincing. You know what a miserable workflow is? Preparing all my build scripts according to the half baked build tool that our in-house Deployment team cooked up to use fabric to spray my deployment onto a bunch of nodes in our data center, all the tests and script linting pass, so then I go push it through our CI/CD pipeline (which can’t do the above tests for me because again this is all threadbare maintained by in-house tooling teams who could not be less aligned to end product deliverables), and lo and behold it breaks. Good thing I’m not paying so much for CloudWatch so I get to literally go ssh to the worker node of the CI/CD system where my task failed and cross my fingers that it wasn’t reclaimed for the next task and that the log files weren’t rotated off, grep my way to tracebacks that probably involve things no one on my team has ever heard of, spend 1/2 a day chasing them all down (if I’m lucky), and finally kicking the box to restart a build a few times and it goes. Then it goes to the in house homebrewed kubernetes deployment scripts where it promptly chokes on a secret that wasn’t configured right (thanks home made script linters), so then I have to go down a rabbit hole of eye melting Helm config of secrets for an hour but thank god I’m not paying for a simple cloud UI for this. So we get the secret cleared up and everything is great, until the kubernetes scheduler can’t schedule my pods because our data center is at capacity and now I literally can’t get work done until I chase down some probably incorrect and certainly unmaintained Grafana chart of who is using how much excess memory on which nodes, and wrangle someone with authority to force another team to stop eating lunch and downsize pods right now. After all this, 6-7 hours later and 20-30 labor hours later, my workload is finally deployed, but it will die overnight for an uncaught reason that doesn’t alert anyone, and even if it did, the team managing compute resources is in a permanent state of PTSD from never ending pages so they wouldn’t bother caring about why my workload failed and will just assume it’s application logic bugs or Docker bugs on my side for as long as they can just to avoid yet another un-debuggable failure landing on their plate, so that’ll be great for morale in the morning when I get to derail yet another day and spend time doing every type of ops work that is not my job and prevents us from delivering product goals.

This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.

Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.

I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.

This is the type of cultural accuracy I've been looking for.

Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind.

The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.

The ego and arrogance of people in tech really shines through here.

People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.

A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.

Exactly, which points out how much better cloud tools are (given the level of seriousness most organizations are applying to them).

> there is such a strong brand of "anything new must be snake oil" that runs through the HN community.

I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well.

I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions.

Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it.

This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.

> The ego and arrogance of people in tech really shines through here.

You mean, "the ego and arrogance required to bend code and machines to their will and not the other way around". Maybe try to pass that and learn a thing or two.

Cool story, but the majority of that is not related to serverless. You still need a CI/CD pipeline with serverless and I’ve dealt with just as many miserable setups there.

> I can’t give the cloud providers my money fast enough

People run VMs and k8s on the cloud as well and run into every one of the problems you just described (aside from literal hardware failures).

Your rant seems to mainly be about having a good CICD pipeline and that’s not really related to cloud or serverless.

I think you completely miss the point. With cloud tooling, my team is unblocked to spin up the resources we need, manage CI / CD in our unique way that solves our problems, quickly try out serverless if it’s a good model for a given use case, change logging or alerting, get new compute resources like GPUs, etc., all without being blocked by a choke point of in house engineering limitations and staffing limitations and political in-fighting over who owns what and who gets to define how abstract vs purpose-specific the tooling will be.

You toss out “build a good CI/CD pipeline” like that’s just some localized, solvable problem, and it’s exactly that lack of imagination and lack of understanding about the true multitude of use cases that causes this problem in the first place.

I agree “serverless” isn’t a panopoly solution to everything, but externalizing and democratizing all infra tools to cloud vendors is a major transformative solution that is worth a lot of money and solves a lot of problems.

For example, if someone like you had any leadership influence in my company’s infra org, I’d be trying to spend money on cloud tools just to get away from your authority as fast as I could, and I would be 1000% correct to feel that way and would end up with much healthier tool solutions and less blockages on use-case-specific delivery requirements by doing a giant spin move around your myopic perspective.

This has nothing to do with technology or tooling, this is your team having the autonomy to do what works for you.

At my current client, my team is dependent on the "Platform Team" for _everything_ that isn't directly related to the code we're writing. Do we need a new Azure queue? That's a Platform ticket in JIRA. Do we need a new tool in our CI/CD pipeline? That's a Platform ticket in JIRA. Do we have a build failing for something other than tests? That's a Platform ticket in JIRA. Do we have an issue in production? We need to engage with Cloud Ops because we don't have access to anything in prod.

Plus, it takes six weeks to get a feature from development to production because horrible workflow and manual processes.

Plus, development teams are prevented from implementing anything without the approval of Solution Architects who are too far removed from the business to understand what's needed and too far removed from the technology to be able to design a working solution.

It's all fucking horrible, and yet we're using all the modern cloud tooling: Spring Boot, Azure, containers, K8S, Gitlab pipelines.

In contrast, a couple clients ago my team had full autonomy to do what we believed was the right solution. Stories would often be groomed, implemented, and delivered into production all within a 2-week sprint. No muss no fuss. When they weren't, it was normally due to us having to wait on another team to deliver their work.

I'm sure you will be able to setup proper pipelines on new projects within a 24h time frame and be brilliant at finding root causes by reading the stack trace and associated sources in the blink of an eye when you will be a senior hacker, I'm not worried about that!

Anyway, I think your comment proves that once we decide to invest in something, our brain does everything it can to justify that choice, Human nature really :)

I don’t understand your comment. It seems like you think it carries a rhetorical punch to (I guess) suggest that in the cloud version we’d still be slow and make mistakes? The writing is so unclear it’s hard to tell. But if that is what you’re saying, I think you deeply missed the point.

Yes of course my team won’t solve bugs instantly just because of cloud tools and of course we will make mistakes implementing bad infra designs, especially early on as we are still gaining experience with efficient and cheap cloud patterns. Nobody said otherwise and my comment before has no connection to anything like that.

Rather, if we have control over our own infra, we can adaptively fix those issues for ourselves without getting delays and philosophical or gatekeeper arguments from central data center admins. We would have autonomy to change our own deployment system, provision more compute resources, generalize something to work with GPUs, test out different delivery and runtime frameworks, create our own ingress failure robustness patterns, all without blockages and mandated central infra bottle necks.

To create such a configurable and adaptable API-driven portal into bare metal or a data center yourself is way too slow and costly, only the largest companies can do it, and then they turned it around and made it into cloud vendor options.

When medium sized companies try to do it, the most common outcome is a ton of exceedingly arrogant SRE types will create piss poor tooling that barely works for maybe the top 2-3 use cases and then will whine and complain that they can’t support generalizing it for the long tail of other use cases, and because they have political power as the gorilla in the room sitting on top of the only interface to the data center, nobody can argue with them, and you get a burnout mill with constant turnover on all the other teams who have to do SRE’s job for them while listening to the arrogance about what a rip off cloud tools are.

I don’t understand your comment. It seems like you think "anyone talking about the rip off cloud tools" are "exceedingly arrogant SRE types creating piss poor tooling that barely works, whining and complaining that they can’t support generalizing it, sitting in actual data centers, with a burnout mill with constant turnover on all the other teams". But apart from that, I'm sure you're a lovely person to discuss with ;) Anyway, I was just saying that senior hackers have more options than you seem to have, also that we tend to find excuses to justify our investments once we are locked in them, it's perfectly natural and nothing to be ashamed of.

>"...compared to the nightmare of shipping code on internally created developer tool & deployment platforms..."

I feel sorry that it worked for you this way (well rather it does not work). I have totally opposite experience. It takes single click of a button for me to build, test, deploy a code change using on premises and rented hardware.

I did cloud as well when it was mandated by client and it did not feel any simpler, faster etc. More complex and way more expensive it sure was but since it was not me paying the money I could safely ignore that part (not for my own projects and servers though as I do count my money).

The fun thing is that many C-people are fed the VC kool-aid about scalability. They want everything to br scalable because they want to cash in. Except that not many companies reach the level where you need to scale your tech stack "exponentially". And you usually don't want to scale the dumb prototype that ended up in prod anyway. The trap here is that those systems will allow you just that in a totally unauditable way... Good luck replacing parts of those systems as well when everything is interconnected...

"You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow."

Even without cloud APIs and containers and whatever, I find there's a creeping thing that happens here which has been helpful to become consciously aware of and intentional about fighting back on.

Basically, the thing is that wrappers become interfaces. You write tool A, and then tool B, and then some kind of script C which calls the two tools and produces a result. Then someone says "dang, so convenient", and sets up a Jenkins/Rundeck/Airflow/whatever job that runs script C against some source of inputs automatically— maybe there's even a parameterized build capability where arbitrary inputs may be submitted to it on demand.

Before you know it, the fact that the original tools and script exist has been forgotten, because everyone's interface to this functionality is a handy webpage. And when someone wants to add more features, they don't even go back and change the script, they end up just adding it directly into the job definition. Before you know it, the full scope of what is being done on "the system" isn't even really available any more to be installed, invoked, and tinkered with locally, not in a meaningful way, anyway.

It's not even necessarily all bad, but when I see this happening in a domain where I have the awareness of the underlying tools (or I may even have written or contributed to them), I do try to help demystify them by writing up docs on how to do the local setup, helping others get going with it, etc.

You can solve anything by adding a layer of abstraction - except having too many layers of abstraction.

"Serverless" always reminds me of CGI/fastcgi, which was still a thing the last time I wrote web apps for money. It had the advantage of "scaling to zero" and could run a huge number of low-usage apps on a single 2000-era server.

The cloud makes money for people so it's important. Would you rather that money went to shareholders or to wage earners?

> I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.

One word: Worldcom

I still remember coming up to 2000 sitting in a meeting room of an equipment provider I recently joined looking at projections of market growth, competitors and our own sales and profitability. Shortly afterwards that part of the business was sold, I remained with the other non internet part and we sailed through the .com crash unscathed.

A lot of what AWS offers makes no sense. A lot what some AWS customer's teams are building is overly complex and a horror show to maintain. But there is also value: Maintained images, backups, on demand resources and specialized resources. We hit one of the inflection points in Moore's Law and that drives us into horizontal scaling and specialized hardware.

Like with Worldcom and the fantastic internet growth story back then there is a kernel of truth and value. Find people that understand it and be glad others are playing around and subsidizing the investments in the core.

I love the scalability "argument". The capabilities of a single thread of processing in a modern x86 CPU are underestimated by several orders of magnitude in most shops. The unfortunate part is that the more you chase the scalability with the cloud bullshit, the more elusive it becomes. If you want to push tens of millions of transactions per second, you need a fast core being fed data in the most optimal way possible. Going inwards and reducing your latency per transaction is the best option for complex, synchronous problem domains. Going outwards is only going to speed up if you can truly decouple everything in the time domain.

Judging by the occasional post on HN, even a lot of experienced software engineers (the "experienced" being my take) seem to have no good handle on what a single thread and a single machine should be capable of handling, what the latency should look like and what it should cost as a result. The posts that I mean are about "look how we handled the load for a moderately active web-app which is 90% cacheable with only five nodes in our k8s cluster".

I really don't know why that is, my guess would be that few people have really built something beginning to end with the most boring tech possible (start out with PHP+MySQL like everyone did 15 years ago). Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it, like their text-editor not running smooth unless it's on an i9, because the text-editor now is a pile of abstractions running on top of Electron when vim was able to run smooth decades ago. It's sad and both an opportunity at the same time, because you can be the one with the correct gut feeling of "This should really be doable with a single machine, if not we're doing something fundamentally wrong".

I think you're right on the money with people having gotten used to it. Once I truly started harnessing the power of Vim combined with shell scripts and terminal multiplexers, my patience for many other programs and tasks decreased even further.

We have the computing power to run complex physical simulations or AI training sequences on a normal home computer, but for some reason use programs that take 100 times longer to start than old software despite not having more features, and websites that sometimes even come with loading screens. Electron isn't even as bad or slow as many people think, but somehow developers still manage to throttle it so hard that I might as well just use a website instead.

As someone who is just starting with nodejs and web development I find a lot of the tech feels nice but sometimes also unnecessarily abstract. Sure, it tends to make a lot of the code very elegant and simple, but every additional framework adds another guide and documentation to look at, another config file that has to be correctly referenced by an already existing one so npm knows what to do, another couple of seconds of build time and another source of bugs and build problems. Then of course you need that trendy package for easier imports maintenance - which IDEs automatically handled in the past, but now we gotta use an editor running in a hidden web browser which started from scratch in terms of features but is just as slow in return.

Once I get our current product in a good spot WRT maintainability and sales pipeline, I am planning to spend some time (~6 months) looking at developing ultra-low latency developer tooling. I feel like I can deliver these experiences through a browser using some clever tricks.

Going through my current project has really worn me out with regard to tolerating UX slowdowns in developer tools. I am getting so incredibly tired of how shitty visual studio performs on a threadripper with effectively infinity ram. I've had to turn off all of the nice features to get our codebase to be even remotely tolerable. And, dont get me wrong. This is not a plea to Microsoft to fix VS. I think everyone involved can agree that fundamentally VS will be unfixable WRT delivering what I would consider an "ultra-low" latency user experience (mouse/keyboard inputs show on display in <5ms). Any "fixing" is basically a complete rewrite with this new objective held in the highest regard.

Current developer experiences are like bad ergonomics for your brain. This madness needs to end. Our computers are more than capable. If you don't believe me, go play Overwatch or CS:GO. At least 2 independently-wealthy development organizations have figured it out. Why stop there?

> mouse/keyboard inputs show on display in <5ms

That's not possible, at least if you measure the full time from actuating the key to the screen showing the updated content. The record holder in that regard is still the Apple II with 30ms[0].

But I agree, modern software should react much faster. It's kind of weird how on one hand we have hardware that can run VR with 120Hz and extremely low input latency, and at the same time some programs that only need to display and change a couple bytes in a text file need 4 seconds to even start.

[0]: https://danluu.com/input-lag/

I am referring mostly to the latency that I do have control over in software. I.e. time between user input event available for processing and view state response to user.

Yup. At the last place I worked me and a colleague would joke that given the traffic our ecommerce platform actually had, we should be able to run the whole thing on a single Raspberry PI. I think he even ran some numbers on it. What we actually had was two containerized app servers and a separate large RDS instance and SES, SQS, SNS, ELB and all the other bits and pieces AWS buy in gets you. The cloud bills for the traffic we handled were ridiculous.

> Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it

That's a lot of it, I believe. One startup I dealt with was honestly proud of how they were scaling up to hundreds of requests per second with only a few dozen AWS VMs to handle it.

I'm old enough to know how ludicrous those numbers are, having built systems handling more load on a single desktop machine with 90s hardware.

But I've come to realize some of the younger developers today have actually never built any server implementation outside of AWS! So the perspective isn't there.

> You deploy individual stateless functions. But not inside a Java monolith, that is old, but on top of a distributed system. You would have been laughed out of the door if you had proposed that in 2000, and you should be laughed out of the door right now, but such is the power of marketing.

I’m sorry but this absurd and blatantly ignores the reasons why people choose serverless. No mention of scale to zero, virtually infinite auto scaling, automated deploys, arbitrary runtimes, ability to spin up code in multiple regions around the world with zero operational overhead, the list goes on...

These are non-trivial things. You’d have to pay engineers to build the same systems provided by serverless if you want smooth deployments and high availability. Also, imagine if you wanted to start operations outside of the continental United States. You’d need to find a data center provider, buy machines, pay someone to install them (and hope they do it correctly), and then, when ever you have a technical problem, you need to email the datacenter team to have someone plug in a KVM with 5 seconds of lag so your engineers can debug, and then you have to pay admins to constantly maintain the servers, and consistently pay for their rack space even if they aren’t being fully utilized. None of this is a problem on a fully managed serverless platform.

This article is just click bait capitalizing on FUD in the software community. Serverless will be a part of the future of software infra whether luddites are mad about it or not

Disclaimer: I work on serverless products so I am biased

Why is the alternative to serverless, colocating servers in a DC?

The alternative is renting VMs from those same clouds. It works fine and is cheaper when you work on software that actually has users. It also doesn't result in the cloud provider cramming a crappy under-baked programming model down your throat.

Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.

> Why is the alternative to serverless, colocating servers in a DC?

Because that’s what the author was comparing it to. Yes you can rent vms, and that works for many workloads. However serverless platforms provide other benefits like deployments, load balancing, and auto scaling that you’d need to setup yourself on raw vms. Plus, with services like AppEngine Flex you can run serverless on vms to cut costs if you have consistent traffic.

> Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.

It really depends on your problem. Sometimes you might have a very low QPS service (like maybe a few QPH) and running a whole vm would be overkill. Some business have really bursty traffic and maybe some regions won’t get any traffic at all during the night but they still want to deliver low latency service to the people living on that side of the globe. Additionally, these are just reasons to serverless for scale to zero and ignores other benefits, like reduced operational costs, that business can use to justify paying more for compute time.

There are lots of reasons to use and not use serverless for compute. I don’t think it is the solution for everyone and everything, but I do think the hate and FUD directed towards it is unwarranted and uniformed.

Disclaimer: I work for Google, opinions are my own

Edit: typos and disclaimer

I agree. I am working on a devops team that has assembled a half decent CI/CD infrastructure for widespread use at our company of 600 devs and thousands of apps. We are on our second iteration, which is even leaner and more invisible to the devs. However, we are seeing a move to serverless would be a net gain for all involved, specifically the scale to zero and not caring about the container and kubernetes configuration.

I think "scaling from zero" is actually the more interesting part of server-less. With AWS for example, it's super trivial to get a lambda running: no EC2 instance size to choose or elastic IP address to set up. You don't even have to choose a back-end framework, or set up monitoring to make sure your system stays available. You can just start writing code directly in the console the very first hour you start your project if you want, and already have a functioning API.

When you're working on your MVP, you can skip dev ops completely and get your infrastructure cost very close to zero. Then if you actually do end up building a product which is fortunate enough to have to worry about scale, you can work on cost optimization. I think this is a very relevant and valid use case for server-less.

>when you work on software that actually has users.

And if I don't have users? What's the number of 'users' where it becomes cheaper?

Most of the web is idle 95% of the time. Theres no reason people should pay for so much VM time that they're not using.

The best alternative IMHO is to just rent machines in DCs, it's cheap and powerful and you don't have to manage the hardware: https://oneprovider.com/dedicated-servers-in-north-america

lol. You are totally conflating 'serverless' with using the cloud, period.

Not going with Lambda in AWS doesn't mean you can't use EC2, EKS, ECS, Elastic Beanstalk, and so on and so forth. You're the one spreading FUD.

I think you are miss understanding my comment. I was responding to the author’s comparison of serverless and the cloud to collocated servers in a DC, I never argued that other services in the cloud are bad/shouldn’t be used/inferior to serverless in all cases.

I suppose this wasn’t clear because another commenter had a similar question/response

> You’d have to pay engineers to build the same systems provided by serverless if you want smooth deployments and high availability. Also, imagine if you wanted to start operations outside of the continental United States. You’d need to find a data center provider, buy machines, pay someone to install them (and hope they do it correctly), and then, when ever you have a technical problem, you need to email the datacenter team to have someone plug in a KVM with 5 seconds of lag so your engineers can debug, and then you have to pay admins to constantly maintain the servers, and consistently pay for their rack space even if they aren’t being fully utilized. None of this is a problem on a fully managed serverless platform.

What are you even talking about dude? You don't have to build a rack in a colo to run a non-Serverless app. There's this thing called The Cloud now, and Virtual Machines, and other fancy new technologies from 15 years ago.

"Serverless" as in Lambda is a joke. "Serverless" as in Fargate is legitimately great at lowering ops overhead. But "zero operational overhead" is absolute bullshit. Even Lambda has operational overhead.

Serverless is just the abstraction of a function into a compute unit. There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything. And the severe limitations of many Serverless technologies make it very unwieldy to use.

Give it 10 years and it may morph into something more like a monolith whose functions become "Serverless", like a Beowulf cluster where forks are executed on remote nodes and there's some ugly glue to tie it all together. That seems a lot more seamless and useful than trying to architect 100 tiny pieces together with crappy custom integrations everywhere.

> What are you even talking about dude? You don't have to build a rack in a colo to run a non-Serverless app. There's this thing called The Cloud now, and Virtual Machines, and other fancy new technologies from 15 years ago

I think you are miss understanding my comment. I was comparing serverless to colo because that’s essentially what the author was doing. I agree that there are plenty of other cloud compute solutions that work for many workloads.

> There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything

I totally agree. Serverless is usually just a piece of infrastructure that is combined with other resources like databases, batch clusters, blob storage, vms, etc. I don’t think you’d be able to create a very interesting app using stateless functions alone, though I could be proven wrong ;)

From my anecdotal evidence, people are attracted to serverless because they spend way to much on aws and they hope that by turning things off they'll spend less.

Not many people have the problems solved by a serverless architecture (or they need more freedom), which is why it's not as common as just running your own long running instances.

Not denying your experience, as it's probably true that the folks you've seen pursuing serverless may be misguided, but I want to contest your last point where you say "not many people" would benefit from serverless.

I would say that "literally any single person" benefits from serverless if they don't have the skills, knowledge, or time to handle the server/infrastructure responsibilities that come from running software.

If I'm a one man development shop with a sufficient budget, I'll probably find it far more productive when using serverless architecture to host my business logic than not.

Serverless comes with its own problems though - a less seamless dev experience and cold starts plus you have to host your data on their choice of data store etc.

I totally see the advantages, but how many benefit from that? Most companies don't need to deploy on servers world-wide, and don't need infinite scaling.

And cloud debugging has its issues too: I remember endless fudging with print statements, deploying once more and waiting for the changes to propagate in a AWS lambda handler, because the bloody thing simply wouldn't run locally.

> I totally see the advantages, but how many benefit from that? Most companies don't need to deploy on servers world-wide, and don't need infinite scaling.

That’s a good question. I think serverless has a lot to offer, global compute, and infinite scaling are a subset of features. I think it really comes down to your specific workload—it’s definitely not a be all end all silver bullet

One example I can think of I mentioned in another comment in this article which I have pasted below:

the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me. Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless. Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it

> virtually infinite auto scaling

Ah, the bullshit. When we showed GCS engineers our hadoop workload, their eyes went really big and told us that they don't have that kind of capacity in a single AZ.

Then you are working at hyperscale yourselves already, and there are maybe 50 organisations like you out there.

Maybe.

Right now I don't think that there is a single telco that has the scale to break GCP, or a single manufacturing company (exempt Tesla) or a single energy company. Or bank.

I agree that pharma, social media and streaming may do this. But since Netflix is on AWS I am doubtful of the last.

Total & Aramco come to mind.

Ok I buy that one. Oil and gas have some big problems..

I worked for one of the top-10 ecommerce shops (by transaction value) in the world. It was interesting for me to find out that the aggregate CPU power of our hadoop cluster seems comparable to what an F1 teams uses to simulate air dynamics to tune their cars.

There's a big world out there, there are probably tons of big computer users that we are largely unaware of.

Plenty of scientific computing applications fall over on the "cloud", too.

The LHC produces somewhere between 20-30 petabytes/year[0][1] (at a rate of about 200 Megabyte/second[2] after ignoring many "uninteresting" events)

The LHC is a big name, but there are many other applications in disciplines like physics, genomics, meteorology etc for whom "just use the cloud" is a terrible suggestion.

[0] https://www.popularmechanics.com/science/a16716/lhc-data-vis...

[1] https://home.cern/resources/faqs/facts-and-figures-about-lhc

[2] https://www.lhc-closer.es/taking_a_closer_look_at_lhc/0.lhc_...

I think that alphafold may indicate that biotech isn't as disjoint from cloud solutions as many suppose. Super computing is also getting eaten as grid solutions go into hyperscale, but yes ..I find you LHC and the mega telescopes.

Actually, Grid Computing.

i think you greatly misunderstand the size of some telco's and Infrastructure providers.

also, some manufacturing giants are truly enormous..

samsung for instance makes everything from smartphones to artillery installations..

same goes for hyundai, mitsubishi etc..

I did big data at one of the biggest European telcos. The only thing that kept any of it on prem was capex Vs opex (and we had sunk capex so that wasn't coming back into the budget) AT&T claimed their Hadoop installation was ten times bigger than ours, but it was still will within the parameters of any of the hyperscalers.

> or a single manufacturing company (exempt Tesla)

Tesla is TINY.

Try things like Siemens.

EDIT Besides, the point is not the whole of AWS of GCS; it's the about availability zones vs singular datacentres. Distributed computing doesn't really like to throw data even within the same AZ - in-DC will always be faster. AZ and the modern idea of distribution suit certain things - streaming, for example, benefits from it, but not everything.

My company is migrating a huge monolith to microservices. One of the proposed benefits is that it makes end to end testing easier because each service only interacts with a couple other services, so each service will be easier to write integration tests for. But in the monolith, those services communicate over interfaces, and are currently tested as unit tests! So we're replacing unit tests with integration tests and calling that a win because our integration test coverage is higher.

Just wait until BA asks for data from different sources and you have to connect those parts...

> So we're replacing unit tests with integration tests

Nit: those were always integration tests.

But now they get to use docker and some homegrown scripts in their tests, instead of a testing framework!

Reminds me of the tweet - we replaced our monolith with Microservices so that every outrage is like a murder mystery

I believe the technical term for a tightly coupled set of microservices is a 'distributed microlith'.

Presumably you intended outage, but outrage also fits and is poetic in its own manner.

If one part of the application needs to be changed frequently and/or is used by other applications, then it makes sense to separate it as a microservice. However, for other parts, I think it might be better to stay as a monolith since when you have a network in your calls, things get messier.

Networks are unreliable even if it is your local area network. In my career I saw some unrelated network equipment was sending a RST packet to load balancer with application servers IP address and resulting in service unavaiable errors. It took us weeks to find that out since it was happening very rarely and there were many layers between load balancer and application servers.

Network means slowness, connection errors, unknown results, etc all of which should be handled carefully.

Not sure if rate of change truly matters. In my opinion, microservices make most sense when dealing with multiple team. It's much easier to update core services when everyone depends on it through the network, rather than requiring people to redeploy their stuff. If you're a single team operation, it's not really a problem if all your stuff runs in a single process. It may even benefit you in terms of performance and scalability, to be honest.

Rate of change may matter when your monolith application takes too long to deploy (i.e a big war application with lots of initializations at start), although rolling updates may help here. It will also be easier to rollback if things go wrong.

On the other hand, I agree with your point of multiple teams. I tried to mean same thing with "used by other applications"

How long are you talking? I can't think of a process I've ever worked on that took more than a few seconds to start. As you say, if you have any sort of rolling deployment mechanism the start time of your application is somewhat irrelevant. Unless your application takes a massive time to start, it seems like the wrong thing to optimise for. Surely you only deploy things a few times a day, if that, and hopefully most deployments don't require rollbacks.

Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.

Depending on other services via a well defined network interface is good for decoupling teams. So it depends on the number of employees you have coding for the same platform, rather than on clients or anything like that.

> How long are you talking? There were some Spring applications that took more than 5 minutes to start in my old job. However, things are most probably improved now with lazy initializations of beans and maybe developers of that applications could have done better at that time. I don't know it is still the case now.

> Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.

Network speed was not the problem, decompression / expansion of the application was. There were war files about 100-150 MB size. Also I saw one with size of 1.5GB however it was an exception. These may or may not be a concern depending on your CPU utilization, disk speed, memory, etc.

Most decisions depends on environment. For some environments, as you said, deployments are not a concern for microservices, for some they are.

If networking is not a problem, why not deploy it already decompressed? I'm assuming the 5 minutes time included decompressing it? It seems a bit weird to take so much time to allocate some objects in memory and possibly establish a few connections.

Anyway, can you share which kind of environment even a 5 minute deployment would be a problem? Most places I worked at it took a lot more than 5 minutes to actually implement the change that's going to production, so deployment times were pretty much irrelevant overall.

OP here. The move is definitely needed in our case as it's to the point where IntelliJ won't even load anymore. But I had to laugh when improved testability was brought up as one of the benefits.

I stopped reading, frankly, at...

> I’ve now worked with “The Cloud” for long enough to see that there’s still a long way to go before it becomes materially better than, say, the oldschool method of renting a couple of servers

Lol, it takes a lot of hubris to assume you're the only person, or one of few persons, who understands how the rest of us have been conned. I'm not getting into a debate about the hundreds of ways our infrastructure on Google cloud is "materially better" than "renting a couple of servers." It ought to be sufficient for persons of humility and sense to know that we're also smart people with experience, and we continue to re-examine these decisions and run our stuff in the cloud because, yeah, it's materially better.

And for those lamenting that software engineering is "fashion driven", welcome to humans. To some extent we're all stumbling forward in the fog, and the sound of other footsteps nearby is comforting.

34 million adults in the U.S. still smoke.

"Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise."

This hit home... the amount of money I've seen businesses pour down the drain in my past few years of experience is insane. Offices with less than 10 people being talked into server setups costing tens of thousands of dollars for nothing more than a NAS, recoridng camera footage, Door access, etc.

You can still get "scale to 0" with cgi-bin applications. Yes, it's old (but it works). Yes, it's slow (so is Lambda).

Unlike Lambda, CGI works on any web server and any OS. It's also extremely easy to mock because the input and output is stdin/stdout, not AWS API invocations that cost money every time you run.

What you miss out on is "scale to infinity" (read: your bill goes to infinity too).

I recall when AWS first released Lambdas I thought of cgi-bin, especially with PHP where the deployment model is just pushing files. As you say, it has pretty much the same upsides and downsides.

That model does scale well for the web layer at least. Just lots of cheap VPS nodes. Of course, the bottleneck then is the database or whatever layer provides state and persistence.

I think the main selling point of Lambda (and most SaaS) is you don't have to worry about the machine running your code. Personally I haven't used it so correct me if I'm wrong but you don't have to worry about things like monitoring, network administration and security (beyond your code and general opsec).

Yeah, FastCGI is a little closer since it fixes the startup time problem.

It's weird to me that PHP uses FastCGI but Python support for it is really limited. I'm using FastCGI on Dreamhost and it was a pain to get it to work.

https://lobste.rs/s/9o4zsx/back_70s_with_serverless#c_9hxzjs

https://news.ycombinator.com/item?id=24683304

CGI and FastCGI can also be used with containers, although it appears that almost nobody does it.

I think if "bubblewrap" (user space container tool) starts to be installed on shared hosts via Red Hat, we could hack it.

> XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.

There are a few fundamental differences between these languages, but it's just plain wrong to say that any of them are not "structured." Did OP mean "schema-less" instead? Because the only fundamental difference at that level is that XML has a built-in mechanism for metadata referring to a file's own schema, while in JSON (and I believe the others, even though I've used them less) the schema is not linked from within the file itself. You can still verify the schema of these languages, though, just as you could with XML. And once you've verified the schema, local code can do stricter validation to make sure things hang together properly, that values are within limits, and so on. After that, I expect modern tools like Terraform and CDK to check with the target platform that the scheduled change can be applied before trying to change anything.

That said, OP has several valid points:

- Many projects try to make a microservices architecture long before it's needed, even in projects where it might never be needed. I suspect some people think microservices are actually easier to maintain, since they are smaller pieces, but only extremely trivial or shoddy microservices will be easier to maintain than a corresponding monolith, because many things which are well-understood in a local setting are much harder to do in a highly distributed setting, simply because the infrastructure is now a bazillion separately managed pieces.

- The feedback loop can be abysmal. Without any easy alternatives to do "local cloud" deployments, having to wait for several minutes for feedback about a tiny change is frustrating in the extreme.

By that logic, is https://json-schema.org/ is the reimposition of a schema within JSON?

I'd say that JSON is "structured", and probably HCL as far as I know, but the others have problems.

I haven't worked with an API that uses XML in a long time, but does it express the difference between "0123" and 123 well? I think that it either doesn't, or you add some really awkward schema language with XML namespaces ...

I just translated my .travis.yml to JSON, and I learned that

    deploy:
      on:
        branch: master

gets translated to {"deploy": {"true": {"branch": "master"}}} because "on" is a synonym for true. I'd call that unstructured. It also has problems with keys that start with -, etc.

Also, all of the languages are often used with say Go templates, which makes them more unstructured. They're no longer statically parsable, which is "table stakes" for doing anything else (validation or static analysis).

I took some pains to make shell statically parsable [2], and I think it's now a better foundation than these config languages, per my other comment here: https://news.ycombinator.com/item?id=25483270

I think HCL is probably a lot better, but it also seems to be the least used! It's structured, and has its own mechanism for expressing duplication that's not the equivalent of a 70's-style #include.

[1] https://github.com/oilshell/oil/blob/master/.travis.yml

[2] http://www.oilshell.org/blog/2019/02/07.html

> "on" is a synonym for true

And "no" is a synonym for false, hence "the Norway problem."

https://hitchdev.com/strictyaml/why/implicit-typing-removed/

> I haven't worked with an API that uses XML in a long time, but does it express the difference between "0123" and 123 well? I think that it either doesn't, or you add some really awkward schema language with XML namespaces ...

Is as easy as declaring

    <!ATTLIST data field NUMBER #IMPLIED>

    <!ATTLIST data field CDATA #IMPLIED>

with DTDs.

> There are a few fundamental differences between these languages, but it's just plain wrong to say that any of them are not "structured." Did OP mean "schema-less" instead?

I read him as referencing what people would have called "structured programming" in the 1980s, and pointing out that we have these settings/metadata/control files in languages that have no capacity for "structure" in the sense that, say, Pascal or C would have over BASIC or assembly, that is, thin-to-no factoring or abstraction capacity. Hence the contrast with "Configuration as Code."

Now, depending on your personality and experience, you might find the idea of using a complete language as config an opportunity for someone to do something non-obvious or horrifying, and the closer to the "settings" end of the setting/metadata/control spectrum you're on the more plausible that position is. When you get to a level of complexity that's the other end I'm not so sure and horizontally scaling systems might well be arriving there.

> the only fundamental difference at that level is that XML has a built-in mechanism for metadata referring to a file's own schema, while in JSON (and I believe the others, even though I've used them less) the schema is not linked from within the file itself

As long as you're using plain DTDs for your schema, yes. Like early XML-based config languages such as idk JSPs/web-app and Jakarta (née J2EE) deployment descriptors used and still use. But later XML configs use XML Schema and namespaces, which are similarly detached from an instance document as eg JSON Schema and the others you mention [1].

[1]: http://cmsmcq.com/2005/xml2005-166.html "How schema-validity is different from being married"

The right answer to this, as usual, is probably somewhere in the middle. Perspectives like this are pessimistic, but important because the industry has age bias and yet we need to understand where technology might be going full circle. But instead of taking that as a bad thing and stopping there, we need to understand why and how the context is different this time and what that means. Managed services in general seem to be the ultimate cloud computing offering so far, and aren't reinventing the wheel. They are expensive and costs can be hard to reason about (probably on purpose), and cloud computing companies are profiting greatly from inefficiencies, but they take care of entire categories of operational concerns. This feels like the natural progression of the field. We can take advantage of these things selectively while being conservative in when and how we adopt them.

Complexity is a difficult topic, not the least because the many ambiguous meanings it has taken on. When people bring this up, they talk about it like it's a fixed thing that could go to 0 if only someone actually tried hard enough. Yes, complexity is a problem, but that complexity is always increasing is inevitable as long as technology advances. The first episode of the 70s series "Connections" comes to mind. We need to be able to identify and eliminate waste while dealing with this fact.

> Perspectives like this are pessimistic, but important because the industry has age bias and yet we need to understand where technology might be going full circle. But instead of taking that as a bad thing and stopping there, we need to understand why and how the context is different this time and what that means.

I feel like this is a critically important aspect of software engineering that we are missing. I want to say this is because the field is so young relatively speak. The fundamentals have been changing so rapidly and are truly not completely well understood enough to be captured as reliable axioms. So instead we go down this path of recreation, sometimes with improvements, sometimes without.

In a hundred more years maybe things will be better, but for now this is taking on a bit of a vicious cycle. Literally people in the wrong age group are getting "moved" out of the scene not to mention the thousands of others in the "wrong" industries, tech stacks, and culture.

Software in today's age is a fickle industry when it comes to people...

Coming at this from data & analytics, my hope with serverless is to drive some management focus into the underlying data structures that support our applications. In Enterprise world when software grows the hard thing to do is integrations and data management. Take some of the major offenders in these 2 categories - SAP and Salesforce - the challenge is how to make these systems talk to other systems AND get some meaningful analytics out of them. SAP can't even talk to itself sometimes due to incompatible deployments across subsidiaries (this happens a lot when companies do m&a). I don't quite like serverless either but there is an area, databases, where it's going well and it might push companies to take better care of their data. Having 1 scalable Aurora (Postgres) database in AWS serving multiple microservice applications is a data analytics dream - entities, tables, users would be all automatically integrated by default. Running databases is super hard, so maybe removing friction with serverless can push companies to reconsider the absurdity of spending millions for crap like SAP and Salesforce whose main goal is to lock your data in their systems.

I dunno seems like 90% of this is mostly caused by a need to reduce complexity. In my old job I basically talked to alot of startups about what they were building etc... Alot of teams get free aws credits and use lambda as their process so they can serve data for low to no traffic websites, save on a devops team to get things going. It's because the scope of things you might have to do is so small and the traffic served is also really small. So in the beginning you'd probably rock such a system. When you or if you ever scale out where you are dealing with multiple concurrent requests ( most things are not like they were in early 2000s server logic heavy requests). In fact we've moved to fat clients vs. the old paradigm of thing clients. Now caching layers ship entire heavy weight clients to an app and the server does more mundane things like handle database writes that are shipped off to probably another hosted solution. Additionally actions that require a db write are probably few and far between. Login/ if you're gdocs maybe a periodic save on 15second interval.

Again probably the focus should be on scale and scope. If things are easy water will flow to the path of least resistance. I've done all kinds of deployments over the years (private data center, vps, aws, gcp, etc... with all kinds of deploy management and configuration tools. Having a well maintained permissioned, monitored up time sync'd, has the right packages on the right machine, machine has a non trivial upfront cost. That if someone is going to give me 50-125k of AWS credits (as a startup) then why not. Alternatively if I have a low traffic website and want to dogfood building a product then why not especially if the use case for a db is maybe tracking payments ocassionally handling a login.

My two cents there.

Upon seeing this article while drinking my Sunday coffee I was reminded to do my weekly AWS serverless checkin.

I noticed one of the Lambda functions briefly spiked to 6 concurrent instances. Within a couple clicks I confirmed a bot was scanning the API, possibly looking for vulnerabilities. The built-in API throttling hopefully dissuaded the bot from further scans.

Before serverless, my capacity planning for servers would have required accounting for this spike in “noise”. I would pay for a peak burst scale of 6 servers, but practically only use 1-2 for routine processing.

My serverless infra are just “cattle”. AWS Lambda does some magic behind the scenes and determines when to kill an instance, recycle, or create a new one. I just feed it an occasional monolithic JAR that handles all API calls and queue handlers.

From the OP _[With serverless].. You deploy individual stateless functions. But not inside a Java monolith, that is old, but on top of a distributed system._

This is where the Author seems to conflate the benefits of serverless with a particular architecture or code deployment model. The two are completely separate.

Serverless is not just a technology, but from a lifestyle perspective it is a game changer. A solo software developer can rapidly experiment and test ideas until one starts to scale. Most of the experiments can be run in a “free tier”. And the costs of scaling are comparable to provisioning your own EC2 instances or containers.

> A solo software developer can rapidly experiment and test ideas until one starts to scale.

In all fairness, this same thing can be done with virtual machines. VMs have been "cattle" for a long while now. "Serverless" just removes the start and stop buttons from your UI. There are still servers and there is still configuration, it's just being done on the conflations of others.

What I find interesting is how many people feel the need to justify using whatever it is they are using when others are exercising their right to question whether or not all the tech we've built is "worth it".

In many cases, having all this power of the cloud has just removed value from companies as they allow their employees to spin up whatever and leave it running because they don't know any better processes than what has been sold to them by crafty sales and marketing people.

How many millions have been "blown" on cloud just to say something is started or running? My observations say "a lot".

For all its flaws in design or philosophy, cloud computing has a sweet spot.

As an indie, lone dev, I can run my site, email and a basic web app for about 10 USD a month and, if my app kicks off, I can scale it to a million users without breaking a sweat.

To me, that is the true power of the modern cloud computing paradigm. An influx of great apps and services enabled by cheap scalable and sweet competitive cloud services.

Was this even possible in 2000?

Properly scaling a site is more than just clicking a button. Your architecture needs to have been built from the ground up in a way that actually can scale horizontally, for starters.

True. I made an assumption that the app is built that way. Of course, if te app architecture inherently unscalable, no amount of computing power or private servers is going to help.

> True. I made an assumption that the app is built that way. Of course, if te app architecture inherently unscalable, no amount of computing power or private servers is going to help.

Isn't it usually a mistake to design your app to be scalable to 10,000 users when all you may ever have is 10?

Nah. Scalable design is good design. It forces you into thinking a level deeper about your app and forces you to use good architectures.

Yes, albeit it was a bit more expensive.

I was running multiple sites by myself. There was less encapsulation (unless you were working with someone on bsd), for sure, but I'd say the advantages are more visible in terms of security (hacking was much easier back then) than ease to deploy.

Doing your own ops has never been that hard (YMMV) and I don't think it improved much nowadays. Scaling to n machines didn't change as well. Deploy n instances, load balancers, scaling vertically then horizontally, db sharding. You still have the same fundamentals.

One pro of today is having a single artifact to deploy (tons of cowboys deployments and FTP uploads in the past).

One big cons you have nowadays is learning all the latest crap $currentCompany is using and having to peel the layers of abstractions to see why things work so badly.

> Doing your own ops has never been that hard

I kind of disagree. It's doable, but still requires maintenance. For instance, even running a simple blog off of a VPS requires you to worry about setting up an SSL certificate and keeping it renewed, staying on top of security updates, and putting together some kind of scaling solution assuming you ever want to write a post which gets to the front page of hacker news.

I think there's definitely an argument at a certain scale to just outsourcing that complexity so you can actually work on what contributes to your unique value proposition.

Your examples are poor. LetsEncrypt and certbot automated cert renewals a long time ago. If you’re still having to manually renew your SSL certs, you’ve picked a crappy vendor. Security updates are also as simple as a cron job that runs your favorite package manager update command. You want to run those commands on multiple machines? Use the several different FOSS orchestration platforms that make it easy for you to deploy and configure your distributed services. Sure, these things take some initial set up, but it’s nothing that can’t be accomplished in a few business days for a start-up.

Wait until you see the "data transfer" costs for what is basically or literally free from the bare metal/colo hosts. $10 is too much for what you're describing. You could be paying $0-$3.

Most users of the top public clouds are paying 2-10x more than they should just because using the public cloud "is what you do".

For running a static site on Amazon, for low traffic requirements, it indeed costs about 3 to 4 USD per month.

Also, which bare metal server providers would you recommend?

I'd be curious how thing would look if the cloud providers broken up horizontally. Like, if the companies that provided VM's were different from the ones who provided higher-level services. That's kind of how it was originally with Heroku and its add-ins, and AWS/Rackspace. I liked that so much better and wish it stayed that way. If you want a VM, go to Rackspace and knock yourself out. If you want a hosted thing then go to Heroku, or whichever service hosts things in the language of your choice. But the point is, you make a choice, when you sign up for whichever service, what level of the stack you want to operate in.

I can also imagine this approach having benefits as a consumer of services, but it's hard to imagine a world where vertical integration would not win over time. There's just too much of an advantage in terms of large cloud providers being able to optimize their hardware setups at data-centers to make their services cheaper, more performant, and more tightly integrated than 3rd party competitors.

I guess. But then Anthos, Arc, EKS Anywhere, they're all horizontal, and so it's odd that they are also all run by the cloud providers themselves. I'd think those services would be the next layer, run by different companies.

Still, what you said is right. I mean, heroku still exists, but the ever present complaints are that it's too expensive at scale and that it is too constraining. So, maybe the vertical is the best approach.

> [XML] lacked all the facilities that apply to writing good code, so principles like “Don’t Repeat Yourself” went overboard and copy/paste programming ensued

Technically, XML has entities (text substitution variables) and entity references, along with a evolved infastructure to share declarations and declaration sets via parameter entities. Of course, that doesn't help with tools such as maven which outright forbid use of entities and use their own ad-hoc mechanism for reuse. Technically, this makes a pom.xml non-XML-compliant, but in any case, whining about XML pointless when the creators should rather blame themselves for using XML in the first place, and then using it inappropriately.

As to FaaS, I don't see the problem with CGIs; they've stood the test of time with ultra-robustness and uptimes in the range of years and decades, including service updates with zero downtime by virtue of Unix processes being detached from their binaries (unlike Windows processes). Also, since you can run CGIs as one-shot invocations on the command-line, developer feedback as postulated in TFA is excellent. Of course, CGIs itself use environment variables rather than function or command-line arguments, so their lifecycle is bound to Unix processes. What helps is to use wrappers rather than coding to CGI specs directly; we had this in 2007 already with SCA (service component architecture). Maybe it's time to refresh SCA with new bindings for AMZN and other proprietary FaaS to bring back the portability we valued highly in pre-Cloud times?

As to FaaS, I don't see the problem with CGIs

I'm sympathetic to this view point, but a few problems:

(1) Startup time. FastCGI solves that, but it's a more complicated protocol that requires language support, unlike CGI.

(2) Language support and (web) framework support. Python's FastCGI support is kinda crappy; you'll probably see worse in other languages. Although I noticed Rust has some FastCGI bindings, so I guess people are still using it?

(3) Container support. This is probably the biggest one. All the hosting providers that I know of that support CGI or FastCGI don't give you root (for good reason), and they also don't support rootless containers (good reason 10 years ago, maybe not a good reason now)

So if shared hosting starts installing "bubblewrap" rootless containers on their machines, then we could build our own "serverless" FastCGI. But we can't really do it now.

I use FastCGI on Dreamhost, and I just tried it on NearlyFreeSpeech, and neither is quite adequate. It's funny how small a distance there is though.

https://news.ycombinator.com/item?id=25488958

https://lobste.rs/s/9o4zsx/back_70s_with_serverless#c_9hxzjs

In a similar vein, other things I find interesting in the software industry:

- Startups looking for a CTO but they already know they will run in the cloud and have chosen their frontend and backend stacks.

- The way mainstream technologies get established not by quality but by pure luck. Javascript was a pretty poor language added to the Netscape browser in a hurry, but since everybody hated Microsoft at that time it succeeded.

- The way old technologies remain relevant just because many people know about them and there is a good ecosystem/large codebases. C/C++ comes to mind.

- The way compensation is hardly tied to the value created. Often you can only advance by switching jobs all the time. That will not let you learn from your mistakes, and systems become legacy very quickly.

- The overkill of infrastructure in the cloud just to get a simple system running. This is the complexity the author mentions, which leads to huge and expensive support orgs and high costs. I still love building a system from the ground up as much as possible with very few moving parts instead of gluing together the latest AWS services. It's shocking how many orgs believe they need to rely on K8S just because everyone else does.

Wow. All the bashing at kubernetes and serverless functions. I’m not sure what the point of the author was other than the complexity aspect.

K8s and containers solve some very painful problems. I’ve seen two multi month migrations to k8s and cloud-y systems from hard metal servers.

Hard metal is nice but it’s hard to guarantee that state between replicas is the same and there is no version discrepancy.

Everyone that moves to cloud is because they want a good balance between compute and storage. They don’t want to deal with figuring out how many servers to rent. Let it scale up and down as needed. When you care about 99.99% reliability you want redundancy and multiple replicas load balanced. You want prod to run what you staged and tested. You want isolation guarantees. You want ephemeral systems that can be recreated from scratch and destroyed easily.

If you’re just a couple of devs and you don’t care about multiple 9s of reliability and everything fits on a single server then yeah the cloud is not worth it. K8s is not worth it. Know thy tools and the problems they solve.

The "glass half full" perspective here. The OP misses one hugely important factor: cost and hence availability of computing resources.

During the recent history of computing, the technology improvement opened new possibilities at each step:

- Punching cards: allowed more people share mainframe,

- Video terminals: allowed faster turnaround of mainframe jobs,

- Mini-computers: gave more people access to computing,

- PCs: got even more people into computing,

- Commodity hardware in datacenters: more people could develop services and more people from general public could get access to applications and data,

- Cloud: more people could afford to build applications,

- Microservices: bigger teams could collaborate on increasingly sophisticated services,

- Serverless: reduced the burden of capacity planning and service distribution and further lowered the barrier of entry into backend application development.

Each statement above comes from personal experience and I am looking forward to what new possibilities the future brings.

While I think this article is a great discussion about the increasing complexity of software and how the trends driving that complexity are marketed, but I think any article discussing these trends without also talking about the organizational structures of the teams delivering and running the code miss the point a bit.

When you have an organization with hundreds or thousands of developers delivering code that works together to support a business there is some types of complexity that are inescapable. So the question is “where do you want the interfaces between people and teams?”

SOA and microservices push that interface to the API layer which drastically increases the complexity of the code, but is a logical place to put the complexity if you consider the organization, people, and code a single system.

I think you're correct but I also think the article is correct.

At scale there are levels of inescapable complexity. There are a lot of tools that help manage this complexity. However too many in smaller teams emulate bigger companies and inject wasted complexity by using tools that solve problems they don't have.

Most organizations don't have hundreds or thousands of developers working on some coherent product.

In fact this is quite seldom. So all the tech that solves problems that occur in such constellation should be quite exotic.

But it isn't. The reason is that computing "on a mainframe" (the cloud) can be sold orders of magnitude more expensive. The rest is marketing of the mainframe owners.

Depends on how you define product, there are a large number of tech enabled Fortune 500s that do not sell a tech product but have built an integrated internal product for their operations that do need that scale. Airlines for example.

But yes, I agree that most companies jump to this level of complexity too quickly.

Airlines? How many billions of flights and passengers do you have to manage daily as an airline? Don't forget, computers are fast.

Also I think airlines do just fine on core system form the 80'ies until now. They also don't tend to have software development teams with "hundreds or thousands" of people to my knowledge. So maybe not a good example.

There is only a hand full of companies globally really in need of all that "serverless" stuff as I see it.

I am intimately familiar with the software and IT organizations of multiple airlines. You are wrong, they have large distributed software development teams building very complex software to run every part of the airline’s operations.

Some buy off the shelf, others buy components off the shelf and focus on custom where they feel it gives them an advantage, and others have built the whole thing themselves.

So airlines were one of the first cloud users because they couldn't scale otherwise?

And they're doing "serverless" these days as their IT operations are a good fit for this paradigm?

Or could it be that they're small enough to get things managed with off the shelf software and/or something built and run in-house?

Airlines have been running their own data centers and fully understand the costs associated with managing their own infrastructure and many have decided to migrate to the cloud because they have decided it is worth outsourcing that function, it is not a key competency and they cannot keep up with the pace of innovation in the IaaS market. If they look at their IT as a complete system, the extra costs of the cloud are offset in a number of key areas, including increasing their ability to scale.

You mention of the shelf software, what do you think the organizations that build the COTS stuff (like Sabre) in this industry look like? Where do you think it is run? They’re all moving to microservices and cloud hosting too, including serverless where they think it makes sense (and a lot of times when it doesn’t, just like everyone else).

The smart ones (who understand Conway’s Law) are making these moves because they are enablers of smaller, more independent teams which we know are better at delivering software. The really smart ones know they’re paying the price in increased complexity the comes with a distributed system, but are willing to pay it because that cost is lower than the cost of the collaboration necessary to build a monolith of that size.

Logistics companies are one of the most interesting intersections of software and the real world. Most organizations are making a mistake by focusing on scaling their software too early or unnecessarily. Airlines and logistics are not, they have business problems that their existing software cannot solve in areas the COTS options do not fit in their operations.

Lately, the main reason for using “serverless” (aws lambda in particular and all related wax) has primarily to do with security “compliance” at enterprises, cost of it in particular, and nothing with developer productivity or technology... aws just makes it super trivial and cheap to meet all the compliance requirements if using only this dilapidated stack.

It’s no wonder any startup with half the brain would still pick rails/ruby for actually creating something of value, while enterprise middle managers and bean counters argue about how to not write code.

Love the article and the perspective.

Don't think the general notion that serverless is either a return to the past or a negative development is true.

Long feedback loops seem to be more of an issue with a lack of engineering discipline where I've worked. Implementing a comprehensive, fast test suite is up to the team to build, not AWS. Serverless is a tool like any other that needs to be used in a responsible manner.

If you're using cloud features that geniunely cannot be tested locally or quickly I would reconsider using more stable technologies that are perhaps enhanced by your cloud provider instead of totally replaced by them. _You_ are responsible for creating a short feedback loop regardless of your deployment environment.

I've been using serverless in both GCP and AWS for many years now and have helped migrate mid sized startups onto those platforms for event processing and API build. In that time I've seen bills go from thousands of dollars to a few dollars a month.

Mid sized startups are now able to compete in data heavy fields precisely because of technologies like serverless that have enabled small teams to build scalable systems. When you work in transaction processing or analytics I think you'll understand that scalability directly limits how many customers your platform can support.

Not everyone just makes simple json http endpoints.

Thank you for providing a more nuanced answer.

I definitely agree that feedback loops, local dev, and testing are the things I hear about people struggling with.

For the most part that is a one time cost that your team can breeze through if you invest some time in initial setup.

If anyone out there is looking for a painless transition take a look at netlify and vercel. If you outgrow them later there is minimal lock in.

I also use pulumi for infrastructure mgmt, and it works extremely well as a lambda dev pipeline that manages complex stacks with ease.

Great article, it resonates with everything that has gone wrong in the industry.

I do not think that serverless is the thing that the author thinks it is. It's definitely not "batch", and it's definitely really cheap. I think that the fact that the author doesn't look at the economics of various deployment mechanisms is pretty revealing - within the first three hours of my first cloud engagement we were making faces about the costs of different options. To be honest this is the biggest preoccupation of every team doing cloud that I have looked after. It's also the reason (the only reason why business are going to cloud. If you can get 20% e-2-e soup to nuts savings that is fantastic but in terms of system architecture that's a very thin and high risk landing zone to aim for - it's really easy to screw up and get +20%. Serverless is one of the methods that can help a project drive overall costs under the line, and I say "good for serverless".

An example of this is how lambda millisecond resolution has given me a clear reason to push for code optimization as costs savings initiative.

Previously it was harder to budget for that kind of work, but now there is a clear target and roi.

I think most of the people in this thread don't really see the actual savings these services provide.

I hired 5 new devs last year off of what I saved by moving to lambda from eks / ec2.

"Cloud is the new mainframe"

This is the message we've needed.

Cloud is an expensive, overly complicated, lock-in boondoggle.

Open source needs to rally hard to prevent cloud providers from running their tech.

TFA is talking about serverless, not cloud computing in general (e.g. rent a VM, run stuff in it).

I don't love that I pay aws to run my open source domain(s), but the extent of their job is: keep my VM up, provide me with outbound email that works, offer backups. For now, I can live with that.

"Serverless" has to be the most confusing misnomer ever. How about calling it "Stored procedures" or something.

> Cloud is an expensive, overly complicated, lock-in boondoggle.

Only if you go with companies that make it such.

Renting web hosting on someone's collection of rack-mounted systems running some well-known OS is Cloud, and there's nothing inherently bad about that, unless you've defined Cloud to mean "Only The Bad Stuff About Using Computers Other People Own". You can run things like they were run in the 1990s, complete with Apache and mod_perl CGI and MariaDB swapped in for MySQL and everything held together with shell scripts and run via ssh and ftp. No magic, no lock-in, no complexity beyond what every other serious general-purpose computer system is these days.

I agree that Big Cloud Providers layer on pricing models and whatever middle-management-ware they can come up with to obfuscate the underlying mechanics, but if you're against that, say so. Don't catch everything in your blast radius because it reduces the overall impact.

If it is overly complicated for you don't use it.

However discounting it altogether because you don't need or understand why that complexity exists is naive.

I hear this a lot in interviews and it's a clear signal that you haven't worked on anything big enough yet.

> I hear this a lot in interviews and it's a clear signal that you haven't worked on anything big enough yet.

Quite a bold wager, and totally incorrect. I hope you're not presumptive and rude to all of your candidates.

FWIW, I'm a senior engineer at a large tech company where I started pre-IPO and have had pretty great career growth. I've been on a variety of teams dealing with most of the levels of our stack: eventually consistent data models that are active-active, k8s, frontend, tearing down monoliths, etc.

I'm fine with microservices for medium to large orgs, and I love Kubernetes.

Lambdas and cloud are expensive lock-in, and holding this view doesn't mean you hate distributed systems or complexity. You can staff your own platform teams and manage your own capabilities. Imaging machines, upgrading, failing over - it's all automatable.

The excuse I hear is that companies don't want to build things that aren't related to their core competency. But I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk.

I'm incredibly fearful of the future where all software runs within AWS or Azure. We'll all be paying taxes to these companies and we won't be able to run or replicate similar environments on our own. It'll be difficult for new engineers to learn as our field will effectively become a "trade".

Can you imagine the world where it's expected that everything is composed of SQS and Lambda and other bullshit?

Great response.

I'm going to focus on one part of it for now.

"I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk"

This is factually correct, but more nuanced than "just build it in house". Many large orgs externalise this risk by relying on AWS / Azure / Google. Nobody ever got fired for picking IBM is more relevant now than ever, and it is because the largest companjes on the planet have invested huge amounts of time and money solving these problems.

I think you misunderstood me at some point, because nowhere did I advocate that everything should be a lambda. Or that everything should be in the cloud.

The point I am making is that a lot of people I speak to are afraid of complexity and lock in, when they should be afraid of never being able to grow past what they are comfortable with.

To your point about this becoming a trade... Brace yourself, I would wager most development positions will be obsolete in 10-20 years as AI assisted infrastructure and development hits full stride. Especially dev ops and front end.

(Crossposting from Disqus).

I think I am gonna take this with a grain of salt, meaning as the rant that it is :)

It's valuable to hear some dissident voice from time to time because 100% agreement on trending paradigms always creates the risk for the new generations to jump to the cargo cult ship and accept the common wistom as a dogma, instead of appreciating with own reasonment the current state of suggested practices.

As always, virtue lies in the middle and computing is no exception: I was professionally born in the age of in house racks where every SME would own a handful of servers and sysadmins would plan whether colocating applications or buying new iron that had to be accomodated somewhere in the DC. There have been very good reasons to switch from monoliths/SOAs to microservices and from bare-metal to containers/FaaS, reasons that I personally would not trade back for the bad old days of manual deployments: microservices offered live patching of systems with no downtime and imposed design by contract to ensure verifiability of single cogs inside the big clockworks, instead of having to run the full stack in own laptop to ensure end-to-end functioning.

I think the point of the article is really how bad is to swtich to new paradigms for use cases that are well served with single application, single server monoliths (internal inventory management systems, for example) just to change ship and "do what the others are doing".

Yes, the cloud is conceptually the new mainframe, but usability has also improved a lot: the user experience created agility and the agility created the opportunism that creates the commoditization of infrastructure. Is Heroku a sustainable use case for production? No, but it is super for development. Heroku is feedback loop for product development as smalltalk was for pure development.

So it is probably true that following blindly market trends just for the sake of it is a very bad idea, but the author seems also to be suggesting "Java monoliths or die!" which I don't agree upon.

The point on the level of complexity being unsustainable is a good one.

Maybe if the author was using managed services instead of self-hosting his blog, his SSL certificate would be valid

Bit of a miss on why “serverless” (though author & I have different definitions), possibly because of language / toolset.

Lambda / Step Functions make for an easy combo for easily understandable units that you can use without a bunch of servers. Combine it with AWS CDK and you end up with a low boilerplate, auto-completeable environment (with types!) that’s actually easy to unit test locally!

I agree that yaml config and figuring out how to wire things up sucks. I’m open to Smalltalk being better.

But we’ve been using Lambda / Step for a few years now and it’s been highly robust, easy to understand, monitoring / logging essentially free and - best of all - we haven’t had to manage any server upgrades / patches / etc the whole time. (And super easy to spin up stack per user)

Have a queue of items to work on? Combine lambda + sqs (just a few lines with cdk) and now you have code that auto scales linearly with how many entries are in your queue.

We did write our own tool to handle boilerplate of linking python functions to their lambda handlers (but again this is the beauty of actually being able to use programming language to define your stack)

> Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy.

Too many companies have to hope of hiring the level of competence required to not suck. It's painful but true. AWS is cheap and serverless is popular when setting up a linux image security and correctly can take weeks or months.

Which is why it's great that server-less exists. I would much rather live in a world where a single developer can test out an idea in a way which will scale than where you need to hire from an expensive and scarce pool of dev ops engineers to get a product off the ground.

This is clearly a surface opinion without a great deal of experience in leveraging the numerous benefits of cloud and serverless.

Complexity can be managed if you start out thinking about. The biggest problem with the cloud is ad-hoc. If you go in just building shit without a strategy and guardrails, it's going to cost a lot of money and end up in pretty bad shape.

CGI is in vogue again - I am so happy!

I think people miss that there isn’t a single solution that is a perfect for everyone. As your business grows there is a point where co-location makes sense, there is a point where cloud makes sense, and there is a point where maintaining your own datacenter makes sense. Over time you will make multiple transitions because the economics of the moment make it favorable.

Spending time in cloud does break people of bad habits, it’s worth spending some time in a cloud environment, but eventually you get back to the point where it’s cheaper to build your own infrastructure then to rent someone else’s.

> Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy.

This is the best description of modern software development I have ever seen.

> It seems that, contrary to what everybody is saying, software developers are in plentiful and cheap supply so wasting a lot of their talent makes economic sense over the perceived cost of adapting more powerful programming systems.

> but we just throw more bodies at the problem because Gartner tells us this is the future.

This last part is the real reason for it: executives are blind morons who have no idea how to run technology-dependent companies, and their poor leadership bleeds down because hierarchical organization.

I used to blame executives because it seemed like they were making terrible decisions that everybody knew were terrible. But now I realize that the reason they're making the terrible decisions is they just have absolutely no concept of how the organization should run. They don't understand the customer, the employee, or the product, but they have generic ideas of how to make generic things, so they just do that and hope the people under them will fix everything.

So of course we're managing everything as tiny little components that get glued together. They're "Fordisming". And "Globalizing" is already here too. Many companies already have one microservice made in India, another in Romania, a third in Spain. Who cares if they barely talk? Who cares if they reinvent the wheel and use conflicting technology? It's all just a game of "reduce cost and increase profit".

The next big hotness is going to be the API-gateway-service-mesh-distributed-tracing monolith that every service will be plugged into. Developers will be given more power to turn on and off things that are connected to this monolithic service frontend. They will thus be able to have faster feedback, but nobody will be coordinating the changes they are making, so actually things will break more often due to conflicting changes.

Finally, some time after that, the developers will figure out that it would be better if, instead of depending on all the external Infrastructure/Configuration-as-Code, they should be using frameworks that have everything built into them, and have their application tell the framework what to do, so a dedicated person won't need to write it all using external tools and weird DSLs. This will persist as a sort of "Java framework for Kubernetes-native Applications", although it'll extend into all the thousands of shitty extension apps that plug all the holes in K8s. And then we'll truly have a new monolith, but a distributed one.

> You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow.

You can run and test FAAS, even the proprietary Lambda, locally. In fact, if you aren’t doing so, I dare say you’re “doing it wrong.”

Lambda is just one more tool in the tool chest of pluggable components that scale infinitely. “Given input X, do Y,” is boiled down to a predictable, repeatable service that requires almost zero infrastructure knowledge.

We were definitely doing it wrong at my last job. We had 200+ lambda functions (a lot of redundant lambdas that were breaking DRY rules). Each function was terraformed and there was no way to provision the AWS resources to run the lambdas locally. So whenever I made a lambda, I wrote all of my code locally without testing and then had to push it up to delpoy. Then I could run it and debug in the console. Nightmare. It was even worse with batch.

A lot of truth in this article, but dont get me wrong, the added complexity is what is fuelling the market and we are all parts of this.

I dispute that I am a part of this. HN is a big "we" - you don't get to include everyone here in your "we".

I've been working with serverless projects for about 2 years now and I'd agree a lot of these gripes are valid. The slow feedback loop is the most painful. Building new features in serverless apps is going to almost always require multiple pushes in order to get what you want. It's very challenging to get it "right" the first time.

As an old timer trying to get back up to speed with the new stuff... I keep coming up with the answer that Lazarus/Free Pascal is a perfectly valid, powerful, and quite unfashionable tool for just getting things done.

Unfortunately, nobody is going to hire a Pascal programmer, no matter how productive. Serverless, here I come.

He started out talking an out how a quick feedback loop was of paramount importance, and then got pretty far afield. Eventually he asserted that Serverless elongates that loop. Maybe so; I don’t know, I haven’t used it.

That a fast feedback loop is important was the only contention he made that I found important.

Wow, great post! I have the same complaints:

(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!

(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.

We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:

(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.

(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.

Debugging distributed systems locally could be the norm, but it's not.

The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).

(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.

XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.

I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:

https://news.ycombinator.com/item?id=25343716

Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.

Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.

Yes, it is one of the most entertaining and insightful articles I have read on computing and programming language for a long time.

All the best for your new Oil shell, and I agree that we need to have similar to POSIX standard for distributed systems but it must put local-first software as the first class citizen [1].

Personally I think the local-first software cannot happen soon enough given that what we have endured for the past 30 years with regard to inefficiency and drudgery of the web based applications.

Just wondering about your thoughts on SDLang [2] as the type alternative to JSON, YAML etc, and lightweight alternative to XML?

[1]https://martin.kleppmann.com/papers/local-first.pdf

[2]https://sdlang.org/

Here's my recent comment about sdlang:

https://news.ycombinator.com/item?id=24945449

So I think Oil will be very familiar to SDLang users, BUT it has programmability, which you need in cluster configuration and build configuration, e.g. to express duplication.

So I would divide the space into a few categories:

(1) Stringly typed: .ini, YAML

(2) Typed Data: JSON, XML to some extent, SDLang

(3) Programmable configuration: HCL, Cue, Jsonnet, Nix expressions, BCL, Starlark, Oil

SDLang looks really similar to HCL too, except HCL is programmable:

https://www.terraform.io/docs/configuration/expressions/dyna...

Very important: With categories 1 and 2, people add template languages like Go templates on top to solve the programmability problem, which we don't want!!! We don't want to move typed data back to the stringly typed realm!

Also, I don't think the "expression style" of HCL, Nix, and BCL is great. It's good for small things, but it starts to get hard to read for large configs. I'd prefer simple for loops and if statements, basically like the Starlark language (a subset of Python used in Bazel).

---

Thanks for the paper reference, I bookmarked it!

true and should hit home with most, but still flawed. especially the biting conclusion.

> Not by going back through our history and figuring out what caused the Personal Computer Revolution. Which is a shame.

the thing is, these aren’t the days of the PC revolution. there is a baseline complexity which is easily an order of magnitude greater than the most complex systems of that day. to stick with the concept of a single machine, just the ME in a PC today is more complex than the entire machine back then. mostly to our benefit.

I find the observation that 'programmers' are too cheap interesting. I think this might indeed be one of the core problems. Because devs are often a bit timid and clumsy regarding the negotiation process they are taken for granted and all those hyper-complex systems are forced upon them. If devs would charge 500% more, maybe companies would opt to change their tune and give control back to the devs.

Another aspect is 'responsibility'. Most companies don't want to bear any form of 'responsibility' and therefore they late Azure/Amazon/Google take care of it. However, what they forget is that in turn, they become dependent, which is even worse.

I am constantly amazed at how complicated doing simple things has become.

There's very little yak actually there, once you get done shaving it.

i would rather pay $85/month for a 2U box at a co-lo facility than pay amazon 1000 times that price for the same performance....

$85K a month for the equivalent of a 2U box in Amazon. Where did you get that quote?

solve problems with new problems..

You could have a repo per function, but if you have functions with common dependencies (whether libraries or build tooling or whatever) then you need to deal with keeping those dependencies up-to-date and notifying upstream when a change breaks downstream. "Monorepo" is just one solution to that problem, and it's exactly the same as what monoliths do today. You can have the same problem in a company with many monoliths, but with some amount of code reuse--"monorepo vs multirepo" is orthogonal to "microservice vs monolith". Note that monolithic monorepos tend to work well because "monolith" typically implies that all dependencies are using the same language, and most languages have decent incremental build tools; however, we don't have good polyglot/general-purpose build tools--instead we have Bazel/etc and Nix which each suck for entirely tractable reasons. As good leadership and investment coalesce, this class of tooling will improve, but for now building and deploying a polyglot environment is painful (but there are other kinds of pain to consider, and the microservices people argue that this pain is less than e.g. organizationally-derived pain points).

> Conceptually, hiding a lot of the scaling and coordination machinery is not a bad idea; programming systems like Erlang and OTP have shown for decades how well that can work and Elixir is giving the platform a well-deserved surge in popularity. But there’s a big difference here: a platform like OTP handles pretty much everything that a platform like AWS Lambda handles, but it does it in a single programming language.

The author is really close here. What remains is a more careful analysis of why single-language programming systems feel better than multi-language programming systems and vice-versa. For example, Erlang (like any language) isn't good at all things (e.g., you might have data-science subsystems that you'd ideally write in Python or Julia or whatever, but you might also have other parts of the system where performance and quality matter [Pythonistas, this is your cue to downvote me]), so mandating that all components be implemented in Erlang might cost more than making heterogeneous systems talk to each other. Or perhaps procuring Erlang programmers is costlier than programmers in other languages? Ultimately it's hard for a single-language environment to beat out a multi-language environment, especially at scale (i.e. "as the diversity of problems grows" not "as the performance demands increase"). I think Erlang actually is a very cool platform, and I wish it well, but it doesn't mean that it's optimal for many/most businesses today.

This is a rare article in that while I disagreed with its conclusions and most of its reasoning (yes, serverless is stupid if you believe the economics of software companies haven't changed since the days software was distributed on floppy disks), I still walked away appreciating it for some reason. I guess that it's interesting to consider the similarities between mainframe computing and serverless even if they are largely superficial.

> These systems took ages to compile, hours to run all the unit tests, and clearly interactivity went down the drain. So “split it all up” was not a bad idea. It was an unnecessary idea, mostly driven by the shortfalls of mainstream programming languages, but necessity is the mother of invention - even though your average Java business app was much simpler than the Smalltalk IDE before you added a single line of code to write your app, it was already too complex to maintain and “divide and conquer” was the answer. Service Oriented Architecture, and later microservices, was born. Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy.

Maybe I'm misunderstanding, but it seems like the author misunderstands the raison d'etre of microservices. He seems to posit that they exist because of performance or complexity limitations in the host language, so we break them up, and because we break them up we also break up teams to work on them. Or something like that.

In truth, microservices are really just about embracing Conway's Law--your architecture will resemble your organization structure eventually whether you like it or not. So we empower teams to deploy and operate their own services instead of coordinating deployments with other teams and throwing issues back and forth over the dev|ops team wall. Notably, Go microservices are common, and like the author's SmallTalk example, Go services can compile and run tests in roughly a second.

We don't run microservices because of compile and/or test times, but rather because we're optimizing for the realities of organizations--the overhead of synchronization and communication--in a world where software isn't shipped annually in a shrink-wrapped box of CDs or DVDs but on every load of the webpage.

> The return of the Job Control Language this time was in the form of, again, configuration data on how to run your microservice. Microservices were somewhat fatter than the very fine-grained objects of the old days, so there was less of it, but still - it was there. The feedback cycle became worse as well: in the monolith-with-XML days, your XML editor would get you mostly there and a quick local compile and run would leave you all but certain that your configuration was working. XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster. It suddenly felt a lot like being back at that university interacting with a mainframe, but at least you still owned the hardware and could introspect all the way down, especially if you were doing “DevOps” meaning mostly that you had administrative access to your hardware.

I'm confused on a few points. First of all, XML has structure but JSON/YAML/etc doesn't? Is the idea that Spring or whatever statically type-checks XML but that Terraform/CloudFormation/etc don't type check their configuration? I'll be the first to acknowledge that we would be better off with better type checking in infra-as-code, but there is some type checking. And certainly the ergonomics of these tools is rough and it will get better when we stop pretending that infra-as-code can be "as easy as a few lines of YAML!" and embrace the realities that we need to DRY this stuff up exactly like we need to DRY up our application code (and thus we shouldn't waste time poorly reinventing the abstraction wheel--looking at you Terraform and CloudFormation).

That said, is the idea that Job Control Language is analogous to an infra-as-code tool like Terraform or CloudFormation? Does it allow the user to declare the ideal state of the world and then reconcile it against the current state? I've never worked with JCL, but I'm skeptical. Maybe I'm missing the point here.

> Microservices have trouble scaling, and they are very complex. Most companies that employ them have no need for them, but the systems and programming languages they employ are sufficiently lacking that this stacking of complexity on top of complexity becomes a necessity.

Microservices certainly can be a lot of complexity, but I still don't buy that this is a limitation of programming languages. Again, you're trading technical complexity for the ability to spend less time coordinating with other teams, and the return on investment for this tradeoff will vary from organization to organization. This complexity will improve as the best practices become more firmly established and the tooling ecosystem improves.

> pparently, Worse Is Better and you can do worse than Virtual Private Servers, so through a short-lived detour through containerizing microservices and deploying them on a distributed scheduler like Mesos, Nomad, or Kubernetes, we have arrived at “Serverless”. You deploy individual stateless functions. But not inside a Java monolith, that is old, but on top of a distributed system. You would have been laughed out of the door if you had proposed that in 2000, and you should be laughed out of the door right now, but such is the power of marketing.

Again, the author doesn't seem to understand the rationale for microservices / devops / serverless. Even if his conclusion is right, he's not rebutting the actual premises for microservices. Virtual Private Servers, pets, don't scale. You can't get a big organization to maintain stateful servers well, so in the 2000s we had smaller Ops teams who would maintain the servers and any change had to go through them. Businesses don't like paying to build features and not being able to ship those features to customers for months. Further, debugging issues that arise from a massive monthly or quarterly integration of features is much more effort than the collective effort of debugging issues that arise from incrementally delivering features, especially when the stateful production environment is inevitably different than the stateful staging and dev environments. It's weird that the author clearly values fast feedback loops but can't appreciate how a daily or hourly feedback loop is better than a months- or quarter-long feedback loop, (especially for startups, where iterating to product-market-fit is paramount).

Yeah, you could have each team operating its own pet VPS running its own service, but that's pretty close to microservices. And you'll also realize that each team configuring its own process management, SSH configuration, firewall rules, monitoring, alerting, etc is kind of a lot of wasted effort and most developers aren't very good at or interested in that kind of work anyway, so it would be nice if we could have a pool of hosts that an ops-like team could manage which the development teams could deploy to and operate as they like. Congratulations, you're halfway to Kubernetes. If you can buy into the immutable/GitOps philosophy then you're pretty much all the way to Kubernetes.

Articles like this try to project earlier failures on current progress.

Yes, serverless shares some conceptual similarities with older development models.

However, the tooling, infrastructure, languages, and general support for server less as a productive model make these conversations seem naive and mostly just "old man yells at cloud".

I am an old developer, and comparing platforms like vercel and netlify functions to 70's mainframe dev is just dishonest.

> telling the computer how to run it took pretty much the same amount of typing, this time not in a programming language but in a structured markup language.

The myopia of "how much code and config is in my app" is inefficient for code reviews, but weigh that against what you gain.

Plenty of people use languages and frameworks without understanding the massive inefficiencies involved.

With configuration, you get insight into some of that inefficiency. It's also more inefficient during code reviews to have to review N changes in configuration.

But, is the end result fast enough for users and is the time needed to develop and maintain it sufficiently low?

If the answer is yes, assuming you're a foot soldier in the dev brigade, you keep going.

It's good to reduce the noise. Chuck Yeager started as a mechanic and worked on his own planes when he became a pilot. That made him a better pilot.

But, James Hart Wyld was the one concerned about the efficiency of the engine. Chuck just needed to break the sound barrier without dying.

ah yes, the 1970s where you could sit on your laptop at home, go to a cloud providers website, copy paste your code and launch an app in minutes.

This guy is trying too hard to sound smart.

I'm probably the same age as this guy, the article definitely made me roll my eyes a few times for this exact reason.

There's another angle we're not discussing too.

Serverless functions are usually free. This is a huge boon for learning: I'm a college student with limited resources, and even spinning up a $5/mo VPS takes some thought and budgeting.

I learned web dev with free web hosts & free VPS services back when I was 13-15. Further democratization will only help newer generations explore & tinker with confidence (scale-wise and tech wise).

So in the 70s you could pay per request billed to millisecond precision?

Wow times haven’t changed at all.

He has some good points on poor configuration languages and bad architecture, but it’s lost in the rant.

> "So in the 70s you could pay per request billed to millisecond precision?"

It's been a while since I encountered it but yes, mainframe time was so blisteringly expensive that usage was accounted for on a per-job basis and billed to individual user accounts. I don't recall the resolution or pricing anymore but students would be issued an hour or so (?) of processing time to complete their coursework per semester. Perhaps somebody with better recall than I can fill in the details.

Indeed. Unix even has it.

https://www.rigacci.org/docs/biblio/online/sysadmin/ch21.htm

I know on the Control Data Cybers the time was measured by the second with 3 digits after the decimal. Might have been measured with even finer granularity, but that's what they displayed.

I remember seeing somewhere a Linux or Unix program or system that shows how much money it costs for running the program. I think it was from an online course but really couldn't remember where, but the cost is very expensive...

Mainframes were (and still are) billed by fractional second of CPU usage, total storage, bandwidth between storage and compute, bandwidth off box, total number of jobs scheduled, and per-use fees for third party tools such as sort, RPC servers, and such. So times literally have not changed in this regard.

I just tried NearlyFreeSpeech, which uses FreeBSD and jails, and it has resource billing, although a little more coarse grained:

https://faq.nearlyfreespeech.net/q/ResourceAccountUnit

So basically if your website doesn't get any hits, you don't get charged.

Reply on news.ycombinator.com