Etcd, or, why modern software makes me sad
roguelazer.com
roguelazer.com
There are people attacking the author for a statement made about CoreOS, and for some hate towards Kubernetes.
The key point of the article is not really being addressed here: vested interests from large companies are able to introduce huge complexity into simple, well-designed projects. While the complexity may be good for some end that said vested interest has in mind, they are also in a position to absorb the cost of that increased complexity.
In the meantime, the simpler version of the software is long gone, with the added complexity placing a large burden on the solo developer or small team.
It's almost like there's no good representation in the open-source world for the solo developer or small team. Funding and adoption (in the extreme, some might say hijacking) of open-source projects from large corporations dictates the direction of all major software components today. Along with the roses come some very real thorns.
Just my 2c.
Bear in mind the livelihood of nearly everyone participating in this conversation is impacted by the situation being lamented. It's telling that CoreOS employees felt compelled to set the record straight and casually "float ideas".
It's further curious to note the difference in HN's reaction to an extremely salty and offensive post from, say, Linus Torvalds, versus this one. People are really bad at objectively setting aside their ideology and tone is a kind excuse to get out of having to genuinely think about whether the angry individual might have a point, and when it hits close to home, well. All bets are off.
You're 100% correct on your overarching point but the less kind interpretation is "the incentives of modern software engineering value complexity to artificially strengthen a software engineering job economy". The good news is that the more Big Tech screws the pooch with user data management and software engineering, the stronger an argument I have among very receptive ears in government circles to start thinking of Big Web/FAANG as hostile to responsible computing innovation, because they never really bought what Mikey sold them, and the troupe that followed Mikey to Washington didn't hang around to explain the downsides or long-term management needs of the current landscape.
Sometimes I feel like the people who are deep down in this "full stack developer" land should come up for air and look around at the bigger picture. There's a much bigger world of software development out there and actually I think this post was kind of an attempt at that.
The web-facing industry seems to have dug itself down into a very deep rabbit hole of frameworks-on-frameworks-on-buzzwords, in a way it wasn't 10 years ago when I last played in that space.
It looks very weird over there from over here :-)
There is a watershed coming where Big Web/FAANG will lose this mantle, and DevOps, half-baked Agile because nobody ever bakes it all the way, framework engineering, and a complete fundamental ball-dropping around security and exploitation of user data are rapidly ensuring it.
At some point, Big Web/FAANG and everybody delivering Web services of some kind got the idea that the Web was computing. I think the prototypical version of this thinking led to Rob Pike's observations about systems research in the early oughts. This forum tends to operate with that underlying assumption about the lens to view computing through no fault of anyone in particular, which is why I say that.
I'm a little confused by the reference to healthcare.gov: if Big Web is not the right paradigm to handle a... big web site, then what is?
I realize that’s an underwhelming answer, but if we asked that everyone who disagreed with the status quo simultaneously proposed a perfect solution to fix it, we would be nowhere in any sense. I assure you that thinking about the answer to your question is a significant part of my life, and talking to people who use computers to shape that thinking is key. Government was largely responsible for setting the computing agenda and spurring all of the right technologies in the beginning, and I believe there is merit to revisiting that thinking rather than ceding computing to engineers on the West coast.
There were precursors to this... GWT was an early example of abuse of the web browser stack to accomplish something very non-webby. But it's really gone full bore now.
Back in the mid-90s when I made web pages for a living when some designer came to us with a photoshop mockup in a PSD and said "make the page look like this" we felt perfectly fine saying "no, the web can't do that. design for the web, not photoshop"; no such luck now. And so here we are...
In order to deploy a web app in 2020, one must write his frontend using react and typescript that would transpile the code into a nicely bundled and minified javascript and css files along with thousands of your 3rd party dependency libs. Yes, you heard that right! I just counted the number of libs on the node_modules in one of my small side project and it's over 1000s. You might also wonder why the typescript code would produce css when transpiled. That's because in 2020 we no longer write stylesheet directly into a css file. That's so 90s! We write the css as javascript objects now, then let the transpiler generate css from that. Isn't that neat?!
For the backend, you can choose one of the many cool frameworks and languages. Heck, why not use 5? In 2020 the backend should be written as a collection of microservices that talk to each other over grpc. Of course you still need http or how else would your react frontend talk to your microservices? So your microservices now talks multiple protocols, which is very cool!
Now pack each of those service into a collection of docker images, then write 4000 lines of yaml files to orchestrate them on kubernetes, and you're ready to go! Now you have a glorious, scalable app with 99.999% sla that can withstand your show hn frontpage traffic thanks to kubernetes, until you eventually screw up that yaml config and the whole house of cards collapse, but that's on you because you're bad at kubernetes orchestration!
I think we're indeed in a deep trench but damn it's quite an impressive trench we got here!
Don't forget GraphQL on top of that, otherwise you're not really in 2020.
I am wondering why someone just hasn't forked etcd if it is truly so awful now.
While I do like cheering for annoyed people to fork projects it's not always practical for a number of reason like:
- The project is large and really requires multiple people to maintain.
- It's written in a programming language that the person is not familiar with, or interested in (they might even dislike it).
- License issues.
- Their work might prevent them from working on open source projects due to intellectual property concerns.
I have witnessed that firsthand. Despite the anecdote being backend, I’d bet my 401(k) it’s the same up front.
When you are still trying to figure out exactly what HTML and JS should be responsible for, having React, Vue, PHP, Elm and many more as options for "just making a simple web page" can slow down the learning process by an order of magnitude.
I'm not sure what the solution is here- clearly each of these tools has a niche which requires them, but making a basic webapp without a web-engineering degree is quickly becoming an impossibility.
In 5 years react and kubernetes will be considered legacy tech and we will march on to other new shiny tech.
> "the incentives of modern software engineering value complexity to artificially strengthen a software engineering job economy"
Not sure I would agree. Hanlon's razor comes to mind. To me, the referred complexity is reminiscent of a number of other complex patterns that somehow found their ways in software engineering, out of possibly good intentions. Regardless, an idea needs to be sold and when the marketing is well executed the community will ingest the kool-aid. Which it then tries to digest for a few years. Then begins a game of chicken that consists in not being the first to admit that you're in pain, out of fear of being branded an infidel, or out of sunk cost bias. Eventually a few courageous and influential souls finally spit it out in disgust and scream "wtf is this shit?!?" Then the rest awakes from the common stupor and a holy war ensues. I still remember SOAP. I still remember with what verve I was sold Scrum.
As for Kubernetes, only time will tell.
It seems that systems and principles invented for some specific situations get rapidly entrenched as "best practice" due to the wild success of their root corporations. That's twisted much like fashion - "that dress looks great on that model, maybe if I wear that dress, I would look good too".
- the companies' successes are often predicated on strong ability to develop technologies
- good (well-maintained) technology requires resources and the successful tech companies have those resources.
You could just as easily level similar rants against the likes of React and it's wider ecosystem, Tensorflow, Typescript (many will disagree), Docker... I'm sure others have their own bugbears.
Much of this is subjective, of course. But to me, it feels like software development is trending towards unecessarily complicated development processes and architectures.
And the only beneficiaries appear to be the large technology companies.
I suppose in exchange, you're getting a guarantee of maintenance. But is that really worth the additional complexity associated with the common use of these tools?
JavaScript taught me to love async, then functional programming, and TypeScript taught to me to love static types. I'm now desperately wishing for a world of OCaml/Haskell, but where are you going to find teammates using those? And so I'm back at TypeScript.
I think all of these "higher order" languages that do transpiling (including Scala, Closure, Kotlin, Groovy, etc) fit this same scenario. Increased toolchain complexity for a decrease in development/test/maintenance complexity.
With any toolchain, though, it can be hard to know where to draw the line for "worth it for this project" until you're already an expert in using the toolchain, at which point, why wouldn't you use it?
The same thing in TS adds at least one step (not to mention the rest of the tooling you will want)
So while a prefer it over JS, there's no arguing that it is more complex as now you require a build step for a language that only exist because people wanted a language without a build step.
And the group mentioned is (I deduce) not generally individual devs, enterprise devs building some internal thing, and so on but teams in companies doing public-facing SaaS, teams in startups, companies like Amazon/Google/Facebook/Apple all the way to AirBnB etc, and so on.
So, you don't really need stats for that.
It’s native typescript.
``` #!/use/bin/env node
console.log("hello cli") ```
While it does depend on node, and there are arguably better crossplatform languages for this purpose, it is a zero-tool chain use case that is very convenient for us.
<script type="module">
A lot of fortune 500 companies with some developers who missed the trendy stuff still do it that way. I made a medium size website (30 pages) in React with pure javascript and dependencies being script tags in index.html to vendored files.
So not even JSX. I did it that way because it was the easiest way to develop and deploy in that environment
But then if that's an option, I think Typescript will be the last language I migrate to, because Typescript development culture, tending as it does towards overcomplicated solutions to simple problems, is unpalatable to me.
I'm drawn to the idea of using Rust over WASM as a frontend language, and I think I'd rather choose that approach to develop any browser UI where type safety is critical, provided there is no discernable difference in performance (when compared to TS over WASM).
In fact, I wonder how ECMAScript will fare in a post WASM world... I suspect it would still thrive tbh. Or perhaps people will take to other flexible, expressive languages for UI development. Like Python's niche in computer graphics, or Lua in games and AI research.
I can still see myself using JS in that future. But not for everything.
I've written >50k LoC of TS in last few months for sure (doing a huge frontend migration for a client) and I can't remember the last time I googled anything TS related. Actually I remember - a month ago I wanted to know how to define a type with a field excluded, took 30 seconds of google.
Meanwhile the project started out as mixed TS and ES6 because most of the team was unfamiliar with it and there are a few dynamic typing evangelists - we ended up going back and just using TS all over the place, the complexity introduced is minimal and the productivity boost from good tooling on product of this scale is insane.
Which is why you use the up-to-date official documentation.
That said, the type system has come a very long way even in just the last year. The biggest improvements imho being around recursive types (which was one of the biggest holes for a long time imo), literal types / literal type inference, and tuple types.
It's not complete by any means, but it's improving quite rapidly.
I would assume even Array<T> is still bivariant as well...
Both of those are horribly unsound, just for convenience. Sure convenience and compatibility are Typescript's ultimate goals, but to actually praise it for its generics? That's very strange.
> TypeScript absolutely does allow you to express these relationships
How would you express a class with covariant or contravariant type params in Typescript?
[1] https://www.typescriptlang.org/docs/handbook/type-compatibil...
``` function partialCall<T extends Arr, U extends Arr, R>(f: (...args: [...T, ...U]) => R, ...headArgs: T) {
`
But I love strong type systems; people who prefer weak type systems would likely consider things like this to get in their way.
> partialCall takes a function along with the initial few arguments that that function expects. It then returns a new function that takes any other arguments the function needs, and calls them together.
The type signature looks like:
type Arr = readonly unknown[];
function partialCall<T extends Arr, U extends Arr, R>(f: (...args: [...T, ...U]) => R, ...headArgs: T) {
Now, looking at it piece by piece:
function partialCall
<T extends Arr, U extends Arr, R>
This function's first parameter is:
f: (...args: [...T, ...U]) => R
The `...` in `...args` makes it a rest parameter[1] (variadic parameter in other languages). The type of `...args` is `[...T, ...U]`, which you can think of as the concatenation of the T and U arrays. (It's a bit strange to see `...` in a type specifier, I can't recall having needed this before.)
...headArgs: T
I don't know if I'd call this normal or clear compared to the signatures I encounter daily, but it's pretty elegant for a higher-order function that takes a function with any signature, and some arguments that match the given function's parameters, and returns a function with the matched parameters removed. And the implementation of partialCall is just this one line!
return (...b: U) => f(...headArgs, ...b)
[1]: https://www.typescriptlang.org/docs/handbook/functions.html#...
I’ve seen some JavaScript written so tersely it was nearly impossible to figure out without spending Possibly hours on unwrapping the code. That’s the value provided here.
Implementation approximate, but otherwise true story, both LHS and RHS:
`let { statefulMutatingAccessor } = require(obj["prop."+tree+".subtree"])`
One could completely forgo that, and come up with PureScript or Elm. But then one can't beverage most of the existing code.
Same tradeoffs as what Kotlin and Scala make for Java's sake.
The argument for simpler tool chains for simpler projects is that the time it takes to on-board should not outweigh the time saved by the toolchain’s amenities.
So the argument for TypeScript despite its complexities is that you believe its approach to type decorations on top of JS strikes the right balance for ease of on-boarding (IMO fairly straightforward) and supporting code maintainability (IMO a big improvement in 99% of the cases) despite the extra complexity of lengthening the toolchain (IMO it has some counterintuitive parts but you can mostly just forget about it once you find a working setup.)
I won’t use TypeScript if I’m whipping something up quickly, but if I anticipate open sourcing, adding unit tests and CI, etc., then TypeScript feels like a useful addition.
Unless you have to work with legacy code or libraries.
But seriously, if modern C++ has one thing going for it, it has a great backwards compatibility and forward refactoring story.
I'm not a huge fan of the "evolved" syntax, and I rarely use C++ anymore, but I like modern C++ substantially more than C++03, which was where I first cut my teeth as a programmer.
This isn't always an option. Google's C++ guidelines generally forbid using exceptions, but concede that you might have to if you're dealing with existing code, particularly on Windows.
https://google.github.io/styleguide/cppguide.html#Windows_Co...
Having walked that way I can recommend you to look at F#, and if you aren't a solo developer (or you want to get real regarding jobs and colleagues) make sure you do not have allergy to C# interop.
So complexity (in tools and process) do not necessarily serve the individual developer the way they serve larger organizations.
Easy!!! No not actually easy because of all the constraints.
It had to be stowable. So a hard cover was out. It couldn't produce toxic fumes if it caught fire. So most plastics were out. Cloth was a problem because it couldn't get sucked into the intake if someone tried starting the engine without removing the cover.
He designed heavy canvas cover with metal stiffeners. And snaps. And then had to switch to a draw string because the snaps failed at -40F. And tended to get clogged with snow.
Whole thing took him three months.
Then there was the friend at college who worked on a VCR to record gun camera footage. Also a bunch of requirements. Higher video quality. And can't lose lock when the pilot does a hard pullout after firing his munitions. And the total production run? Well exactly how many fighters does the airforce have? Couple of thousand?
I think military tech has a problem that it's trying to keep up with commercial driven tech which operates on a scale that's a 1000 timers larger. Military produces a few thousand artifacts to commercials few million artifacts.
Not to excuse military contracting pork and cost padding, but this is a good point that a lot of people seem to miss. There's also the fact a military contract will be for a production run and follow on support.
So they buy a thousand full units and parts/tools to fix and maintain them up front. You can't just go to Autozone and pick up a new tank tread or parts for a jet turbine.
There are aircraft that were designed and created in the 1970s that are still in use today. Sure, many of the components and internal systems have been modified and upgraded, but much of the original design is still there and operating.
>> a military contract will be for a production run and follow on support.
When "follow-on support" has to last for 50 years or more, it makes a big difference.
First flight: 15 April 1952; 68 years ago
Introduction: February 1955
Status: In service
I've seen (and removed) plenty of requirements that were put in the specification just because. Because someone needed X amount of Y technology in their projects for the year. Because it seemed cool. Because other people were doing it. Because they wanted it on their resume. Because. But not because it was appropriate to the project or its purpose.
I'd say the majority of the work I do now is pruning these nonsense "just because" clauses out.
Its entertaining at least. I regularly shake my head in disbelief as I go through a specification and wonder, "who hired these people?" or "Why is the person who hired them still working?"
Fortunately for me I also get to hand these questions back, much like the good uncle who only needs to uncle and not actually parent: "Here, have them back!" I say at the end of the visit.
All is well. Go pleasantly amid the wastes.
The the hammer was waaaaay more than $1000. But that's because the military said "We'll pay you a total sum of X, but to make it easier to fund the project we'll let you break it into n parts and pay X/n per part".
The contractor decided that to make their cashflow smoother, they'd include "manual impulse force generator" as one of the deliverable parts.
The hammer was part of a larger contract that included spare parts and R&D effort linked to those parts. When the spending was reported, the same absolute amount of R&D ($420) was allocated to every item, inflating the apparent price of a $15 hammer to $435. By the same token, the engineering work on more complicated systems (e.g., an engine) was an absolute steal at $420 and since the total amount of R&D spending was fixed, nobody really got ripped off.
Those tend to be an accounting artifact, I heard. If you order a thousand different items for a million dollars total then each one of them will show up as costing exactly $1000 no matter what they are.
As with many government related procurement systems, there is so much paranoia about abuse, and desire not to repeat various disasters from the past, that the system has by perceived necessity become complicated.
Of course it makes it frightfully expensive to the degree that few companies can actually throw the resources at it to be able to navigate it. The lack of competition can result in projects costing a large amount of money, and the buyers in the agencies having no real options to look elsewhere.
At an e-government company I worked for, we a service for the state that helped companies navigate the state's procurement system. We created a centralised place for all data to be gathered by the applicant, generating all the forms they needed, and provided a way for the applicant to track the state of their application. It introduced quite a change for the state, dramatically widening the potential pool of companies that could bid for contracts. Also pissed off quite a few of the already entrenched ones :)
HIC was a great company to work for, as was NIC from the limited interactions I had with the larger corporate powers that be. Most websites/services were provided for free to the state / state agencies, instead relying on small fees per transaction on some of the stuff we did, to fund the work that didn't have transactions (I forget the amount, but we're talking something like 50c per transaction).
Being free was a key incentive to getting various agencies to come on-line.
I think React itself is awesome and I've always enjoyed it. While I'm not an expert, it's conceptual foundations and core abstractions felt right, and I do think it makes lots of frontend tasks simpler, especially for non-small projects.
At the same time the frontend ecosystem is a mess, as we all know, and as my CSS skills advance I see more just how much could be done with HTML / CSS / limited use of JS and your classic REST app, that's not a single-page app, and how much complexity might be able to be avoided and thus time saved.
At the same time, I don't think it's just "trendiness" that has caused the growth of React and SPA's. I think they give us various wins that we probably take for granted because we're now used to them.
So, the short answer to my long ramble is, I don't know.
But I do think React itself, specifically, hit a sweet spot in terms of being a powerful yet understandable tool, and still remaining a tool, as opposed to a framework.
Dev A: I need an API to CRUD
Me: But we've been doing CRUD for 30 years without an API this is a small project
Dev A: But I don't know how, its not best practice, my team lead agrees, here is a medium article, get with the times, etc.
Me: Ok so you don't know how to do your job.
Dev B: Here, put the React SPA in a Docker container and run it on some cloud its easy...
Me: But all I need is still CRUD, I dont need to scale it, I dont care if its isolated, it should have been 100 lines of code
Dev B: its not best practice, my team lead agrees, here is a medium article, get with the times, Docker run everywhere, etc.
Me: But what I need to connect it to internal resources and use DNS?
Dev B: load balancer! If only you have an AKS cluster!
Me: We work in a 25 person company that needs some very basic CRUD. It will never scale past one deployment, ever, if it does, dont worry we will pay someone to do it right because we will be swimming in money.
The problem is that people are starting to not know how to do simple things. So you get grumpy admins losing their marbles over the complexity of this BS. Most of the time things could be fixed faster, easier, deployed and managed easier using simple technologies that have been around for 20+ years.
But lets wack it with the Python, Docker, React, NoSQL, GraphDB, K8S hammer because...? Not only that, the juniors don't know multiple technologies anymore they just know X. So sysadmins complain that devops is pushing more roles on them, while I have developers that have written full Node.js sites and dont know that IP or a Port, or TCP, UDP, is...
So basically, same shtick different decade.
I sympathize with the OP for single developers or small teams (by small, I mean a team of 5 not in a larger corporation), but whenever you have more than a single team, you want to keep system management overhead low. Unifying around a single paradigm like gRPC and Docker containers means that not every team will have their own bespoke chroot configuration and you won't have to retune an HTTP client every time you interface with a new service.
I think there's in general too little representation from small-scale or indie developers. I spend a lot of time working with the Gemini protocol outside of work and I appreciate the simpler approach that the protocol takes. I would love to see these sorts of stakeholders have a say in greater net architecture as well, but let's not pretend like this is all complexity for complexity's sake; this stuff is needed at scale.
Such bullshit. Since when we use UDP for a CRUD service ? Since when SCTP is even considered outside of the telecom world?
HTTP and Rest-like API have been able to handle properly simple CRUD API since 20 years without problems, way before gRPC was even a thing.
gRPC has its usage for large, complex API that required a proper RPC framework. But in 99% of the case, yes it's an overkill.
There is the problem. "Twenty years old? How can that be any good" is a common POV in my experience
You're making a strawmam out of this. I'm responding to the following:
"while I have developers that have written full Node.js sites and dont know that IP or a Port, or TCP, UDP, is..."
But thank you for the insult.
> HTTP and Rest-like API have been able to handle properly simple CRUD API since 20 years without problems, way before gRPC was even a thing.
Hm, what are you talking about? Are you talking about the client? Are you talking about a load balancer? I don't think anyone is proposing that a simple static blog use gRPC to serve content to its users.
> But in 99% of the case, yes it's an overkill.
Compared to what? I'd argue that CRUD over HTTP is wrong. In fact, folks have been writing CRUD over TCP for ages. Why do we need to massage CRUD syntax over HTTP when we can just stream requests and responses over TCP? In fact, TCP predates HTTP by 20 years, so if you're using the historical argument, raw TCP is even older. How much bullshit is there around AJAX and Websockets and HTTP multipart and HTTP keep-alive when we're just trying to recreate TCP semantics?
So why are you drawing the line at HTTP? Seems arbitrary to me.
It's probably acceptable to not learn the details of what you're doing, as hardware is cheap and all that, but it also constitutes a glass ceiling for how much the organization learns. That's a bigger problem in the long run.
I am not aware of why it was chosen to retire that and separate the website into its own service - maybe there was a good reason, so I won't comment on that.
However the approach they (or rather some frontend developer) chose was a React front-end (fair enough) with a Node.js backend that translated between the REST API and GraphQL (WTF).
The existing API was served on the same domain so the new React website could've directly interacted with it without any problems. But no, this guy wanted to put "GraphQL" on his resume and so introduced an unnecessary extra component and potential point of failure that the company now has to support.
I'm a bit confused as to how it's become so popular for normal development when it seems to have a lot more boilerplate and setup than a simple REST API.
Do you really, _really_ need to support queries? 99/100, I'd guess no, and thus constraining people to a more defined interface and access pattern is simpler.
Meanwhile the API returns highly structured data perfect for a RDMS, but we dont know how to query SQL without an API in the frontend. So welcome to my hell.
The former doesn't need explaining but the latter can be explained as developers being concerned about requiring the benefits of GraphQL in some uncertain future and decide to include it from the start, even though in most cases they end up never actually reaping the benefits while still being plagued by the extra overhead of using that technology.
The head of the company was pushing to modernize our process flow via software meant to drive manufacturing. We had contractors on site from three different companies who were each using us as a test-bed for their software/manufacturing integration. Every week or two they'd plot out some new data they'd need to move from sales to engineering, manufacturing, QC, etc.
In our case, having GraphQL as a translation layer for the sales website saved everyone involved time. However, I can also see many scenarios where that wouldn't have been the case.
It definitely comes down to using the right tool for the job. Knowing how to identify which tools fit and which don't is one of the skills it's very important to help new devs develop.
There's a simple solution to this kind of problem: let the resume driven developer use his skills - fire him.
Looks at a specification on the desk Yep. We'll keep the first and last page. Everything else is gibberish. Next!
Edit: Looks like a few people disagree. Perhaps they might like to explain why I saved a bunch of companies around 100k in AWS fees just by pruning their server requirements? No? And yes, one of the projects was a simple set of web forms that required 7 servers to run. We pruned it down to 3 and that was just for redundancy and load. That's just one example.
If you disagree state your why.
> I saved a bunch of companies around 100k in AWS fees just by pruning their server requirements. One of the projects was a simple set of web forms that required 7 servers to run. We pruned it down to 3 and that was just for redundancy and load.
EDIT
and if that thing about cutting down the specification really happened, just tell the story – no need to wrap it in a performance piece:
> I've actually had people come to me with specifications so over-engineered that the whole 20-page doc could be simplified down to a single page without loss of functionality!
EDIT 2
and i'm not saying "never use hyperbole"! just don't make it the whole point of your comment :)
Amusingly, they eventually did "scale". They started selling through WalMart in bulk, and accepted their fate as yet another nutritional drink, on the shelves alongside Ensure Plus and Muscle Milk. Their "tech stack" was irrelevant to that.
I remember working on an simple CMS site, amongst other things, for a pretty large company. When we begun the project, we were tasked with re-purposing an overworked Plesk instance to host the site. We eventually managed to do it, but then found the small amount of disk space that was left to us was getting chewed up by logs.
So I reported this to my project manager, suggesting that we procure more HD space. I think I said something about us 'running out of memory on our hard drive'. The PM promised to feed this back to the client... A week later, our PM said that he and the client had resolved the issue. The client will pay for a new server with a stonking 96GB of RAM!!!
That ought fix our 'memory' issue, right!!?
I mean, it also came with a 1TB HD, a second box for redundancy, dev time for migration, and additional dev time for a switch to Journald, or Logrotate, so I wasn't rushing to point out the misunderstanding... Working with the second box over RedHat Pacemaker was all new to us though, and a complete PITA.
But it was also another 'feature' to sell back to the client. They loved the sound of that. A site that couldn't go down... We made it work, but there was absolutely no real technical need for a Pacemaker cluster and 96GB*2 of RAM. It was just a simple CMS backed site.
Occasionally, that's where such complexity comes from. Not the developers themselves, but some loose cannon of a saleperson. That said, these 'sales people' may even be developers themselves. I often think that's a large part of how and why questionably complicated software exists...
I have 100 different devs telling me "do this thing, it's easy" and it's easy for them, but now I have 100 different things to think about and make work together and devs never think about how their tiny thing fits into the big picture.
99% of web apps could be pure HTML + CSS on the front end, and Flask or something on the back end, talking pure SQL to Postgres. Really. That's all you need. And probably 99.99% of those apps that run internally to one company.
It also enables pushback along the lines of "your container doesn't run, and it's 100% your fault, as it is supposed to contain all its dependencies". So developers have unwittingly played into the hands of ops there ;-)
Also if I was gonna complain about someone I wouldn't complain about devs, I'd complain about team leads, engineering managers - the folks who should be guiding devs - and, the people at the business level who choose the team leads / engineering managers.
The devs are at the bottom of the food chain; it's not their fault.
GraphQL is unfortunately still part of a low grade hype cycle where more and more companies are using it because "it's the next REST". But if you don't have a real need for the advantages it offers (for example a mobile app whose client side requests you can't easily change in sync with your API, or a data model that really is graph-like), it can be more cost than benefit.
React on the other hand: I've lost count of the number of small prototypes I've started in vanilla JS, thinking "this time I can just roll it myself", then ended up pulling in React. Even the simplest things are easier and simpler with it, and hugely complex applications are also easier and simpler with it. It has a fantastic API surface.
please note that what's "unnecessary" for you is not necessarily unnecessary for others.
this is an important point.
i guess what these projects need is a way to communicate explicitly the costs of maintenance and developer mind-share when adding more complex features to their product.
Because especially as small dev team it is difficult to figure out these costs during a short evaluation of the software, is not easy to pick the best tool for the job unless some members have got experience in some previous job with these tools.
Consider that "unnecessarily-complicated" and "unnecessary" do NOT mean the same thing -- particularly in this case.
most people don't mean there is a simpler implementation that has the same feature set and execution properties.
Most people mean it has features i don't need, hence it is too complicated
Redis also comes to mind.
Docker seems to be optimal for medium size teams. Configuring a server the old way is easier than docker for 1 server but less so for when you have 10 or CI/review apps.
Hah, with Google? Right.
That said, I also see the opposite when this gets some traction with "normal users". The Abseil issue was fixed, and the TensorFlow 2.0 release did make TensorFlow more usable by smaller teams in many situations.
I think there's a big swath of people adopting fancy tools to try to solve problems they don't have and won't have for years, but I don't see that having destroyed the ability to do things the easy way.
Pulling the snark would have resulted in more people getting the main point.
Pull the snark out and there will remain any element of this post that kicks off a violent HN centithread that conveniently avoids the core issue. You know it, I know it.
In fact the author liked etcd in the time period when it was directly associated with the "bullshit" project. And even mentions "but that doesn't really matter".
Bashing K8S is at least related to the main point.
However, I'll also point out that focusing on the snark and ignoring the rest of the article is also a choice made on the part of the commenters.
Given a choice, I personally prefer to take the strongest and most charitable interpretation of someones ideas before choosing to criticize them. At least the discussion is interesting that way...
Or resulted in very few people reading it at all.
If anything, I hope that we can recognize that there has been advancements in computing. Back in the day you would be open to attacks left and right. It was the wild west back in the day. Today, you can take that tiny webserver and put it in a VM, and voila, now it doesn't matter if anyone breaks in. As long as you didn't add the SSH key to github or anything like that.
We can have complex programs, as long as every-man doesn't have to know the details behind it, or implement it or risk losing access he previously had.
I think new protocols that are extremely hard to implement falls under that, and I don't think Qemu or Linux does, as nobody loses anything when Linux adds new complex features.
We have to make sure, at least, that we continue to support HTTP/1. I think we already lost a lot when we moved away from IRC.
There's an underlying presumption about computing progress that has weaseled its way into your thinking, and the devil of it is that there are vanishingly few people willing to question it (can hardly blame them, given this thread).
Nebulous philosophical arguments do explain things. Quite readily, actually.
Technological Manifest Destiny:
1. Technology is value neutral and will therefore automatically lead to great outcomes for everyone.
2. If we delay the rollout of new technology than we would delay the distribution of the inevitably great impacts for everyone.
3. History is generally uninteresting because the past has nothing to teach us right now.
(https://www.zachpfeffer.com/single-post/2018/11/27/Transcrip..., ca. 37 minutes in)
If you wrote your simple HTTP server well, then not really.
> you can take that tiny webserver and put it in a VM, and voila, now it doesn't matter if anyone breaks in.
1. It might still matter.
2. The virtualized/containerized version could well have its own security issues.
3. The virtualized/containerized version is heavy and expensive (relatively).
4. Why did you move away from IRC? Come back... :-(
This is a dangerous viewpoint to hold. Containers do offer some slightly better security, but they are not a silver bullet or anything close. There are still lots of ways to cause problems whether the software is in a container or not.
But let me ask the question another way. What security do you gain by being in a container instead of on a server without a container?
I'm not a container guru, but... this is obvious, right? You have to break out the container, which is an extra layer. In order to get root on the server, you need:
• Without container: application vulnerability (to gain access to the OS) + kernel vulnerability (to gain access to root)
• With container: application vulnerability (to gain access to the container) + container breakout vulnerability (to gain access to the OS) + kernel vulnerability (to gain access to root)
You make it sound like I'm missing something — what am I missing here?
Edit: are you talking about CVEs like this one (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-3215)? I guess I'm assuming that a container vulnerability will give an attacker less to play with than a kernel vulnerability.
Most of the bad stuff you can do when you take over a web server doesn't require root access, just user level access, and you get that whether or not it's in a container.
If you have an application RCE and a kernel vulnerability, what difference would a running container daemon make?
They are not an additional shell of defence, they do not reduce your blast radius.
Could you explain what you mean by this? I'm still not getting it.
Here's the example I have in mind: I have one application listening on a TCP port, and another one listening on a different port. I don't want the first application to talk to the second application. If they're just two processes on the same machine, they can both see each other, and an attacker who manages to exploit an RCE vulnerability gets access to the port "for free"; if they're both in containers, then they can only see the ports I specify, and it's really easy for me to specify them.
This literally does seem like an additional shell of defence to me! What sort of attack would it make worse?
As for your example, there’s no difference between that and OS mediated mandatory access controls, separation of uids, chroot etc. Those capabilities are present irregardless of whether you’re using containers.
Anyone selling you on containers as a security measure is pitching snake oil. They are a resource allocation & workload scheduling tool. They may help applications avoid inadvertently treading on one another’s toes e.g. avoiding library conflicts or easing the deployment of, say, QA and showcase environments on the same metal, but it’s intrinsically a shoji wall.
There’s even a cultural downside, since developers may make exactly these flawed assumptions about runtime safety, or hard-code paths for config files rather than making them relative to deployment location, make assumptions (implicit or otherwise) about uids and so on, i.e. containers can breed sloppy dev practices.
> Anyone selling you on containers as a security measure is pitching snake oil. They are a resource allocation & workload scheduling tool.
I agree with both of these statements — and I suspect this is where it all falls down. When you put all your applications in containers, you are not done with security, no. Nevertheless, I've found great value in having one file per version per application, being able to upgrade my dependencies one-by-one, automatically rolling back in case of failure, and taking advantage of the tools in the container ecosystem. With all that, the security features such as isolating filesystems and network ports are just the cherry on top. Before containers, I was thinking about how to give my applications limited filesystems to work with, but along came Docker, and I didn't have to think about it anymore, because I'd already done it.
This is why I feel so out-of-touch with many of the commenters here. I'm surrounded by people decrying containers for security reasons, I want to defend them because of the many benefits they have given me, and I think the people preaching some kind of True Security (where everything is 100% perfectly isolated) aren't taking this into account. I feel like I could take your comment — "Trusting the isolation like this is absolutely misguided" — and apply it to any part of the stack. You have to stop somewhere.
I've seen the salespeople. They definitely exist, but I don't think they're here on HN, and they're more likely to be learning than lying.
And developers making flawed assumptions? You know this didn't start happening with containers!
Don't underestimate your responsibilities as an active participant online and operator of a globally reachable computing resource. Like driving a car, if you're not fully qualified and alert then you can endanger yourself and others. This is how PHP got such a bad rap.
The fact that the container deployment tool has helped you configure security elements (such as limited filesystem access) is nice, but those elements already existed, along with the tools to manage them. Containerized software deployment did not invent them, and if you're getting procedural benefits from containers, that's great for you, but that's all it is.
To be fair, more security by additional tooling adds a vulnerability in human error, but that's true of all the tools.
> Containers
These are not the same thing, and have different security guarantees.
Some of this is an illusion of simplicity.
FCGI is an automatic "orchestration system" - it starts up more instances as needed, restarts them if they crash, shuts them down when they go idle, and restarts them once in a while in case they leak some resource. It's like the parts of Kubernetes you really need.
Go is an OK language. Not a great language, but an OK language. It comes with almost everything you need for server-side applications. Also, the "goroutine" model means you don't have to fool with threads vs async. If a goroutine blocks on a lock, no problem; the CPU gets redispatched. If you need a million goroutines, that works, too. None of that "oh, no, the async thread made a call that blocked and screwed up the whole system".
MySQL/MariaDB/Postgres have been around for decades, and they work pretty well. You can have replication if you need it. (NoSQL seems to have come and gone.)
Get the stuff at the bottom right, instead of piling more cruft on top. It's cheaper.
As a solo open source developer who maintains a popular OSS project, this is 100% the case. Sometimes it feels like big corporations are actively working against us. It's almost impossible to get any coverage at all on any kind of media. No matter how much organic growth your project has.
My project has been growing steadily and linearly for almost a decade. Now it has almost 6K stars on GitHub so you'd think that this by itself would draw some attention? Not so.
It's almost impossible to find it on Google search results. Most users find the project through direct word of mouth from other users of the project. We've never received a single consulting contract from any corporation.
However, i’m not quite sure what it does (it obviously does something really well, but I’m probably not precisely the target audience)
It might help adoption and re-share value if your readme starts with some problem and shows how your tool solves it. The cadence/temporal people did this really well: https://www.temporal.io/
It sounds like you’ve built a useful piece of software. However, you don’t get handouts because you’ve built something. Those who are successful actively promote themselves and their products. There’s a reason companies aren’t comprised of just software engineers.
> Funding and adoption (in the extreme, some might say hijacking) of open-source projects from large corporations dictates the direction of all major software components today
This is because large corporations are the biggest users of these pieces of software. Its literally that simple; money only plays a smart part of it.
Etcd's biggest user is Kubernetes. I would estimate that 2nd place isn't even close. I would estimate that Etcd wouldn't even exist today if not for Kubernetes. The direction of open source projects is usually some approximation of a democracy; its users contribute the code they need. If they need gRPC, it gets added. gRPC wasn't added for no reason, or because politics; its a sister CNCF project, and would increase the performance of the API surface, which greatly benefits all of its users.
Lets do a little experiment: Go start an etcd clone. Just go do it! The code is still there! It would be four clicks in the Github UI. Strip gRPC, add the HTTP API, and maintain it. I'd bet a thousand dollars the author won't do it, for two reasons. First, the author hasn't actually used etcd. I know this because, as I said, startlingly few people outside of Kube use etcd directly, and the author isn't providing any substantive technical reasons why the gRPC change was bad; just philosophical ones. And Second, because maintaining large open source software projects is soul crushingingly, destructively, enormously hard. Its way too easy to armchair-quarterback decisions like these based on your own personal flavor of ethics, then cherry-pick the evidence you want to say "corporations are evil! they destroyed etcd by getting rid of HTTP!" Meanwhile, people out there do develop it, and use it, and service billions of requests, and deliver value to their users, and sometimes, turn off their computer and cry because despite all of the hard work and positive results people with no skin in the game still pipe up to say what they're doing is horrible.
Did the author join in on the discussion when these changes were made? Did they voice their concerns? Or did they just read about it after-the-fact and say "jeeze, that piece of software I played around with two years ago really took a turn, I'm going to write a post about how evil corporations are."
Are you making a case in this package that complex open source projects are impossible unless completely controlled by very large, highly opinionated companies? That can be proven wrong by endless examples.
> Did the author join in on the discussion when these changes were made? Did they voice their concerns? Or did they just read about it after-the-fact and say "jeeze, that piece of software I played around with two years ago really took a turn, I'm going to write a post about how evil corporations are."
You don't know this to be true, so why make this accusation? Why not address the argument as it is instead of inventing ad hominems?
Its startlingly easy to write five-hundred words. Its startlingly difficult to accomplish what the etcd team has; I therefore give the benefit of doubt to etcd.
I don't fully understand why etcd switched from HTTP to gRPC. It seems weird to me. I wouldn't do that for any project I run. But I'm not going to write a blog about it. I'm not going to get angry about it. I'm not going to spin a conspiracy about ex-Googlers spreading gRPC everywhere they can. Instead: there are a ton of people developing and using etcd, who aren't me, who know what they're doing, and their results speak for themselves; They probably made the right call.
> You don't know this to be true, so why make this accusation?
Those are questions. But, yes, my tone was accusatory. Generally, if you have skin in the game, you don't write posts like this. That's why having skin in the game is so important; even if you lose the fight, you're left with positive remnants of the work you've done and the people you've interacted with. There are positive ways to have a negative discussion about technology. Instead, this article goes obscenely negative about etcd based on one thing the author disagrees with, then spins it into being negative about Kubernetes, Electron, Docker, corporations, and finally goes personal by directly criticizing the "expats from a megacorp ... who just want to build big ungainly architecture"
But also, you can look at their github and see their lack of contribution to etcd, if you really wanted to go full-stalker and demand evidence.
That's a VERY confusing comparison and conclusion to draw from it.
If you are that tied to your work you may be treating it as a child, and as a child, your moral reasoning will proceed with the underlying premise: "what is good for the child is good", leading towards a parenting of the project ahead of other needs.
But a software system is not a child, in fact. It's just an expression of ideas. "Skin in the game" signals that you've been dragged into being a parent. Is being a parent for software the right thing? Perhaps, if the software's premise draws upon a strong justification. But most of these tools have not been emerged from standalone justifications, but from solving something else, which neatly ties back into the author's argument: the corporate needs are what are complex. And here attachment-based morality recurses: if the corporation is the child and you are tending to it, once again, you will put it ahead of other needs, and hence will develop the justifications for software complexity.
But if we zoom out a bit and look at the world generally, it's operating from a point of indifference: if the output of the corporation is pragmatically convenient, it is good, if it presents an obstacle then it is bad. And ideas - and hence software - that persist in the indifferent world, outside of the attachment relationship, are the ones that survive.
Which perhaps means that all of it is wrong, which isn't a very stunning conclusion if you still have to work with it. But that is philosophy for you.
At a (very uneducated guess) I'd imagine that the query payloads would be smaller with gRPC since it's binary which would save money when running in the cloud for things like traffic egress, cross-zone, or cross-datacentre billing. HTTP can be unbeliveably verbose.
It's also probably quicker to deserialise which means you get more bang for your buck on your machine spend.
> It's also probably quicker to deserialise
These are pretty educated guesses :-). This kind of reasoning has been behind a lot of Google initiatives, such as protocol buffers. If you're slinging enough data around, smaller payloads and faster serialization/deserialization does have a significant payoff. Once you're marinated in that mentality it probably sticks.
gRPC (Google RPC) was absolutely added because of politics. Yes, it's performant, but let's not kid ourselves that gRPC wasn't picked because it falls nicely into the orbit of Kubernetes and the Google ecosystem. It's also why etcd can even threaten to remove support for HTTP+JSON (!!!) in the API, when it costs nothing at all to keep that support. This is exactly what the post is bemoaning.
> Did the author join in on the discussion when these changes were made? Did they voice their concerns?
Did you? Did anyone outside of a few people in a SIG or an RFC? This is also the complaint the author is obliquely getting at. Google may be 95% of the etcd _volume_, but they are basically 0% of the institutional users. That a small group of people can push a project in a direction that uniquely benefits them while adding an incidental complexity tax for everyone else is worth complaining about.
These are all projects under the Cloud Native Computing Foundation; its Platinum-level sponsors include: Alibaba, AWS, Apple, ARM, Cisco, Dell, Fujitsu, Google, Huawei, IBM, Intel, JD, Microsoft, NetApp, Oracle, PaloAlto Networks, Red Hat, SAP, and VMWare. Also included are 20 Gold-level Sponsors, 415 Silver-level sponsors, 3 Academic sponsors, 13 Non-profits, and 100 End-user supporters. That's a total of 570 organizations with a vested interest, and voting rights, on the direction of their projects.
But Google does a lot of technical oversight, right? A non-zero amount. Among the eleven people of the CNCF Technical Oversight Committee, Google has ONE representative (Saad Ali. Learn their names. These aren't just faceless mega-corps we're talking about; they're real people). Microsoft has the most, at two. Also represented is Apple, Intuit, Docker, American Express, Aqua Security, Lyft, Rancher, and Alibaba.
But, ok, Google "sneaks in" a lot of code, right? They're super-evil, that's the message we're trying to convey here. Of the top ten contributors to etcd [1], there's only two people who seem like they work at Google. The top contributor, by far, works at AWS.
Alright, fine. Kubernetes was created at Google, this much we cannot deny. Started by Brendan Burns (now at Microsoft), Joe Beda (now at VMWare), and Craig McLuckie (now at VMWare), all Google engineers a decade ago. Yeah, the thought leaders behind the project sure are still making that sweet sweet Google money, sure seems like it.
Its just tremendously incredible to me that anyone still believes Google has a large say in CNCF projects.
And your condescension and repetition would show that you don't know what you're talking about.
These were all--100%--developed on by Google and moved into CNCF, which is uncomfortably influenced by Google. Big providers have made peace that they lost the standards war. (And while etcd was not originated from Google, its adoption into kubernetes was obviously influenced by its use of golang.... developed at, where was that again?)
Kubernetes and gRPC started at Google, were exclusively developed by Google, and then much later moved into CNCF as a form of legitimacy.
If you have never tried to get a change into gRPC, I would encourage you to try. It took 3 years for Google to allow an upstream change that would not require rebuilding PHP protobufs from scratch on every request, because it would change the way their C library did things.
Google knew they couldn't catch up to the AWS offering by selling service alternatives, so they intelligently and astutely zagged, and went all-in on an "open source cloud" approach.
Kubernetes is the Trojan horse - it gives people hope that they can develop with open source, and be cloud agnostic. But when the complexity overwhelms and drains them, they invariably look for a "managed" kubernetes solution to take the load off. And, wouldn't you know it, Google happens to offer a managed Kubernetes solution! Sure, so does AWS, but EVERYONE KNOWS (whether you like it or not) that Kubernetes was developed by Google, so surely they would be the ones best at managing Kubernetes ops in production!
Over and over again.
Say I want to fork etcd or build a greenfield alternative - I now have to compete with (or support) the Hivemind's preferences in direction/progress. Who's to say their way is the "best" way?
I think this tension is much greater than it was 10 years ago.
Now that corporate interests dominate in many corners of the FOSS sphere, this benefit of FOSS is diminishing. Which is a shame, because it's a big reason why FOSS was so successful, and why so many old commercial projects collapsed under their own weight, even though they always had more features than their FOSS counterparts and thus theoretically should have only grown their user base.
"The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely."
This person is not looking for a conversation, but looking for a fight. He seems to be angry and cannot direct his anger to "make good art" (Neil Gaiman - Make Good Art), instead trying to insult people and pushing them to a place where their emotions takes control and can be dragged into the mud.
There are some irony on the text, but to understand this is very naive.
The critic is at malicious/dumb people who latch to technologies without understanding them.
For argument sake, see how many people are defending coreOS etc, and how they claim to have decided to like it based on their superb expertise, yet they will likely fail to explain to you chroot or anything else containers are based on. Or how they will proclaim that LXC is pure garbage and Docker is better (i will not explain the joke, let them downvote :)
Bottom line: we are being sold bad ideas from the past, in new clothing, by the incompetents/malicious people, which megacorp is made of.
Read my comment again and you will notice there's not a single comment against or in favour of tech. Not even an adjective next to tech, only to people.
The comment is exactly that some people can't tell the difference between two completely identical techs, just because they buy the marketing of one of them, despite it being a brand on top of the first.
Here is one from me with the solo-dev/small-team in mind - https://github.com/imaginea/inai . Good or not I don't know. It came out of some reading I was doing and I'm having fun with it due to the "deploy first and then dev" mindset it encourages.
Do you work with Kubernetes in a professional setting?
Availability heuristic: large companies get more press. Partly because they're large and therefore more likely to attract readers for stories about them, partly because their scope is vast and so journalists know about them, and partly because they intentionally seek press for their work.
The scratch-an-itch period of mass-market OSS is long gone. If indeed it ever truly existed.
“Under the radar” comes a new generation of better systems and components that learns from these learning vehicles of computing and will change how we deliver services fundamentally, but without k8s and Linux we would never have gotten there. Open source RISC-V co-evolved with seL4, WASI, webassembly and NATS comes to mind, but I’m sure the future will take its own path.
K8s gives us declarative infrastructure with eventual reconciliation. That's massive, and there's nothing else around that really does that.
I have a hard time taking anyone sesriously that makes the case that k8s is bad that hasn't fully grokked the above.
One is for making it easier to manage large, complex infrastructure. The other is as a quasi-OS for containerized software.
Instead of using a new technology to move infrastructure complexity away from developers we now run development environments on k8s, piling even more complexity and layers of abstraction on.
Not sure what you mean by declarative infrastructure but eventual reconciliation has been implemented in at least one other project (now retired) called Apache Aurora[1] which runs on Apache Mesos[2].
Twitter ran (and probably still runs) a combo of these to huge scale and great success so I don't think it's fair to say that there's nothing else around that does that :).
K8s is great, but we should be careful not to rewrite history.
>K8s is great, but we should be careful not to rewrite history.
Why's that? That's kind what tech is all about, no?
Because they chose to make them open-source (not out of altruistic motives but out of the same profit seeking perfectly good business motives), these solo developers (who are also profit seeking) can benefit from those projects for free and can afford to build to serve lot more users than they would be able to otherwise.
I don't get the entitled attitude of the author. If he wants a 'simple' system by his standards, he can build one easily enough with the modern tools like golang, node.js and rust.
Btw, the web development tools were never simpler than today to operate a safe and secure production web application to serve any number of users. I'm saying this with about two decades of experience doing the same.
Of course, you could argue that the people who made it more complicated in the first place could/should have forked instead of taking over, but that ship obviously sailed a long time ago.
I think it was a good move for an infrastructure piece like etcd to add gRPC. Now I can just grab a generated client in a language of my choice. There's certainly valid critiques of gRPC / protocol buffers but I've found things like gRPC and Thrift to reduce complexity in applications that use them, because there's a thousand interpretations of REST out there, not to mention the crazy things people do with 'simple' HTTP, such as long polling, which introduce implicitly complex semantics into something that is on the surface quite simple.
But I also think that the original author has a point: for many solo developers it's an extra dependency that they have to learn, and they get virtually no benefit from using grpc because they aren't performance-bound. They are simplicity-bound.
Simplicity wise my favorite scenario is where if I'm using Python I can pip install an interface (or maven, or NPM, etc. etc) which is the versioned, supported interface for a component. REST + Swagger can help a lot there. So can gRPC. But I can also get how your favorite scenario might be to explore a REST API and build your own client.
If you're talking latency, to this day gRPC in dynamic languages is still slower than most fast json implementations in _latency_.
It is simply not correct to talk about "performance" this way.
If you care about CPU allocation and not burning 70% of your CPU on JSON parsing, then sure. Most companies don't care about that, and I/O is 3 orders of magnitude a bigger problem than in-process parsing for those problems.
Obviously these aren't hard and fast rules, but Google can't just spin up 10x more data centers to run their resource intensive stuff in something other than c++ or Java. They need the efficiency in a way solo developers or small teams usually don't.
But you can't curl the state of your infra component without creating a program and downloading the client artifacts. This is a very big step backward from an operability standpoint.
> because there's a thousand interpretations of REST out there
...but there weren't a thousand different interpretations of the etcd API out there: there was exactly one. And now there are two, and the one with the most utility (that can be used without generators and in languages where gRPC support is spotty) is being diminished.
While I agree with you that there could be design consistency arguments in the abstract for gRPC vs REST as a global architectural pattern, none of those applied to the concrete artifact of etcd client<->server communications.
Especially since I believe grpc does take steps to mitigate some of the RPC issues you'd have with, say, Thrift. For example having the REST gateway so you can support the curl scenario without any extra work. (For example when building my own Thrift-type services I would do extra work to have a mini REST API for status and other operational curling uses)
As to the second part of your argument about there now being a second API, and how that is not necessarily a good thing, I quite agree and can't really speak to the etcd use case, other than that I don't think it's some property of 'Xooglers' to do such a thing to an open source component. I've spent my career using open source components: I've had lone developers who change their APIs willy nilly to fit the pet peeves of their own organization; I've had large companies own an open source component and have it be very disciplined about version migration; I've had lone developers be similarly disciplined; I've had companies make open source projects really complicated.
I would agree with you except for one thing--it's very clear that's a second class citizen. There is this totally extra thing to support the original use case of etcd, and it comes with all the extraordinary quirks of how gRPC protobuf -> JSON serialization works (a strong term in my opinion).
It is very very hard for me to not see an unintentional conspiracy to push this very bad standard of gRPC. It does NOT make 99% of real-world programs faster. It does not make operations easier. It is not even a "data" format at all: go try changing an enum in downstream systems.
The standard is bad. Objectively, terribly bad. Someone somewhere on HN said that "Google was a ball-pit for gifted children." I kinda laughed at that, but really, after k8s and protobuf3, I won't even grant "gifted".
Google is a ball-pit for autistic children who can be funded by their monopoly on search. I'm tired of dealing with their protocols that make everything worse.
This made me laugh.
You can if you enable server reflection[1] and use a tool like grpcurl[2].
[1]: https://github.com/grpc/grpc-java/blob/master/documentation/... [2]: https://github.com/fullstorydev/grpcurl
Not available in any repositories and largely unknown to general masses (much like gRPC itself).
Personally I like gRPC and sometimes use it in projects. But the author is right — attempting to gradually replace an existing HTTP API via gRPC API is like replacing horses with genetically modified cows. You need a very... corrupt mindset to ever attempt anything like that.
To reemphasize the problem with grpcurl.... it's _still_ static. You must download the artifact. It's like needing a different phone for every person you call (protobuf artifacts) instead of calling different people on the same phone (same artifact, different address and data in payload).
You can still do this because the gRPC API is also exposed via REST.
(I use it)
No, you can just grab a generated client in a language of your choice if that language happens to be supported by protobuf.
This is a massive regression from the openness of HTTP REST APIs.
But it's trivially true in the case where the provider does so wish (as it would be true of any protocol with a fully-functional HTTP gateway infront of it), and it's untrue in the case where the provider doesn't.
And the provider has to do extra work:
"[offering an HTTP gateway] required adding custom options to gRPC definitions in protobufs, and add an additional container running this reverse-proxy server." ( https://wecode.wepay.com/posts/migrating-apis-from-rest-to-g... )
This still seems like a massive regression in openness to me.
Not so. The article, imho, presents all of its views as unabashedly opinionated views of the author.
The complaint that simple, lucid and well-fit functional designs are getting hijacked by large-corp vested interests has been adequately taken up elsewhere in this thread.
In this case, the vested interest are interested in making it as easy as possible for real world customers to reliably deploy their applications to the Cloud.
That's not an interest that favors unneeded complexity. The complexity comes from the need for a container orchestration system that satisfies the needs of large organizations. If you could satisfy those needs more simply, the incentives are to simplify.
Undoubtedly there is space to build a simpler tool with a subset of Kubernetes' features. And sure, you can pick at specific flaws in the way it's evolved.
But I'm not so convinced by the claim that the whole thing is a mess because tech employees like replicating the stack they are used to.
IMHO, there is a simple solution to this - release the software under the GPL license. Then no corporation will want to touch it.
As someone else mentioned, when a giant corp absorbs a free project, you can reasonably expect them to maintain it. If you GPL the project, that won't happen. But if it's a popular project, then if it's GPL it will attract more maintenance from people who can't free-ride on the corp's efforts.
(Note that the relevant measure of "popular" is not number of users, but number of people enthusiastic enough to actually contribute.)
I think where a lot of the pain comes from, and the article hints at this, is there are a LOT of problems that do not require a distributed solution, and while you can totally solve those problems with a distributed system, it's not really useful to do so in a way that dramatically increases complexity.
This is a known issue, and there are simpler models for these simpler cases. I think the problem is all the hype around k8s has driven a lot of square pegs into round holes.
To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.
There's just a reality that we don't really build these systems on top of a distributed operating system. They're starting with an platform abstraction that is already terrible imperfect for the job. There is a ton of pressure to get something out quickly and make it accessible to a broad set of developers rather than to do it correctly (MongoDB is perhaps the most classic example of this). I was building big data systems in the early days, and the whole mentality was that you were building tools that were riddled with flaws, limitations, and outright bugs, but in the right context would make capabilities accessible to a broad set of developers that were otherwise completely off the table.
The truth is, if developers were already experts at distributed systems, they wouldn't need or want most of these tools; a LOT of the value is in accessibility and having something that is "good enough" rather than correct. The products reflect that more than they reflect hubris.
What I do see is a combination of marketing and customer ignorance that does believe in silver bullets and belief that these tools don't have to be understood to be effective. It's an unfortunate byproduct of focusing on that accessibility over correctness.
The thing is that with free software the old software is almost never really long gone. It's still out there, free for your perusal. That is the beauty of free software, it's always additive process. But it does also put responsibility on the user to pick the patches they want, either themselves or by proxy. Ultimately you are responsible for the free code you are running, not some vague upstream dominated by fnaag et al.
I think, in the case of etcd, it would have been better for the k8s folks to fork the project and add whichever features they felt were needed to be able to use it for k8s. Instead, it seems like they swarmed the original project and added a ton of features that made it a behemoth of complexity rather than the simple tool it once was.
This would only make a difference if someone was willing to maintain the non-k8s fork. From what I understand, nobody was willing to take that on, so the k8s version became the only version by default.
Even if it's for the good of the quality of a product, it is not often easy telling your team mate that they wrote an unnecessary component that should just be deleted. I have done it lots in my short career (and in terms of engineering outcomes, it's been spectacular), but it's easier for me I think, since I'm more disagreeable than the average person.
Thankfully, we still have the "minor" components to ourselves.
Popularity has its downsides.
I don’t agree that this is a statement about complexity, so much as it is a rant about the author being inconvenienced by technology they personally deem pointless and unnecessary. If the CoreOS comments aren’t a sufficient red flag, the HTTP/2 ones really ought to be. Also, some etcd users debate their assertions about newer versions being more complex internally. On the other hand, its difficult to argue with overly cynical type attitudes.
Basically, if you want to argue for a somewhat controversial conclusion, you shouldn't back it up with even more controversial and unsupported (or weakly supported) premises. Particularly if "even more controversial" means "in disagreement with the vast majority of your industry."
It is important to note, too, that etcd is permissively licensed and anyone can run the old version in whatever way they please.
The final, and biggest, takeaway? Hackernews apparently has drank the koolaid and will reach for any convenient justification to write off an opinion they disagree with. Stockholm syndrome all through the comments. Conservatism and pragmatism apparently are no longer current in the cloud economy.
That's not really surprising, and maybe should be a lesson to the author. If you want to make a particular point, you shouldn't make inflammatory/controversial statements about other, only-tangentially-related things. It distracts from and dilutes your point.
For my part, I know one of the CoreOS founders, and while I have no opinion on CoreOS itself, it's really off-putting to see some random armchair quarterback on the internet shitting on the work of someone you know. While I more or less agree with the actual real point of the article, the unnecessary attacks really divided my attention.
The article starts off with a cheap potshot at CoreOS, followed by a screed about Kubernetes, and then finally, more than half-way through, we get an admission that the author has "digressed", and we get to the actual point. Certainly the point required some background about Kubernetes, but I can understand why people focus on the negative CoreOS and Kubernetes aspects of the post, since they consist of more than half of the content.
I'm mostly curious about which part of devops you're skeptical about, if it's tooling, or process, or the mentalities or something else.
Perhaps if the author wanted their key point addressed they could have tried focusing on it, rather than on an ill-considered spray across a group of largely unrelated bugbears.
4 is the problem. It's like joining a car club then keeping pushing to have petrol cars banned. It is really shitty to find a cool tool that you can use, then try to force remove features that existing users use.
If the economics of the situation is "large teams have to maintain tools for small teams and individual users," that doesn't actually scale well, does it?
why does cmake come to my mind?
I maintain a (/the only?) etcd3 library for Node.js[0], and used etcd extensively on my former team.
None of these things are new to etcd3 API. All of these are present on v2 as well[1], whose API the author extolls, or are built within clients on etcd's base APIs (e.g. there's no 'lock' API, only leases). However, with etcd3 we get stricter typing, better performance, and better semantics (e.g. watch streams and lease streams over polling) thanks to GRPC.
In general these rich APIs allow 'average' engineers to build complex distributed apps more correctly. I've built reliable sharding, hash rings, elections, and so on based on etcd's API--none in more than a hundred or two lines of code (more in Go, less in Node.js). All of these are classic hard problems that etcd makes easy. Sure, there's innumerable standalone services for each of these things, but often there's no need to take the cost of many tools when one would work.
I quite enjoyed the rantiness on a "being entertained" basis but it did rather work against effectively making the core point.
I've seen the trend toward complexity in other projects too, and it harms not just ease of debugging, but ease of hacking.
Take, for example, Swagger UI[0]
v2 was so simple. It was vanilla JS using jQuery. I, as an embedded systems developer, was able to easily hack it so it could read in the OpenAPI JSON from a database and I even added a little search box so you could filter down the APIs you wanted to see. Super fast and easy and worked just the way I wanted it to!
Starting with Swagger UI v3, it became... extremely labyrinthine by comparison. It was completely re-written in React JS and now I need a bunch of new tooling to make changes and everything was broken out into dozens of different modular files so I couldn't find where I needed to make a given change, also not to mention I've never used React so it felt like the barrier to hackability was dramatically increased.
I'm sure full time React folks love the new architecture because it's so much <cleaner/safer/scalable/etc>, but for me the change was extremely confusing and made the tool unhackable (I tried for a few hours to get it to do what I wanted, but it started looking like I was just going to have to learn all of React and I threw in the towel), and so I'm permanently stuck on v2 for now.
Fork Swagger v2 and make your own improvements. No one is stopping you and it's what open source is all about after all.
Web technology is its own beast. I invested a long time learning and mastering AngularJS only for all that work and knowledge to be flushed down the toilet over the next few years. Web tech has terrible ROI so that's why I "can't be bothered" to learn the latest one. I'm salty specifically about Swagger jumping on <insert latest framework> instead of sticking with simple vanilla JS that everyone understands.
> Fork Swagger v2 and make your own improvements
Yeah, that's exactly what I did...
> for a ~~bullshit~~ unsuccessful project called CoreOS Container Linux that was EOL'd several years ago
CoreOS was actually quite successful to me as an outside observer. It had decent paid user base as well as people using it without paying. It has showed people that Chrome OS can be used to build atomically updating host OS with a readonly fs for container hosting. It has inspired many other container hosts to come like RancherOS and Google’s Container-Optimized OS. Similarly, it was probably one of the reasons why Red Hat was interested in the acquisition.
Furthermore, CoreOS was EOL'ed last month; not several years ago.
I can keep going on why etcd API was switched to gRPC and what benefits this offers, even at small scale. But I don't think it's worth anyone’s time convincing the author otherwise. Based on their tone, it's clear to me that they have trouble with using software when things get a tad bit complicated. Usually, there's community decision-making behind these decisions, and they're often deliberated for months, backed with prototypes and data. I'm pretty sure the author doesn't care, however.
Calling CoreOS unsuccessful is a massive misunderstanding of the market.
Just because people buy snake oil, it doesn't make it a good product.
https://en.wikipedia.org/wiki/Container_Linux
(for those who don't know, it was renamed from CoreOS to Container Linux)
2. I've always expressed strong opinions in my posts. Sometimes this results in multiple upvotes, sometimes in multiple downvotes.
3. About being "unsubstantive" - I underscored the main concept/rationale for Core Linux, and linked to its Wikipedia page. You could argue that it's not substantive _enough_, but when you phrase it this way it's as though you merely dislike the fact that I repeated the inflammatory term from the article.
For example, we've gone a step further than CoreOS did and have a fully open-sourced update server, Nebraska (https://github.com/kinvolk/nebraska). We also generate a list of contents and licenses for each build. Here is an example from the most recent stable: https://stable.release.flatcar-linux.net/amd64-usr/2512.2.1/...
Times have changed and the very people he criticizes are infinitely better at branding and inspiration. There's a reason they're the ones gaining traction.
Like you, I want to be constructive, and I do agree with the author's core complaints. He just doesn't realize he's actively self-sabotaging and harming his own reputation.
Many reasons to switch from JSON to gRPC: * gRPC uses HTTP/2 which means you can concurrently make multiple requests on a single TCP connection, while on a typical JSON API which probably uses http/1.1, you can't. * Type safety of protobuf types. * Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane. * Client libraries are automatically generated, and not error prone. * RPCs are already optimized for bytes on the wire efficiency, whereas JSON is not. This matters a great deal as Kubernetes objects get large in size/quantity over time but controllers still work effectively by not spending too much CPU on encoding/decoding like they do on JSON. * Similar to the previous point, most json decoders don't reuse objects, so every decoded object is a new alloc, whereas gRPC can Reset() and reuse the same object while decoding/encoding. * Builtin authentication primitives (such as JWT/tokens or even adding TLS to client and/or server). * gRPC has support for interceptors which are like middleware functions you can inject to requests/responses on both client and server-side for logging, authorizing etc.
The list goes on, but something to note is that etcd was not developed by Google (and as far as I know, not by ex-Googlers). Both etcd and gRPC are owned by the same open source foundation, so it's natural that they make use of an existing technology.
Was a "JSON mapping for etcd's protocol buffer message definitions" the original "HTTP API"?
"A Critique of the Remote Procedure Call Paradigm" (https://pdfs.semanticscholar.org/e125/57a7582881a62040eee68b...)
"A Note on Distributed Computing" (https://github.com/papers-we-love/papers-we-love/blob/master...)
What are you actually talking about? An HTTP API by itself doesn't actually care about which http version is used, that's something only the HTTP server that serves the API should care about (nowadays, pretty much any http server in any language supports HTTP/2 and 1.1).
> Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane.
Even HTTP/1.1 has mechanisms for that, like long polling and chunk encoding (which allows infinite streams to keep a dual channel of communication open where each chunk can be treated as a message - with "headers" and all).
> Client libraries are automatically generated, and not error prone.
This is true, but we've had similar technology since the SOAP days... you could easily do the same with a XML-based HTTP API.
By the way, most of your points are against JSON, not a HTTP API per se, which usually would allow a number of formats , including XML and JSON at least.
> Builtin authentication primitives...
HTTP also has that when you include cookies and something like OAuth/OpenID.
> gRPC has support for interceptors which are like middleware functions
This kind of thing is better done by using the specific platform you're running on (depends on the language) so I don't see something like this as something desirable on the specific RPC framework you're using. With HTTP, caching, logging etc are trivial to do and one of the strongest advantages of using HTTP in the first place, so I think you're confusingly making a point for HTTP here, unless your focus is on the RPC side of things? In which case, you would have to make a case on why RPC is a better fit than HTTP for etcd, which you haven't.
So all in all, I found your points utterly unconvincing, but presented with so much conviction that you're actually right that I could not resist to respond (even if I don't want at all to get into a pointless discussion on the merits of HTTP VS RPC or JSON VS Protobuffers)!
For example, whole etcd API is located at https://github.com/etcd-io/etcd/blob/4c6881ffe4b3bae257c0720...
It is pretty straightforward and not too hard to understand the API. Designing APIs with protobuf makes things convenient and it brings lots of already written tools with it.
Since gRPC built on top of HTTP/2, we thought gRPC as easiest and performant way of writing a HTTP API with good defaults.
Yes it's easy to make that RPC call. But should you?
I've been off doing other things in the intervening period, but while I had my back turned the industry seems to have turned its back on REST and gone whole hog on RPC. Again.
At first I was thinking this was just internally here at Google, where protobufs and gRPC reign supreme. But it seems to have taken hold everywhere.
What did I miss? Why have we swung this way. Again. Is the pendulum going to go back?
I've heard this argument before, but how does gRPC itself cause this issue to manifest? I'm curious to hear what your opinions are on a better alternative, and how not to represent a remote resource as a local one.
But the fact that it presents the remote resource in an API which resembles a local object means that programmers often get lazy in the manner of which they think about these things. The REST semantic is supposed to make this more explicit.
A remote object is not an object in your program or even your computer. It's something you're taking from something that is computationally miles and miles away. Compared to the microseconds it takes to dispatch a local call, it's an eternity away, even on a local network of the highest speed.
Accumulate those latencies over thousands of dispatches, and trouble can ensue.
I am reminded of an observation from when I first joined Google, coming out of their acquisition of the scrappy awesome ads company I worked at (Admeld).
We had a little service that kept track of ad impression caps / budgeting. We didn't want to serve an ad a single time more than the customer wanted us to, etc. Serving many thousands of ads per second, a process distributed across multiple machines in multiple data centres, this is a bit of a tricky shared state problem. The people who came before me had designed a rather clever solution which used a form of backoff to trickle down the number of ads served as they got closer and closer to budget cap, and to synchronize this state across clusters (this was before there were Rafty services to make this kind of thing easier, BTW).
We had a bit of a show and tell with Google when we first joined. I wanted to know how they were handling this problem, since their scale was many times ours, so I asked the question and got a puzzled/annoyed look:
"Oh we just make an RPC call to our budgeting server."
Summary: if you're at Google you don't have to worry as much about these problems. You still do, but there's an insane amount of infrastructure and horsepower and an army of SREs to help make it happen.
So, yeah, my point is -- just because Google does something or has invented something doesn't mean it's the best way to do it, especially in a smaller more cost conscious organization.
What I don't see though, is how is making a (g)RPC call any different from making a REST call? Like you said, the REST call is supposedly more explicit, but at the end of the day, it seems more of a convention and some hard underlying difference. What's the difference between `httpClient.get("...")` and `grpcClient.foo(...)`?
I mean, internally at Google we have load balancing for grpc (I'm sure the outside world does now, too, but it was new to me when I joined) -- but load balancing HTTP requests containing readable JSON or XML documents, that's far more sysadmin friendly, wouldn't you say? Off the shelf infrastructure, nothing exotic, easier to monitor. Same goes for caching, for proxies, etc.
Being able to just stick a URL for a given resource in your browser, or hit it with wget/curl to read it, that's a serious bonus.
In general URLs follow conventions similar to those down by our Unix forefathers, when they designed the filesystem API. We are all familiar with this model. And in some ways REST done right is very Unix philosophy -- provide a consistent model upon which a bunch of little tools can interoperate.
I could go on... have to go put my daughter to bed tho
I don't think that's accurate... as the article claims (and it seems believable) it has only appeared in etcd because of Xooglers interfering with the project in the name of Kubernetes (also from Google). Or do you actually have examples of gRPC being used in many other non-Google-related projects?
This resonates with me, and I'd like to add HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses and have a strategy/priority when to push eagerly vs serve lazily; I'm even much more worried about upcoming QUIC (and DoH) because there's no impl in sight.
Compared to well-optimized HTTP/1 (e.g. using minified CSS and sprite-sheets), sure. Compared to most HTTP/1 deployments, though: no. HTTP/2 gives you tons of advantages "for free" that you need build-time processes to attain in HTTP/1. With HTTP/2, you can do "the naive thing" that'd you'd have done on the 1995 Web in Notepad, and it'll be the optimal thing.
Also, HTTP/2 means less OS packet-switching overhead server-side if you have ancillary connections (e.g. websockets) open against the host, since those also get muxed into the same carrier socket.
Also, mobile clients wake up less, because there's only one TCP socket to do idle-keepalive on.
HTTP/2 also means that TCP's Nagling has more to work with, and so is less likely to end up needing to waste bandwidth on emitting many undersized packets—it can just pack N short requests into the same TCP jumbo frame, since they're all going to the same place.
I would also point out an indirect "advantage": HTTP/2 makes it cheaper to serve ads proxied through the first-party host (as HTTP/2 flows) than for the client to hit the third-party ad servers directly. People can still block the ads/trackers either way, but served inline to the origin like this, the people who don't block ads, will get a better experience.
HTTP/1.1 pipelining partially mitigates this, allowing the client to "queue up" a list of all the dependent resources it wants from each socket; but it suffers from head-of-line blocking. Which sounds like some arcane thing, but in practice it means that big things might block the loading of small things. (The browser doesn't know how big things are, so it can't effectively schedule them; and the server must dumbly queue results up in the same order the client requested them, because that's the only way the HTTP pipeline's implicit flow sequence counters will match up.)
HTTP/2 is a full mitigation for this problem, since—even without a heuristic "prioritization strategy" for the delivery of dependent-resource flows—the "oblivious" strategy is still a good one: if you attempt to deliver all the resources in the queue concurrently; and you do so by delivering one fixed-size chunk of each flow per iteration, in a round-robin fashion; then you'll end up finishing delivery of resources smallest-to-largest—which means you'll usually deliver the most-critical resources first, no matter where in the dependent-resource queue they started.
Or, in short:
HTTP/1.0 = O(N) required roundtrips for a page with N resources.
HTTP/1.1 with pipelining = O(1) required roundtrips for a page with N resources (followed by O(N) bytes streamed half-duplex), but the page can still take nearly the same amount of time to become interactive as if it were O(N) roundtrips, because of effectively worst-case scheduling.
HTTP/1.2 = O(1) RTTs + O(N) half-duplex bytes for N resources, loading "intelligently" such that the page becomes interactive in O(log N) time.
The only things that kind of break this heuristic are:
• large single-file Javascript SPAs. These are usually just marked with the `async` attribute, such that the initial DOM of the page can first render, then be gradually enhanced when the SPA loads. But with HTTP/2 + ES6, you can also just not pack the SPA into a single file, instead relying on ES6 modules, which will each be individually smaller and therefore will end up being delivered first by the content-oblivious round-robin chunk-delivery strategy.
• web fonts, which are large and will necessarily cause a complete reflow of the page once loaded. Currently, browsers make a special loading-precedence exception for web fonts; though it doesn't matter as much right now, as they're still mostly served from third-party CDNs rather than as first-party HTTP/2 flows.
AFAIK, for regular web-page retrieval, these are exactly the total set of things that currently benefit from being server-pushed along with the first response; everything else just gets handled well even without server-push.
(If you're curious, server-push is really designed for the use-case of pushing secondary API responses along with an initial API response; to allow for GraphQL-like "single-round-trip resource-hierarchy/-graph walking" in a way that's more friendly for caching layers than comingling the results into a single resource-representation. It makes the most sense in a Firebase-like system, where un-asked-for server-pushed resources can be obliviously accepted and dumped into the client-side in-memory set of synced entities asynchronously to the parsing of the initial response; and then, once the dependency is parsed out on the client side, the client can discover that it already has the entity it wants in its in-memory entity store, and doesn't even need to make the request.)
No, the benefit of HTTP/2 is a lack of head-of-line blocking. Head-of-line blocking can be easily seen when big things block small things, but that's not what it is. What it is, is when something doesn't make progress because another thing is being waited for.
Imagine a multimedia container file-format where you can't interleave audio frames with video frames, but rather need to put the whole audio track first, or the whole video track first. This format would be unsuited to streaming, because downloading the first chunk of the file would only get you some of one track, rather than useful (if smaller) amounts of all the tracks required for playback. Note that this is true no matter which way you order the tracks within the file—whether the audio (smaller) or video (larger) track comes first, it's still blocking the progress of the other track.
HTTP/2 is like a streaming multimedia container format: it interleaves the progress of the things it loads, allowing them to be loaded concurrently.
This doesn't just mean that small things requested later can be prioritized over large things requested early (though it does mean that.) It also means that, for example, if you load N small Javascript files that each require a compute-intensive step to parse+load (GPU compute shaders, say), then you won't have to wait for the compute-heavy load process of the previous files to complete, before you begin downloading the next ones; but rather you can concurrently download, parse, and load all such script files at once. Insofar as they don't express interdependencies, this will be a highly-parallelizable process, much like serving independent HTTP requests is a highly-parallelizable process for a web server.
One benefit of HTTP/2's lack of head-of-line blocking, that would be more talked-about if we had never developed websockets, is that with HTTP/2, you get a benefit very much like websockets, just using regular HTTP primitives. You can request a Server-Sent Events (SSE) stream as one flow muxed into your HTTP/2 connection, and receive timely updates on it, no matter what else is being muxed into the connection at the same time. Together with the ability to make normal API requests as other flows over the same connection, this does everything most people want websockets for. So the use-case where websockets are the best solution shrinks dramatically (down to when you need a time-linearized, stateful, connection-oriented protocol over HTTP.)
> new protocol that sets in stone some heuristic
Note that there's actually no explicit specification of the order in which HTTP/{2,3} flows should be delivered. What I'm calling "content-oblivious round-robin chunk scheduling" is just the simplest-to-implement strategy that could possibly meet HTTP/2's non-head-of-line-blocking semantics (and so likely the strategy used by many web servers, save for the ones that have been highly-optimized at this layer.) But both clients and servers are free to schedule the chunks of HTTP flows onto the socket however they like. (They can even impose a flow concurrency cap, simulating browsers' HTTP/1.1 connection limit and starving flows of progress. It'd make the client/server a non-conformant HTTP/{2,3} server, but it'd still work, as what progress "should" be being made is unknowable to the peer.)
It's a bit like saying an OS or VM has a "soft real-time guarantee" for processes. Exactly how does the OS scheduler choose what process will run next on each core? Doesn't really matter. It only matters that processes don't break their "SLA" in terms of how long they go without being scheduled.
The advantages listed above all come from the fact that HTTP/2 runs all traffic over a single multiplexed TCP connection, rather than HTTP/1.1, where clients are forced to open many independent TCP connections (because each is a single channel where responses must be returned sequentially, in the order they were requested, and so block one another).
That happens totally automatically, there's nothing special you need to do to enable that.
There are also other potential advantages once you look at push etc, but using TCP as it was meant to be used really does give you many advantages for free.
it shoves everything down a single TCP connection. This means that if you loose a packet (and 4g is lossy) it stalls the _entire_ queue.
Instead of addressing the main complaint (that multiplexing everything down a single TCP connection is a fundamentally stupid idea) they come up with an entirely _new_ protocol.
It also fails to understand what HTTP2 has turned into: a file transport protocol, with some RPC built in.
So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
(Mind you, they are versions relative to each-other, because HTTP/3 is strictly better at doing what HTTP/2 does than HTTP/2 is, so there's no reason to use HTTP/2 if you have HTTP/3.)
But as I was saying: nobody forces mobile devices to use HTTP/2; and by-and-large, they don't. HTTP/1.1 still has its use-cases; neither HTTP/2 nor HTTP/3 was designed to obviate HTTP/1.1's entire set of use-cases.
You know how SSH has sub-protocols, e.g. SFTP? HTTP/2 and HTTP/3 are to HTTP/1.1, as SFTP is to SSH. It's a mode you can put the connection into, that will optimize it for a certain profile of use-cases. No more, no less.
(You know what else is a mode you can put HTTP/1.1 into? Websockets! Again, a complementary use-case.)
> So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
If you think you're so smart, write an RFC (or even just, build a reference server) that's competitive in production performance for its use-cases over similar links.
There comes a point when explicit OSI-style encapsulating layering becomes an active hindrance; and for the use-case of "a server serving a million concurrent requests"—the FAANG-scale problem—they passed that point long ago.
Yes, we mere mortal client users and backend developers might not find such use-cases relevant; but take the perspective of e.g. an internet backbone ops engineer. By volume, most of the traffic going around the Internet goes to these sites. Optimizing just the traffic going to these sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster, by reducing in size the huge blob of bandwidth-contention the requests to these few sites create.
Also, a design hint, if you actually want to try to build something competitive to HTTP/3: most of the problem being solved by HTTP/3 is that certain things are known to be optimal, but despite that, nobody can just force their corporate overlords to mandate a switch to those things right away. So HTTP/3 needs to be optimized for the case where you do the most fundamentally-performant things (e.g. using zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access); and also for the cases where you do less fundamentally-performant things (like generating and streaming a JSONL or CSV stream line-by-line as the output rows gets computed by an in-process dataflow engine.) One might call HTTP/3 an arbitrary custom protocol, that happens to have semantics such that it can be downgraded to HTTP/1.1 by a link-to-link proxy. And that is a hard constraint to optimize under.
They don't, though? HTTP/2 and HTTP/3 are voluntary to all parties concerned; whether you're a client, a server, or a gateway, if you don't speak those protocols but choose to just speak HTTP/1.1, then it's the peer that has to cope with that, not you.
(There isn't even any fancy forward-compatibility needed in TLS to support the semantics of the ALPN extension. If you just use an old TLS library that ignores the unknown extension data in the TLS stream, the other side will continue on assuming you didn't understand the question, and therefore aren't an HTTP/{2,3} server.)
HTTP/{2,3} are like a local language of a culture, spoken by immigrants when they run into other immigrants from the same culture. If either party is not an immigrant from that culture, it just doesn't come up.
> Why can't the "big sites" not then create their own network and take their enormous traffic there?
That's called circuit switching (i.e. the thing telco and cable services do that's not the Internet), and it's the thing the packet-switched Internet effectively obsoleted. From an Internet engineer's perspective, if you have two data streams, it's strictly better engineering to just feed them into a switch that linearizes those packets onto a high-bandwidth line "as they come" (and upgrade the bandwidth of the switch+line as needed, so that no signal is ever starved of line-time), than to try to time-divide or frequency-divide the pipe; let alone to keep those packets isolated on two separate networks of pipes. Then you'd need to maintain two networks of pipes! (And the people working at FAANG are still fundamentally Internet engineers who believe in Internet principles, rather than telecom principles.)
But besides that, how would that network be delivered into people's homes? Unless you're proposing that these services take the form of their own additional cable going into your house/SIM in your phone, this network has to merge into the regular Internet somewhere. And it's exactly at that point when that traffic once again contends with the rest of the traffic on the Internet. Even if it's only on the last mile, it's still getting in the way.
> it's just Google who's behind QUIC anyways, isn't it?
SPDY and QUIC are the names of "prototype standards" developed by Google. HTTP/2 and HTTP/3 are standards inspired by SPDY and QUIC, developed by HTTPWG, with Google as just one participant in that conversation.
The other backers of the standard are, of course, the groups whose interests are aligned behind having more-efficient HTTP: carriers, bigcorps, switch/NAT/WAF hardware manufacturers, cellular ISPs, etc.
But I see your deeper point—you're saying that this is all Google's solution to Google's problem, so shouldn't the onus be on Google to solve every downstream problem as well?
Well, it is and it isn't. Google is solving this problem for us right now, but it's not a Google-exclusive problem. TCP was created by DARPA, but maintaining a consistent stream over packet loss/reordering is not a DARPA-specific problem. They just happened to be the first group to need a solution for that problem.
The reason HTTP/2 and HTTP/3 are public standards, rather than things going on secretly only between Google Chrome and Google's backend servers, is that other parties see value in them—not just present value to themselves, but also future value.
New big uses of internet bandwidth arise every day. Netflix started sucking up half the Internet ten years ago, and it's already dropped down to less than 15% because other even larger use-cases have eclipsed it.
HTTP/2 and HTTP/3 are engineered to allow small businesses a path to grow into the next big bandwidth-sucking businesses (i.e. things so many people find useful that the whole Internet becomes about using them) without having to solve a thousand little scaling problems in the process.
Would you rather we live in a world where TCP/IP was a proprietary thing DARPA did; any company that needs TCP/IP semantics, has to re-invent it (probably poorly)? No? Then why would you rather live in a world where any company needing HTTP/{2,3} semantics in the future has to re-invent those (probably poorly)?
to which I as a normal person on a normal phone has no control.
> Also, a design hint, if you actually want to try to build something competitive to HTTP/3
I'd need to control a major browser's code base.
Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
> perspective of e.g. an internet backbone ops engineer.
They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
> If you think you're so smart, write an RFC
I already work for a FAANG during the day, I'm not going to put free hours in at night. And I'm certainly not going to get involved in cross company politics.
> a point when explicit OSI-style encapsulating layering
OSI has never been correct, its a fib told to students to make them think there is any kind of order past tcp/udp. I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms. yes its very neat, but
> zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access
from the protocol level, its not actually that much different, unless you are doing some sort of multiplexing down a single TCP pipe.... now, if you trying to balance throughput and latency on the same connection, thats can be really tricky. An answer is not to use the same connection. You do what FASP does which is use UDP for bulk and TCP for control/accounting.
> like generating and streaming a JSONL
you can easily use TCP at one packet a minute, or maxing out a 10 gig link. (I know, I've written a FASP replacement.)
> semantics such that it can be downgraded to HTTP/1.1
you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
You misinterpreted "downgrade" here (though maybe I chose a bad word for this.) I didn't mean that the proxy itself can force both sides to speak HTTP/1.1 to it. I meant that a proxy can be speaking HTTP/{2,3} to one side, while speaking HTTP/1.1 to the other, by translating ("downgrading") HTTP/{2,3} messages into their equivalent HTTP/1.1 messages. This is the reason that HTTP/{2,3} are HTTP in a real sense: the messages that flow over them still interoperate with the rest of the HTTP ecosystem, through bilingual gateways. The messages are a different bijective encoding of the same HTTP application-layer message-flow semantics.
(Nginx, by default, is such a gateway: it speaks HTTP/2 to clients if they offer, but speaks HTTP/1.1 to upstream backends even if they know how to speak HTTP/2. This is because HTTP/2 is useful to carry flows over the Internet, but has few advantages within a low-contention low-packet-loss LAN.)
This also has other implications, like the fact that browsers and caching proxies don't need to change the way they store cached HTTP resources, just because the resources come in over HTTP/{2,3}. They can just transcribe the messages to disk as if they had been the equivalent HTTP/1.1 messages.
> Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
The point is to take a stupid thing that people will keep on doing no matter what (for better or for worse), and help them to do it as fast (or as un-slow) as possible, without trying to make it non-stupid in the process (because that won't be the same thing any more, and so people won't use your smart thing, but will continue doing the stupid thing.)
You know Formula 1 racing? In F1, there are rules against doing most things that could make cars go fast. The point of the competition is to make the cars go fast anyway, without breaking those rules. If you make the car go as fast as it can, it's not an F1 car any more. To win, you've gotta make the fastest F1 car, not the fastest car.
That's HTTP/{2,3}: the cars are the flows, and the rules are HTTP's must-be-all-things-to-all-people semantics and manifold optimized-for usecases. The goal of adding HTTP extensions/versions/etc. is to make your server (or client) talk to clients (or servers) as efficiently as possible, without breaking any random peer from 1996 speaking to you through a bilingual gateway. If you did break those peers, you might have the fastest protocol, but you wouldn't have the fastest HTTP protocol.
> They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
That's an infinite regress, as the people who you're handing the problem off to are also Internet backbone engineers. If your company owns 30% of the Internet's traffic, your leaf-node AS is part of the backbone, and that AS's problems are the Internet backbone's problems. Just like the major universities were part of the backbone until ~1990.
Though, keep in mind, this isn't about fancy application-layer-aware routing or anything. HTTP/{2,3} have bandwidth savings, and that savings is passed directly onto the Internet backbone those messages travel along, in the form of more headroom for other stuff. It's not about needing to pay peering fees; it's about e.g. packing more clients per head-end fibre switch for an ISP. Or, to put that another way: it's about making more money out of the same aggregate delivered traffic volume, by having that volume represent more customers.
That's why the carriers are interested in participating in the development of these protocols: not because they need to do anything with them in their operational lifetime, but because the effects of the choices made by the HTTPWG have the potential to ripple out and create huge (e.g. 10+%) reductions in their aggregate traffic load (which they can then re-oversell); and they want to do what they can to argue for whatever designs best make this happen.
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)
There's so many things where my competence is based on understanding the simple preceding system plus a good intuition of how the class of systems tend to be bungled up over the years. We don't pay down the accumulated complexity, and newcomers don't get the same luxury I do.
It really becomes noticeable with highschoolers and I have to chose between teaching useless but illustrative groundwork, or just explaining and expecting a combination of taking it on faith and memorization.
What a strange comment. "The benefit of this protocol is that it's so complex that nobody understands it, so they have to use an existing implementation."
Bram's Law - The easier a piece of software is to write, the worse it's implemented in practice.
I'm not sure whether I agree with him but it's not obviously wrong.
[0] https://web.archive.org/web/20170629144859/http://www.advoga...
As discussed at length in another sub-thread here, the complexity solves real problems, it's not for its own sake. But if you're not convinced of that, nothing I say here is going to change that.
·
If you've worked with me, you'd know that I take simplicity as a virtue to an extent that's almost detrimental to getting stuff done. I despise complexity for its own sake. I wish as much as anyone that we could start over and redesign IP, TCP, TLS, HTTP, HTML, JS, and a few more things, with all our accumulated knowledge, into a simple and clean application and content distribution system that's easy to understand and observe. But path dependence :( We have to work with what we've got.
It's the picking out of HTTP 1.1 as some sort of golden age of clean and simple and observable that bothers me. It isn't.
(I'm not agreeing with the author, just pointing out why he might care about the ability for an individual to implement the protocols the web is built on.)
For the companies implementing http/2, those improvements DO matter. They have different needs.
Why is this bad?
It allows us to make smaller purpose REST api endpoints and do many calls, instead of trying to bundle everything together in larger endpoints.
Being able to parallelize cheaply greatly simplified our architecture and API design, and it costs us nothing.
First, while etcd did get a bit more complicated over the years, it stayed relatively stable compared to Consul. This was a point of pride of the etcd team and why it ultimately was picked for Kubernetes. Yes, a gRPC API is not as easy to use as HTTP, but this gained massive scale for Kubernetes when it was released. Something like a 10x increase in the number of nodes in a cluster, without changing any Kubernetes features. Just enabling etcd v3's gRPC API. That's pretty awesome.
As a member of the product team, I think we could have made different choices that would have massively impacted the ecosystem. Consul might not exist today. We did not want to add in a DNS server to etcd in order to keep it lean and focused. We had an integration with SkyDNS, but it wasn't on by default. This birthed Consul and its built in DNS server for service discovery. Consul is very popular so clearly this was the right move for some consumers. Kubernetes didn't need this, nor did Container Linux for its update coordination. For us, having this possible on top of etcd was the right trade off. Consul has added a complete service mesh into the later version. Again, etcd has stayed lean and focused. I think you are focusing on pebbles when you take in the overall ecosystem.
Would you rather etcd have turned into Consul's breadth? Would it have been forked?
The second point is around the success of Container Linux. I only include this because I am proud of it. We updated and secured a fleet of machines that measured larger than some cloud providers. That is a huge impact on the container and security ecosystems. Flatcar Linux and Fedora CoreOS remain as successors, along with other projects like Google'c container optimized OS and I think AWS announced one as well.
This work continues at Red Hat on OpenShift, etcd, Fedora/RHEL CoreOS, and tying Kubernetes more closely with the OS to keep the combined whole secure, up to date and successful for engineering teams shipping software.
This, I think, is the key to understanding this whole rant. It is entirely of a piece with the anti-systemd crowd, and I think understanding what was really going on there helps explain this rant, and a lot of other agita in the community these days.
Like, if you followed the drama on the Debian mailing list, it could sometimes feel very surprising that they were willing to go to war over the least sexy thing ever, an init system. Part of the reply is that systemd is more than that, it's a bunch of low-level plumbing all lashed together. But, who cares about low-level plumbing that much?
The answer (other than 'the exact sort of people who run Debian') is this. Linux started off as being written by hobbyists, for each other. It was very often a labor of love. Even the people making money off it weren't doing so in Big Business sort of ways. Linux and the associated stack around it is being taken over by a lot of big corporate interests: Intel, Red Hat, Google, even Microsoft. Are they bad? That's a matter of opinion. Are they inept technically? Probably not, but a lot of it revolves around how you measure things.
But what is absolutely happening is that people who are paid to maintain these projects on the behalf of large corporate interests are more plentiful, both in headcount and in personhours, than people who are maintaining these things for each other. For people who were used to Linux as this anti-corporate space, that's a huge and distressing change. And they can feel powerless against it, because it's really hard to reach a critical mass of people who feel the same way as you, and are willing to do all the things to not just support a fork but to keep the rest of the ecosystem they live in open to the fork.
It's not about any one piece of software, and it's not _really_ about pure technical merit. It's about the culture and the system of values and the way that decisions are made and it's about who's in control. The author's complaint isn't so much that the Google style is bad (although I'm sure he believes that very strongly), but that it's harder and harder to find somewhere to escape its reach.
I too am sick of people telling me to accept ubiquitous containerisation (snapd), locking down of filesystems (macOS), overly complex init systems (systemd (1)), lack of server access (manage it via ansible/k8s, not ssh). Usually the arguments given are "it's necessary 'for security'", "this is where everything is going", "you have to trust the vendor". This is not the spirit of Linux or open source. I totally see the effect you mention of paid maintainers forcing projects into their employer's direction. As such I found it sad when even Arch fell to the systemd camp. And it's one of the few things it really enforces, it supports 6 different network initialisation methods for example, where all other distributions just pick one and go with it.
(1) I'm not all anti-systemd by the way. I agree init was bad. But I think other systems like OpenRC are way easier to work with.
Arch has only supported one init system at all times. We could still have stuck with the sysvinit with hacks to look like openrc, but nobody liked that. The initscripts where a mess to maintain and systemd enabled us to not having to care about these details.
If we stuck with the old system, what would the argument be today? "Arch only supports sysvinit! That is not the UNIX philosophy!"? You don't see any complaints on distributions that only support one init which is either runit, s6 or openrc. But when it's systemd somehow people get sad?
The issue I have with systemd was mainly related to its heaviness not really fitting in with Arch's simplicity. And with the products of corporatism seeping through even one the most noncommercial variants of Linux. Basically what cwyers mentioned above.
I've been putting up with systemd as all the distributions I use have moved to it, but I never really liked it.. I don't like the way the scripts are so far hidden in the filesystem and I don't like journalctl, I prefer having simple logfiles.
I discovered Alpine linux recently as a server distro (previously I thought it was mainly developed for docker containers) and I really took a liking to OpenRC. It reminded me that things could still be simple even in a modern init system. I think it would have been a good choice for Arch, for me it would have been better than systemd. I don't even know the other systems you mention, I'm not an expert on init systems.. My opinion is just that of a user.
Arch is a pragmatic distribution first and foremost. If we can build systemd, and ship it as-is to have a complete init, and more features along with it, that is much more enticing then the alternative.
> I think it (openrc) would have been a good choice for Arch, for me it would have been better than systemd.
It would still entail maintaining some form of initscripts. I don't think most users realize how much of a burden this is, they don't really deal with that part of the system.
And there are things I do like about systemd, I also have to say. Especially the way you can pass a variable to a server with @, like run 2 instances on different ports.
I know very well that the rc scripts were a pain to maintain as I've written some as well in the past. So I really understand the benefit of upstream-provided service configs.
The transition to systemd for me as a user was just a bit more difficult. It deviates so much from earlier conventions, and adapting it I found difficult. It enforces a lot more than just the init system.
For some reason I always end up having to compile some of the software I use myself, and having to make scripts then. I found this a lot easier with OpenRC than with Systemd (having pretty much no knowledge of either).
But anyway I'm just one user and you support millions, I know you're making the right choices for the platform. Thanks for taking the time to reply to me in fact!
I'll be amazed if Arch has a million users. Our napkin math using steam user statistics put us on around 500k-600k users.
Runit will just run 3 scripts then supervise your service directory. _That's it_ and no other choices are made for you. You can compose runit with many other different types of systems according to your own tastes.
Systemd, ostensibly also just an init system, ties you to so many other non-init choices that I can understand the feeling of unease people have towards it.
We forgot what we were doing somewhere. Oh yes, solving problems.
Or we can replace perfectly functional systems simply because replacing legacy systems with modern solutions has some intrinsic value on its own, even if the modern system brings no additional value.
There are bunch of real problems that could be solved with software, but there's no money in it so why bother? It's better to roleplay technical expert and collect consultation rewards doing nothing.
Policy proposal: Accept contributions only from people who have not done paid work for a large organization in the last 24 months. Large organizations are entities with 50 or more paid engineers. Contributions include bug reports, feature requests, pull requests, and donations of money or equipment. Anyone may contribute documentation and training materials.
So you end up in a situation where package and distro maintainers face some work to support systemd, and because everybody else is supporting it it lessens the burden. There's this sort of gravity to these big projects that's hard to escape, and frankly a lot of people don't feel the need to; they view it as something they can get in a stable orbit around, not as something they're being sucked into.
> 1) Add hundreds of new failure modes to your software
In my entirely anecdotal experience, it removes error modes. It turns out that just because k8s offers a feature (it offers many!) doesn't mean you're required to use it.
> 2) Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
Portability, put another way, is a requirement to maintain n different deployment mechanisms because you have no meaningful control over the system on which your software is deployed. This is unavoidable depending on what it is you're deploying and where, but for those of us who have the ability to make those decisions the old notion of portability described by automake and others is actually a bad contract to agree to.
That said, the YAML but is horrible and should be abolished. This is a great criticism.
> 3) Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking
The author isn't considering that I am not, and never will be, Google's entire SRE team. I consider it a general win to be able to apply some subset of their encoded knowledge to the problem of keeping my services running with minimal downtime even if you pick up some k8s-specific baggage along the way.
YAML 1.1 (the most commonly implemented version) also lacks a great way of declaring sets (i.e. unordered collections of unique values).
I actually quite like most of the language a lot but the corner cases can be quite complicated.
For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file:
$schema: ./schema.yaml
Nothing at all. Until you start 'templateizing' it (looking at you, Helm).
It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
k8s API supports JSON almost exclusively, basically one could write in Whitespace then convert to it.
This scourge affects JSON too, such as Azure's ARM templates, which also crowbar logic into the format. An abomination!
I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
It's a bash script though.
Example json:
{
"deployments": [
{
"name": "site-en",
"env": {
"hello": "world"
}
}
],
"targets": [
{
"name": "dev",
"context": "aks-dev",
"env": {
"ipsum": "dolor"
}
}
]
}If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion.
Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
matchExpressions:
- key: app
operator: In
values:
- myservice
But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first.
YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly.
If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
Compare roughly equivalent Compose/Swarm and k8s configs, and you'll see that the k8s one is about 8x bigger.
it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure.
Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
If you're AWS, Azure, or Google, this is a feature, not a bug. It makes it easy to sell k8s-as-a-service.
While I could argue with all the points that he's making, my main counterpoint is this: junior devs don't care that their http/2 server uses way more "complex" code then their http/1 server, it's just a flag away (or in most cases, automatic). Senior devs worth their salt also don't care, if I design an application that is going to run on kubernetes I now know it will run in the big cloud providers and on premise without major changes. It forces you to accept that your app will die and needs to be able to run from a cold start without any issues. I can't count the number of machines I've encountered over the years that couldn't be rebooted because the maintainers monkeypatched the crap out of it while running EoL distributions, libraries and web servers.
And now I've realized I've become a geek yelling at the cloud as well.
HTTP/2 is more complex to implement from scratch (but still quite doable, even as an individual), but it has built-in support in every language & framework worth its salt now, so you don't need to do that.
If you're using an existing implementation, that's usually just as easy to do as HTTP/1.1, because they have almost exactly the same semantics (that's an explicit goal from the spec), and so most implementations have almost exactly the same API.
In practice, it's a syntax & connection management change on the wire that's mostly invisible as a developer building on top of it, plus a set of optional extra features (like Server Push) that you can use if you want or ignore if you don't.
Can't speak for QUIC/HTTP3, since I haven't touched them yet, but I'd be a surprised if that's a hugely different story.
I also disagree with much of the rest of that, but the clouds are getting sick of me today.
if you wanted simple, there was redis or memcached.
it's an arbitrary cutoff to call etcd simple (it implements Raft, for fuck's sake, it implies a distributed system, multiple odd number of nodes to avoid split-brain!)
systemd is simple, fixed config format, comes with every distro, default works, has extensive documentation, compared to undocumented distro-specific init scripts written in sh (or bash, or worse).
sure, if someone just wants to hack on their 8bit toaster, then they might find a simple shell script simpler.
but the post is not lamenting that.
Cool, I need to look into Kubernetes way more then! There hasn't been any single development in the Linux world that has made my job as a sysadmin easier than systemd. Services, Timers and Networking that just works, is concisely defined and can be used across all relevant distros? Hell yeah, I don't ever want to go back into init.d/ifupdown/crontab world filled with bad configuration files, strange footguns and bugs that won't ever get fixed because someone's legacy system might depend on them.
There's a million reasons to hate Kubernetes, the author couldn't be bothered to venture beyond the lowest hanging fruit (yaml)?
If I could downvote this I would.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
I didn't like having to move cl machines to fcos but it really wasn't that bad and I still get coordinated, active/passive upgrades. ¯\_(ツ)_/¯
> With the massive influx of Kubernetes users came, of course, a large number of Xooglers who decided to infect etcd with Google technologies, as is their way23. Etcd's simple HTTP API was replaced by a "gRPC"4 version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys. etcd 3.2 added back a tiny subset of the HTTP API through the "gRPC Gateway", but not enough to implement any of the rich applications built on top of the original API. The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely.
This is the interesting part of the post to me. I can understand the disappointment - but also, nobody's making the author upgrade to the gRPC-based etcd v3. This reminds me of other "traditional" sys-admin folks I've worked with who seem to want to stick to 2010-style administration rather than work with newly available abstraction layers that eliminate some of the overhead (e.g. Docker, k8s). If they want to continue linux administration in that way, that's their choice, and there's some merit in that - but they shouldn't expect their tools and other developers to necessarily follow them.
They're stating that the v2 API keeps being slated for termination. Surely once that happens they won't be able to use it anymore, and will have to either use an unmaintained etcd, fork it entirely, or be forced to use the v3 API?
If the HTTP API has value, why not fork etcd and maintain a version where the HTTP API is the primary interface and gRPC is an afterthought or missing?
If the HTTP API has value, the cost to maintain it should self-justify, right?
Everything anyone ever did had some kind of value to someone, even murdering babies.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
It's a lamentation of the introduction of complexity into a formerly-simple set of APIs.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
Anyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Google is still the plurality contributor, followed by Red Hat. If you count Red Hat and IBM jointly, they overtake Google[0].
Disclosure: I work for VMware, which competes with Red Hat/IBM.
[0] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var...
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
I'll take that quote into my fortune file.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
FWIW I really like systemd, but I agree the scope creep is real and isn't all positive.
Also: https://suckless.org/sucks/systemd/
Personally I avoided it for a while because of the hate, but haven't really had issues with it since using it.
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
Even https://nosystemd.org/ mostly just lists bugs
After moving all my services from varius init systems to systemd, I'd never go back
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
i trust neckbeards over most others.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
$ sudo systemctl restart apache2.service
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
$
$ sudo service apache2 reload
* Reloading web server apache2
* The apache2 configtest failed. Not doing anything.
Output of config test was:
AH00526: Syntax error on line 1 of /etc/apache2/sites-enabled/sd.conf:
Invalid command 'blah', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
If you have a piece of software which is basically finished, and a group of people come along who are interested in extending it far beyond its original purpose, then it's very easy for those people to end up as the official maintainers of the software simply because they're going to be the most active.
So when it comes to deciding whether expanding the software so much is a good idea (where the alternative is to make a fork to add all the new stuff in), the people who are happy with it as it stands don't get as much voice as perhaps they should, because for obvious reasons they aren't contributing many patches.
I was a major contributor for etcd 2.3-3.2 at CoreOS. The people who extended etcd to support gRPC included the original etcd author (hi Xiang). gRPC support was necessary for good performance; we had benchmarks to justify the decision. Likewise, v3 brought about a key-value model change that was incompatible with v2 to better support binary data, ranging over the keyspace, transactions etc. A v2-style gateway for v3 with a pretty JSON API was planned but never completed due to lack of resources; the ugly gRPC json gateway turned out to be good enough for most people. Similarly I wrote a proxy to run v2 requests over v3 instances, which does support the v2 JSON API. This isn't as if a new group of people showed up and ruined the software without caring about existing users. None of us were "Xooglers".
It seems what you're proposing is what the author both argues against and wrongly believes is what happened. We constantly pushed back against k8s influence. If we didn't, etcd would be a k8s sub-project right now. I'd also like to point out that removing the people who do the difficult work of actually writing the software from the decision process is incredibly insulting and devalues their labor. How dare you.
- The problem the author describes can happen (I have seen it happen)
- I do not know whether it happened in the case of etcd
I am not saying:
- I have a proposed solution to this problem
- The people who do the difficult work of actually writing the software should be removed from the decision process
Apologies if that wasn't sufficiently clear.
The problem is not with the Kubernetes but with management/architecture teams of companies that decide to use k8s for projects that have no real scaling requirements or when overhead of just having a constant fleet of nodes would be less than overhead of maintaining working k8s deployment.
Kubernetes is a hugely complex piece of software that is handling hugely complex cases that arise when you try to deploy and maintain a large number of applications with hugely differing scaling needs.
In my experience, k8s deployment absolutely requires a dedicated team with top notch k8s knowledge and debugging skills. You also need to understand k8s is a huge constant overhead as it requires actual knowledge and experience to use and has complex faults that require ability to debug really complex and interdisciplinary problems.
Google could do that because they invest in their teams and because they have exactly the type of problems that greatly benefit from a single complex solution that can handle all of them (meaning your engineers can migrate between projects but they still can use same tools they already know).
The trouble with management/architecture teams is they do not understand they are not Google. They are Google-wannabees. They like to promote how great they are and how great their projects are but are either deluding themselves or are being deluded by their lower echelons and have not set up themselves to understand what is actually going on.
And that's how you land at a situation where a team that has simple application that would require just two nodes (second for redundancy only) and no knowledge of Kubernetes is required to deploy to k8s instance and deal with a host of problems they never had to deal before that are beyond their capabilities.
But if you have only simple applications you will be surprised how complex it is sometimes to debug k8s problems compared to what you would expect from a simple application.
If some of your applications already have very complex deployment needs then you are replacing one complex homemade process for another complex GP solution.
Oh for crying out loud. This Kubernetes bashing is getting so old.
Frankly, I think it's the best thing that has happened in the past few years. Boohoo it requires some YAML. Yeah. And then you get lots of value out of the box.
Systemd bashing is a bit more deserved, but this horse has been dead for a while now.
And there's also some CoreOS bashing too. CoreOS is/was spetacular.
etcdctl ls /registry
etcdctl get /registry --prefix=true
We're not finished. Apparently the pod was insufficiently secure, because now we need to decorate the etcdctl with a couple certificates, a key, and do SSL encryption to make contact, so in the end I have this wrapper (names changed to protect the guilty):
$ cat ~/bin/etcdctl
#!/bin/bash
declare -A EtcdHost=(
[clust_prodA]=clust1-leader
[clust_prodB]=clust2-leader
[clust_qualA]=clust3-leader
[clust_testA]=clust4-leader
)
if [ ! -z ${EtcdHost[$cluster]:-} ]; then
export KUBECONFIG=$HOME/.kube/$cluster
exec kubectl exec etcd-${EtcdHost[$cluster]} -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --endpoints https://${EtcdHost[$cluster]}:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt $*"
exit 0
else
echo "No known etcd host for cluster $cluster";
exit 1;
fi
Ask whoever deployed this cluster. This is not something that Kubernetes mandates.
> and do SSL encryption to make contact
And that's a bad thing?
Not even then. If you're an enterprise using EKS, you still have to work around the pain points, missing features, incompatibilities, and do a metric shit-ton of custom integration. And that's before even touching CI/CD, and ignoring how you build the infrastructure.
It's like a Formula 1 engine. Really complicated, you still have to assemble the rest of the vehicle, and you need a team of engineers to understand what it's doing when you drive it.
I think the author is right. gRPC is bullshit that you only use if you're dedicating your whole organization to one protocol. They actually re-added that wacky backwards-REST API-gateway because people really needed a REST API. On the other hand, Consul not only has a REST API since the beginning, but it incorporates more useful (some would say necessary) features so you don't have to write them from scratch. So you can go with Consul and get more compatibility and features, or go with Etcd for the opposite.
But at the same time, a "lightweight database written around Raft consensus" is like saying "clogs for runners". Most people shouldn't use it, and those that do won't be happy with the consequences later.
> cancerous user-hostile protocols of HTTP/2
I know not to take the article seriously. I'm simply not interested in hearing from someone with that kind of attitude.
As a counter example of Xooglers making things more complicated, I wanted to bring forward the removal of the protobuf exposure format for prometheus 2.0 after it was discovered that it's actually more performant to parse prometheus' text format.
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers)
1.0 was released in 2015, but seems to me the relevant year was 2014:
https://www.wired.com/2014/06/google-kubernetes/
And how was it Xooglers? Wikipedia certainly seems to be saying this is a Google-initiated product, through and through: https://en.wikipedia.org/wiki/Kubernetes
>There were a lot of bad ones
Sounds like he had a point, then. Also, apparently HN would rather read your rather salty gossip, than comment on the article on its own merits. Nice!
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
No, but thanks for explaining my career to me and helping me empathize with toxic people.
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
You're welcome. :)
[0]: https://books.google.se/books?id=0VRnDwAAQBAJ&pg=PA490&lpg=P...
And, yeah, sysadmins used to say no a lot but younger ones tend not to because they haven't been rubbed wrong by the silo's of yore.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
https://standalone-sysadmin.com/the-difference-between-site-...
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic. I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration and they like it. I guess that's OK. What I like about k8s is that it's a common language for this sort of thing -- I've used it for 3 projects, and they've all looked about the same. I've never had to learn anything company-specific; I can just run my software. I don't think that's a bad thing.
(The solid alternative to k8s, it seems, is just paying someone else to run everything but the one application that your company makes. Can't figure out how to run Prometheus? Just buy a Datadog subscription. Can't figure out how to run a frontend proxy? Just buy an Application Load Balancer. Can't figure out how to search and retain application logs? Just buy a Splunk subscription. Can't figure out how to install MySQL? Just buy it from your cloud provider. That is all great, but you end up spending a lot of money because you can't efficiently manage software and instead spend all your time writing rants about orchestration frameworks. Sometimes I wonder.)
Whenever I see articles like this, I have to ask what the transition from "big UNIX" to Linux would have looked like on HN. I am sure some people hated it, and I'm sure some of those people had good reasons. But things got smoothed out and it turned out that Linux and its ecosystem was pretty good. If you maintain a production application today, it probably runs on Linux, and that's good because that's one less thing you have to teach your team. I kind of see k8s in the same place. Lots of people have trouble running multiple pieces of software in production. k8s is a common language for doing those things. You can build it yourself or you can buy a managed service. You can extend it to do the crazy things you need it to do. It's not a bad thing, and I think it's pretty disingenuous to compare it to systemd.
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
If you hire a lot of people, they all will need to occupy a space each. To deserve that space, everybody needs to be either quite good... or good at faking being good.
Truly innovative things are done by great individuals. And corporates are not the natural habitat for great individuals.
Is your fork not getting much traction and community support? Maybe you’re the edge case.
Let's say there are 10 people willing to fork it to keep it simple as it was previously. Where is the go-to place for that to happen in the open source world?
In your 10 people example, it seems that there are a few problems to me: - finding like minded peers - getting critical mass (are 10 enough to maintain the fork?) - getting them all to agree on what subset of features to keep
If I were in the market for a fork, I would search Github first before forking my own, so I think the first item is basically solved. The second varies on a case by case basis, of course. The third seems more critical to me, probably dispersing the effort more than anything else.
I think the unnoticed problem is that the tailors (old school system administrators and dev ops) are being asked to become the managers of Old Navy (kubernetes cluster administrators). Their entire skill set is opposed to this new role. I can't really blame them for being frustrated.
To keep stretching my already thin analogy, there are still tailors in this world. However, most people buy their clothes at outlets like Old Navy. Whining about the quality of the clothes and the crummy manufacturing process at Old Navy won't change that.
But down in reality, most companies are probably running their own made-to-measure deployment and operation schemes with lower quality and consistency than Old Navy.
Again, it reminds me of a carpenter friend who hates Ikea. Yeah, I get it, Ikea is really bad compared to custom built furniture. No one who buys it expects anything else.
This story is older than the industrial revolution. Craftsman being replaced by technology. We even instinctually know this is going to happen to knowledge workers but we seem blind to it when it happens to us.
> I would go so far as to say that Kubernetes is the worst thing to happen to system administration since systemd.
I know it's popular to shit on systemd, but as a casual user, I honestly don't get it. I've written about a dozen systemd unit files over the years, and each time it's been a pleasure. Write a few lines describing what to launch when, one command to make systemd aware of your new file, one command to launch it.
In contrast, I've written a couple sysvinit files, and the experience was horrible every time. Having to manage pid files and logging myself, setting up the proper symlinks, etc. It'd always end up being a ~100 line script that'd be 99% copy-paste.
There might be some use-case where systemd is worse, but for my (admittedly simple) use-case, it's far superior.
My impression from the systemd debate is that those that are against it feel that way because of the breadth of systemd - it's much more that "just" about service units.
I don't find those new, so called "complex", software difficult. I can pretty easily see why that we need those new features and what problem are they trying to solve. I'm really grateful there are ppl out there solving my problem one-by-one for free. I feel those new softwares enable me to do much more and better than I could do 7~8 years ago.
I feel sorry for the fellows who just can't grasp new technology quickly but this world is cruel.
It's only hard if you can't get it.
We all do complex software and you all know its complexity. Sharding, failover, retry, back off, schema etc...everything in a high scale systems.
The problem is user who used it but doesn't need other features feels frustrated. At the same time, user who used it at scale need feature others don't like? What can we do? I don't know.
A good example is Sentry: https://blog.sentry.io/2019/05/14/sentry-9-1-and-upcoming-ch... Say you are a small single person SaaS? are you going to install ZooKeeper, Kafka, Snuba...No. But they need it...
I can understand that people who are used to HTTP based APIs will feel more comfortable being able to tap into a rich fauna of tooling for HTTP.
However personally I am not really interested in raw network APIs as long as they work. I'm interested in programming - not poking around low level stuff.
I'm interested in libraries for speaking to services. When I use services like Twilio and Stripe, I'm not interested in rolling my own. I'm not going to implement my own unless I really have to. It isn't nice to just publish an API and not have libraries for talking to them.
gRPC, at least for me, makes both initial bootstrap, development and maintainance easier.
So true.
With gRPC came as well came as well incompatibility of many clients with less popular architectures and as the article describes, being that a super specialization (because that's the only thing that it is), we are losing the qualities that made OSS legendary.
Having recently learned bits & pieces of Kubernetes, I can see the author's perspective as it's one I held for some time. Kubernetes is useful and makes many things easier. In exchange you give up finer-grained control and you have to work with an abstraction. Such is the nature of all abstractions.
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol [HTTP/2] pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.
Open source is a market. If you stop building it, they might come too.
If you hate the direction the open source project has gone in, why not fork it and have the v2/HTTP version live on forever? The project does not owe it to you to stay crystallized at a moment in time.
But it hasn't been removed yet. It might be cynical, but I think a v2 fork now won't accomplish anything because the people most interested in one would simply continue to use v2 upstream.
https://blog.fastcomments.com/(7-07-2020)-fastcomments-on-si...
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
Ahahahahahah, yes, anyone can implement a HTTP server, just badly. HTTP/1.1 is quite complex, the spec alone spans over eight RFCs: if you can implement all of that I doubt the HTTP/2 serialization is much of a concern. :P
This is probably even true for 99.9% of HN readers.
Having said that, the guy represents the old guard of tech, the kind of people who ask “why do you want to do this” instead of trying to help you, and usually follow this up with a useless suggestion for an alternative that sort of does what you want but not really. Thanks, I guess. They have a very concrete idea of how things should be done, new ways of doing old things are frowned upon, and anyone challenging their “authority” is seen as an imbecile.
I had the dubious pleasure of working with such a man a few years back and while I couldn’t but admire his technical expertise and competence, I grew to hate him to a degree that would rival that which the author of the blog post hates Google. Get off our lawn, old geeks! You’re not the only ones with opinions around here.
The author is a total jerk trashing everything in his path with hyperbole rather than sound logical arguments. Read his rant on protobuf. Why would anyone waste their time tearing down the contributions of others?
Unless you need to operate at a certain scale (that most people never will) then Kubernetes is like taking an F1 car onto a regular road. Recipe for pain.
Everything post says is bad is bad, everthing the post says is good is also bad.
2. Speaking of forking, it's still quite possible for a few interested developers to fork an clean older version, call it freetcd, and rebuild the reputation. It worked for LibreOffice after all - and that's when OpenOffice wasn't terrible, just somewhat mismanaged.
All that said - I don't quite see why you need a specialty database for a simple key-value store for /etc settings. Aren't there plain-vanilla kv-stores which would do?
I think that developer tools have always taken up a significant proportion of the resources available on a developer machine. Back in the 90's, Visual Basic was criticizes as being bloated for requiring a 4MB of RAM (the exact number may be off).
Now we have vastly more powerful computers. I think using those resources to have easier, more extensible, and more capable developer environments is a good tradeoff.
The "solution" to bugs today is forced updates. Vendors love being able to forcibly impose whatever they want to do on the user. Turn off features, put in more ads, whatever. If software was reliable enough, nobody would upgrade, which damages the business model. It's so convenient that software seems to need constant "security updates".
etcd needed a gatekeeper like Linus Torvalds [1]
¹See https://xkcd.com/1208/.
My take on this is that CoreOS and google have different use cases, and the beset solution would have been to fork the project. Since google is big [citation not needed], most development effort would probably end up there, but people who wanted the simple version could stick with it.
The difference between a HTTP API and gRPC is one of scale. If the HTTP one is a few bytes more per message, and takes a few more clock cycles to decode, then at google-scale that might mean extra energy use on the order of a small town. RPC makes complete sense for google here.
Then there's reliability. If something fails one in a million times, it's a minor inconvenience to someone who uses it once a day but if you're running it four billion times a day then you start to notice. gRPC is strongly typed so it actually removes the malformed HTTP failure mode - and you can bet that google's SREs are taking care of the other ones too. When the OP says "Quality is, alas, a dying art", I'm not sure whether that's an honest description of just how many "nines" the google SREs are building into their projects. It's not like they invented gRPC to make their systems less stable!
I can recommend rachelbythebay's comments on this, she's a SRE who has done the rounds in silicon valley - appropriate posts here are "Some Items from my Reliability List" [1] where she famously says "there is far, far too much JSON at $company" and ends that section with "Why aren't you using some kind of actual RPC mechanism? Do you like pain?" - at the scale she's operating, RPC is less pain it would seem.
In "We have to talk about this Python, Gunicorn, Gevent thing" [2] she criticises the use of a popular python framework at the scale she's operating at (I'm sure this was shared on HN in the past).
This doesn't mean that python, "RPC over HTTP", JSON or etcd are in any way bad. In fat their simplicity makes them great things for beginners to learn and medium to large companies to use - they're not just prototyping tools, people can and do build a lot of real stuff with them. It's just that at some point before you get to the ridiculous scale of google's infrastructure, there's a tipping point where going full RPC starts saving you a lot more than it costs; that's not a bad thing and it has nothing to do with trying to "capture the market" or anything like that. A corollary of this: precisely because google works at such a scale, there will be a lot more developers working on grpc-etcd than the "consumer" one.
Summary: developer builds great tool for A; big company has use case B.
[1] http://rachelbythebay.com/w/2019/07/21/reliability/ [2] http://rachelbythebay.com/w/2020/03/07/costly/
7. My favorite HTTP/2 interaction has been finding and reporting this bug in haproxy. The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 S: Aa is just a list of the 61 most popular headers from Google properties. ↩
8. HTTP/3 is all the badness of HTTP/2, but run over a worse layer 4 protocol named QUIC that totally fucks up networking for everybody in order to get a tiny bit more optimization for Google. That's all it does. It makes the Internet strictly worse for everybody but slightly better for the hugest of huge web properties. Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to work. ↩
"Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to [continue to] work."
These protocols do not make it more difficult for advertisers or advertiser-funded companies like Google to serve ads and optimise serving ads, they make it easier. For example, they allow for faster and more efficient serving of third party resources, e.g., ads, and siphoning of user data without any affirmative interaction from the user, and cramming more data into more headers that are more difficult to monitor. As an end-user I aim to send the absolute minimum headers needed in the particular instance. 1-2 usually works. From an end-user perspective we should be trying to decrease headers not increase them. Websites should only collect the information they actually need, nothing more. That is not the sort of web envisioned by the "new" HTTP protocol advocates.
I actually use HTTP/1.1 pipelining with a user-agent I wrote myself and this original HTTP pipelining has worked great for me for over 20 years. I am not an advertiser or an ad-supported company; I am just an end-user retrieving text in bulk from the web. Thus in addition to state-aware firewalls and load-balancers, I want HTTP/1.1 pipelining to continue to work. The proponents of these protocols want their audience to believe HTTP/1.1 pipelining doesn't work or has serious problems. But that is only for their use case. In fact it does work for end users doing efficient bulk data retrieval without retrieving any ads or other cruft. But these Google-sponsored protocols are not written for end-users wanting to avoid ads. Those are not Google's customers.
Better get out of here before they notice us.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
Same with all these technologies.
Kubernetes is a grassroots phenomenon of developers experimenting with self-immolation while padding the resume.
Major difference. Your example comes from the top. Kubernetes comes from the bottom, and is more insidious in my opinion.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.