Hacker News Reader

> so there are more niches where hiring someone to babysit a few hundred VMs is profitable

This makes a lot of sense. The same thing happened in the past with new technology, such as the electronic spreadsheet:

"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added."

Episode 606: Spreadsheets!, May 17, 2017, Planet Money

Basically, "why technological innovation creates and transforms jobs, instead of causing unemployment".

...except that 400k in 1980 translates to 608k in 2017, if you factor in overall labor force size. That means even though there were 600k jobs "created", it's still a net loss.

[1] https://fred.stlouisfed.org/series/CLF16OV

This is a weak argument, since it ignores the productivity gains in all the industries that necessitated more accountants. Accountants don't only do each other's taxes, so more productivity in accountants is an indicator for a more productive economy

I think the point of the comment in discussion is:

"Though the spreadsheet was supposed to make clerks obsolete, it in fact just upgraded their job requirements to allow the operation of spreadsheets"

And probably their salaries too.

This is more in line with the discussion related to OP, as well, as devops employees exist, even though apparently it is automated.

In 1980 - there were 90M employees in the US. Now there's 151M.

Given that the US has transitioned out of manufacturing and into businesses services - I don't think much of this is explained by technology creating new jobs.

I think it's just explained by the workforce growing - and the shift in the US's role in the global economy.

workforce grows as more is goods and services are produced.

So the real question is - whether the ratio of different roles changed vs total population.

ATMs caused a decrease in teller transactions, but led to an explosion of bank branches.

SREs don't normally write Kubernetes alternatives. They are the people who operate/write automation that interacts with/advise teams how to run their software on/ Kubernetes to solve business problems like ensuring availability.

> but instead that operational load of running in-house software should be borne primarily by the developers of that software

Go back and read a few DevOps books and blogs by the founders of it. We will always need separate disciplines for dev and ops, just like we need mechanical engineers and mechanics/racecar drivers. But we need them to work together and communicate better to solve problems better and not throw dead cats over walls.

You can of course give devs more powerful tools, more access and agency to enable them to develop the software better. Sentry.io is a great example of what is needed; makes everyone's life easier, devs can diagnose issues and fix bugs quickly without anyone in their way. That doesn't require operations work because it's just simplifying and speeding up the triage and debug and fix and test phases. It's the fundamental point of DevOps.

My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it.

You want the person who designs the car to know what a car is, and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving.

The most popular alternative to "DevOps" is a team of developers who do not run the software, and may not even have the skills or capabilities needed to boot up the server process. They do their development in an IDE, run unit tests to verify functionality, and do not have permission to log in to the production environment.

Meanwhile the "ops" side consists of people who may know basic shell scripting, or some Python if they're a go-getter, but are unable to read stack traces or diagnose performance excursions.

throwaway787544 deleted their reply to this post. My response was as follows:

---

  > So you're familiar with Six Sigma then? Value stream mapping? TPS?
  > W.E. Deming? Martin Fowler? There's more to DevOps than deployments
  > and CI/CD.

None of those have any relationship to DevOps.

  > I haven't worked at Google, but I expect somebody gave you some tools
  > and some access to cloud infra and said "good luck".

I was on Borg-SRE. My job was to build the tools and maintain the cloud infra.

  > If you're lucky there was a small army of ops people behind the
  > scenes keeping things running for you

The point of having an SRE team is to avoid having a "small army" of ops people. Manual operation of compute infrastructure is uneconomical at scale.

  > if you weren't lucky you were expected to know how to correctly build
  > and run complex systems at scale by yourself

Google does expect its developers to understand basic principles of distributed systems, yes.

  > > every distributed system is self-driving
  >
  > I didn't know Google was in the bridge selling business.

What I mean is that when your service runs 24/7 and downtime is reported in the New York Times, there is no room for manual action during normal operation. The system must have the capacity to maintain itself and self-recover.

Responding to the inlined kubectl command mentioned above:

As an outsider, that command looks really easy to mess up.

  * shell interactions with quotes
  * double quotes
  * interpolating into image namespace with no autocomplete
  * easy to forget an argument
  * do you get autocomplete against the deployment name?

Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right.

  * var x = new List<Dictionary<CustomType,int?>>()

This will light up warnings if you get anything wrong;

You get:

   * go to definition of `CustomType`
   * autocomplete on classnames
   * highlighting on type mismatches
   * warnings about unused vars.
   * if you initialize the var, the IDE will try hard to not let you do it wrong

So structurally,

1) in the code realm, for doing minor work we have really strong guarantees,

2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.

I think this is behind the feeling that the k8s system is not ready yet.

Normally you're never going to run a kubectl command in prod that changes anything except for kubectl apply -f. The yaml file may or may not be script or template generated but it's likely. In a lot of shops they're just going to write to a wrapper around the kubernetes API in whatever language the shop uses whether it be python, go or whatever. And there are plenty of linters for the yaml and ways to test without impacting prod.

> So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD.

>> None of those have any relationship to DevOps.

Deming. I think you mean maybe that Deming has less of a relationship to SRE (Google style). The DevOps Cafe guy wears a Deming t-shirt and talks about Deming all the time. Maybe he has denounced Deming for some reason?

Value stream mapping. This has been brought up in at least one talk at DevOps Days.

Six Sigma. There's some obvious overlaps if you look it up.

Thank you. I didn't want to get into a whole "thing" because the parent just doesn't know what they're talking about and I can't teach them everything in a comment thread, but suffice to say that Gene Kim and Jez Humble would agree with me, and that anyone who wants to know more can look up books and blogs by them.

> My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it.

Cool, you probably know my old colleague Hugo, he was one of the first SRE's at Google in a team of like 20 or so. (I wasn't there, could be mistaken).

Anyway, what "Production" was, is different than what devops is. DevOps is different things to different people. In the beginning it was "What happens when systems administrators do Agile".

(Genuinely), Confusion came about because of the "100 deploys a day" talk and the conference being called "dev ops days" (to include developers).

The two things got conflated and now everyone thinks devops are.. well, build engineers or sysadmins or developers who learned a bit of bash and terraform.

> build engineers or sysadmins or developers who learned a bit of bash and terraform.

From what I have seen, it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime.

> it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime.

That sounds like sysadmins.

let's say there's 5+ leading definitions of DevOps. I'll add this which I think is quite unpopular but true:

DevOps is how Engineering respects the work of sysadmins.

> And there's no analogy to an Indy car driver in software, every distributed system is self-driving.

You typically build systems to sell a product. The driver is the customer that uses the app. You build the app to be as simple as possible to effectively operate while providing the best possible outcome for that customer. The driver doesn't care or need to know how the engine or the rest of the car works, just that the engine and car works well.

Did Google really use the “DevOps model”? Observing from the outside it looked like SRE was their alternative

The major point is that "mechanical sympathy" for how to operate software should be considered early in the development cycle. Fulfilling the business goals in operation should be a major design consideration that might warrant trade offs in other areas. Traditionally, this has been seen as more of an afterthought and DevOps solution to the problem is that involving the people who design and maintain software in its operation would automatically create incentives to make operational improvements because the people making decisions are the people feeling the consequences of those decisions.

Generalization:

- each role in a company tries to optimize/nudge whole organization toward this role's convenience.

- specialization improves local optimum (advances certain role) at the cost of global optimum (everybody has to dance around new roles processes)

- joining seceral roles into one, creates the oposite result, optimum is searched at more global level (not necessarily found)

- Separation of responsibilities (aka creation of new role) can generate a fractal (e.g. tester of left winglets blue stipe's thickness meter)

- complete joining off roles will create homogenous chaos after n employees (everybody should do everything)

prediction: we will see constant experimentation, first roles will be split, then some will get joined. then split again. then joined again. (people cant search for local optimum and global optimum at the same time)

This is a very insightful comment.

I agree in general. Extreme example I've witnessed ~8 years ago: dev team is building linux server. Dev team is not allowed to touch production. Server is deployed via scp by ops team. Ops team is on the other side of the planet and generally pretty unknowledgeable. Dev team is deploying by creating a ticket and literally telling ops team what to type in terminal. Ops team fails at copy/pasting instructions. Deployments fail, dev team has to fix things by telling ops team what to type over the phone.

Lesson learned: kubernetes isn't that bad

If you look at construction, they have a system that's very similar to what DevOps espouses. Lots of trades and specializations, because building to code is complicated and important, and having the experience of doing one job well is important to be able to work around problems on the fly. But - this I think is the critical part - they must know about the other trades that are affected by their work, so as not to impede others' work or the project by accident. It's as much about compassion for the guy coming after you as it is reducing cost and speeding up construction. All of that is the aim of DevOps.

> DevOps is the idea that there shouldn't be separate "dev" and "ops" organizations,

I agree with this definition of DevOps. However the vast, vast, vaaaast majority of real life uses of the term "DevOps" I've seen are just rebranded sysadmins. Sometimes it at least implies a more engineering approach to their coding. But in these institutions the Devs and Ops are very much separate groups of people, unfortunately.

Agreed. The DevOps that I'm familiar with is two job descriptions in one job, one headcount, one pay.

I agree. I think the grandparent is maybe describing an ideal state, but in practice most "DevOps" and "SRE" positions I see go by on job boards are for either sysadmins who can do some automation and infrastructure as code, or possibly devs who know how cloud infrastructure and networking work, depending on your perspective.

If you're lucky the hiring manager might have at least read the Phoenix Project or the Google SRE books.

essentially Jevons paradox

https://en.m.wikipedia.org/wiki/Jevons_paradox