Comment by jmillikin | Hacker News Reader

Comment by jmillikin | Hacker News Reader

Parent Full thread

throwaway787544 deleted their reply to this post. My response was as follows:

---

  > So you're familiar with Six Sigma then? Value stream mapping? TPS?
  > W.E. Deming? Martin Fowler? There's more to DevOps than deployments
  > and CI/CD.

None of those have any relationship to DevOps.

  > I haven't worked at Google, but I expect somebody gave you some tools
  > and some access to cloud infra and said "good luck".

I was on Borg-SRE. My job was to build the tools and maintain the cloud infra.

  > If you're lucky there was a small army of ops people behind the
  > scenes keeping things running for you

The point of having an SRE team is to avoid having a "small army" of ops people. Manual operation of compute infrastructure is uneconomical at scale.

  > if you weren't lucky you were expected to know how to correctly build
  > and run complex systems at scale by yourself

Google does expect its developers to understand basic principles of distributed systems, yes.

  > > every distributed system is self-driving
  >
  > I didn't know Google was in the bridge selling business.

What I mean is that when your service runs 24/7 and downtime is reported in the New York Times, there is no room for manual action during normal operation. The system must have the capacity to maintain itself and self-recover.

Responding to the inlined kubectl command mentioned above:

As an outsider, that command looks really easy to mess up.

  * shell interactions with quotes
  * double quotes
  * interpolating into image namespace with no autocomplete
  * easy to forget an argument
  * do you get autocomplete against the deployment name?

Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right.

  * var x = new List<Dictionary<CustomType,int?>>()

This will light up warnings if you get anything wrong;

You get:

   * go to definition of `CustomType`
   * autocomplete on classnames
   * highlighting on type mismatches
   * warnings about unused vars.
   * if you initialize the var, the IDE will try hard to not let you do it wrong

So structurally,

1) in the code realm, for doing minor work we have really strong guarantees,

2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.

I think this is behind the feeling that the k8s system is not ready yet.

Normally you're never going to run a kubectl command in prod that changes anything except for kubectl apply -f. The yaml file may or may not be script or template generated but it's likely. In a lot of shops they're just going to write to a wrapper around the kubernetes API in whatever language the shop uses whether it be python, go or whatever. And there are plenty of linters for the yaml and ways to test without impacting prod.

> So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD.

>> None of those have any relationship to DevOps.

Deming. I think you mean maybe that Deming has less of a relationship to SRE (Google style). The DevOps Cafe guy wears a Deming t-shirt and talks about Deming all the time. Maybe he has denounced Deming for some reason?

Value stream mapping. This has been brought up in at least one talk at DevOps Days.

Six Sigma. There's some obvious overlaps if you look it up.

Thank you. I didn't want to get into a whole "thing" because the parent just doesn't know what they're talking about and I can't teach them everything in a comment thread, but suffice to say that Gene Kim and Jez Humble would agree with me, and that anyone who wants to know more can look up books and blogs by them.

Reply on news.ycombinator.com