Comment by jiggawatts | Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Parent Full thread

I'm looking for opportunities to implement GitOps at $dayjob, but there are some challenges:

- Even developers don't know how to use Git properly. When it's just "code" with no actions associated with it, a mistake can be undone without side effects. With GitOps, a mistake can break infrastructure automatically. In a sense this is no worse than CI/CD, but I've found that many teams are allergic to even that level of automation, let alone CI/ plus automatic infrastructure changes.

- You're going to say something about repo security and policies to prevent uncontrolled merges into 'main', or something like that, right? Okay, now you've reinvented cloud IAM/RBAC and policies, but with none of the tooling around the cloud infrastructure roles and policies that the native platform has. Good luck establishing a new security policy that doesn't hamper productivity by getting in everyone's way, but also enforces reviews and tests!

- Most developers know nothing about infrastructure, and don't want to. With IaS their code has "infrastructure effects" that they can't avoid. It takes an immense amount of training to get people skilled up enough to understand the consequences of their actions. I've had an uphill battle explaining to developers that they need to regularly update their container base images for security!

- SecOps would have a fit, because with GitOps there are virtually no guardrails. Either you allow anyone with commit access to deploy load balancers with public ingress, or you're not really doing GitOps. Either SecOps needs to be involved[1] in Git merges ("DevSecOps"), or they're forced to put roadblocks in place that break the elegance and efficiency of the whole thing.

I like GitOps in principle, but at larger enterprises with random developers and minimal cross-team cooperation it seems like wishful thinking...

[1] For reference, I'm yet to meet anyone in a SecOps role that has ever installed Git, used Git, or knows what a commit even is.

I find it best to start small with this kind of thing.

Have a single team start practicing GitOps, with an eye to eventually accepting contributions from other dev teams. Establish good patterns in IaC repos, write documentation, add linters and other static checks over time. Tools like Atlantis and ArgoCD can provide deployment automation as well as rollbacks, with opportunities for manually diffing and reviewing changes before applying them.

Edited to add: if your "GitOps" implementation involves anything more complicated than the standard "short-lived PR to mainline" pattern, you are doing it wrong. Avoid environment branches at all costs and follow continuous delivery -- deploy incremental changes frequently.

Upvoted for the edit

Your workplace is not a good fit for GitOps (right now?). Anything *Ops needs political willpower to make them happen, you as a lowly IC have no leverage to implement something like this successfully. Sorry to be a downer.

Or maybe it’s just the wrong idea…

Maybe I don't understand what's meant by GitOps - is it the concept of Infrastructure as Code, deployed typically automatically after merge? (Or whatever triggering event.)

I don't necessarily view GitOps as requiring Product Devs to do Infra Stuff. I mean, you're right, people specialize, and you will have some who are deep in the Infra weeds and some who aren't (they're deep in other weeds instead, hopefully).

At my last job (disclaimer: tiny company), we did lots of GitOps. Infra stuff was cordoned off into its own repos generally, which made security controls and such easier for sure. Not sure I get your point there, though, because if you care about insider risk, you need the same style of controls for the product code anyway; it's equally important.

We had some enterprising product devs who contributed to the infra repos, but as you observed, most didn't. Most of the infra code was maintained by the 'Platform Team' (essentially SRE-adjacent).

Right this just creates a bottleneck in the organization where you need an infra person to do anything. It’s not self service which is the best pattern we’ve found

100% I’ve never seen Gitops implemented in an organization that has been successful. Most devs just do get it and shoot themselves in the foot with it.

This gets into gatekeeping in software. “I’ve built this perfect system and if you don’t get it you’re dumb” or “we just need to create a culture where people do this stuff”

These are unhelpful ideas in an organization. We should be meeting devs where they are and empowering them. Most devs don’t have time to fully understand infra and for 90% of use cases that should be totally fine.

Learn from Golang and Python. You don’t need a perfect system, what’s most helpful is to have a system people understand and can use to produce value.

GitOps not necessarily means developers should define every aspect of it. In my current work, SRE uses gitops and they are the main contributors to most repositories. But things like creating an AWS ECR repo for a new project or creating k8s resources is happily delegated to developers.

Reply on news.ycombinator.com