“Who Should Write the Terraform?”
zwischenzugs.com
zwischenzugs.com
A pattern we're seeing increasingly commonly are Platform Engineering teams doing the bulk of the work, including all the fundamentals, guidelines, safety railing, and conventions, while Software Engineers only use those, or write their own simple service-specific Terraform Stacks which however extensively use modules developed by the former.
This does also seem like the sweet spot to me, where most of the Terraform code (and especially the advanced Terraform bits) is handled by a team that's specialized for it. If you don't have a Platform Engineering team, or one that is playing its role (even if its called DevOps or Ops or SRE) in even a medium company, you'll probably start having as many approaches to your infrastructure as there are teams, complexity will explode, and implementation/verification of compliance requirements will be a chore. Just a few people responsible for handling this will yield huge benefits.
And yes, I can wholeheartedly recommend Spacelift if you're trying to scale Terraform usage across people and teams - and not just because I work there.
Disclaimer: Opinions are my own.
[0]: https://spacelift.io
Presumably your core competency isn't building CI systems or job runners so why bother? I'm sure at the core of your own infra it's job agnostic The value-add is the management plane on top of it.
Overall, if your setup works for you and you're happy with it - keep using it!
We've seen a lot of companies (many now our customers :) ) try to build their own on top of existing systems (GitHub, Gitlab, Jenkins, etc.) and waste a ton of time and engineering resources, while ultimately not achieving anything that works well.
What Spacelift does is it gives you a bunch of much better-suited building blocks which let you build your required workflow very quickly.
And it obviously does integrate very deeply with your VCS provider - Commits, Pull Requests, Comments, etc. - everything is supported and customizable using - amongst others - Push Policies[0].
[0]: https://docs.spacelift.io/concepts/policy/git-push-policy
> ...you'll probably start having as many approaches to your infrastructure as there are teams, complexity will explode, and implementation/verification of compliance requirements will be a chore. Just a few people responsible for handling this will yield huge benefits.
Agree with the centralization of "how infrastructure should be managed/defined". A "platform" team composed of M platform engineers (where each platform engineer works 80% of their time for a given product team) can handle such centralization.
This is my experience as well. Having a single platform team has been a great experience for laying foundations, establishing shared architectures, and centralization documentation.
As soon as two or more teams need something from the platform team, it becomes a battle of priorities. A good platform team will recognize this and work on a division of labor and coordination strategy that can start to scale. A bad platform team will treat this as an opportunity to claim the company’s wins for themselves and leverage their bottleneck position for political gain.
The company’s management of the platform team is key. I’ve also seen a single platform team abused as the engineers who are expected to own all the hard work while other teams get to walk all over them with demands. This results in a lot of employee turnover, which is the opposite of what you want on a team tasked with holding the core knowledge of the company’s infrastructure.
Yet so many people insist that the one is the same as the other, when one is a duck and the other is an elephant wearing two swimming fins on its face.
I think reality is more complicated than a one size fits all approach. It's going to be specific to your org, your project, the stage it's at etc. To add to that, the right thing to do is often in flux.
Dedicated capacity is necessary, as is embedding. Not always at the same time or in that order. That's where only the information found inside the walls of your organisation can help you decide what is necessary to solve your problem.
The platform team owning base terraform functionality works well for the product teams that are the 3rd or 4th user of said functionality.
For the early days of the platform, and the early users.. your product is constantly in dependency & priority battles with said platform team. This is where "embeds" help continually unblock while making sure the work is done in a platform centric manner that will be reusable for other product teams.
Simply saying the product teams need to go down into the weeds at this level just puts too much disparate responsibility on product teams who exist to deliver a single product. Similarly it encourages vastly different approaches to similar problems, with all the wasted duplicate & re-work.
Having a team owning the platform doesn't necessarily need to mean shutting yourself in a cave. Granted, promoting cross-functional collaboration is a challenge in and of itself, but similar to OSS, projects that invest in the community aspect are the ones that eventually gain critical mass and set themselves apart from the rest.
Pendulum back to the center
The idea is not to go back to the Software Engineer asking the Ops team "Hey, can you provision a Postgres database for me please?" and then waiting a week for it.
It's that the Software Engineer takes a module that was prepared by the Platform team - i.e. "terraform-postgres-mycompany" - which already includes all the requirements the company has for handling databases (think backups, monitoring, encryption, etc.). They can then proceed to use it in the small service-specific Terraform configuration, which really is just putting such ready-made modules together.
The important bit being - the Platform team isn't a bottleneck here.
At my last org..
The old process of "Hey, can you provision a Postgres database for me please?" was managed in a web ticketing system, change managed, and had 24 hours turnaround! As was most other requests - VMs, NFS shares, FTP servers, network/FW changes, etc.
The new process was "Hey is there a terraform module for Postgres?" followed by weeks/months of prioritization and specification battles with the platform team. Somehow despite the platform existing 18 months, we were the first team to need a.. database? Then they didn't want to support Postgres and were forcing everyone onto Aurora. Then it look N months for it to come up in their queue.
Rinse&repeat with every fundamental building block of a cloud offering, ad infinitum.
This is why I strongly prefer the embed model until the platform has proven itself to at least be at the "80% solved" end of the spectrum.
I'm glad those 2 ops at my current spot finally have help but they were alone for years while the company was generating a ton of revenue which bothers me but seemingly was their own decision; nobody told them they were overworked until us more SR ops people arrived and went out of our way to pull work away from them.
Having all sorts of different teams (style, culture, language) relying on you to not gate-keep is.. stressful at times. I try to context switch as little as possible but sometimes it just can't be avoided.
now, a new requirement comes in for the product team(s) - a new service which bulk reads/processes another team's data (e.g., an export service).
Turns out this is too slow if the bulk reads use the usual web api route. So direct access to another team's provisioned database is required.
This now falls onto the platform team to produce a method for doing so.
Imagine this, but multiplied by N, where N is the number of different features being worked on at the same time, and all of them needs time from the platform team to produce something new for them!
It’s time for the ecosystem to move beyond the half baked config language known as HCL
Disclosure: put hidden info forward, e.g. source of potential bias
We could also probably use more abstractions similar to Pulumi, theres been talk on HN about storing all of the state for applications like this in the tags of the underlying cloud resources. There are some caveats with this approach, but it would provide a interesting tradeoffs
So, sysadmins and programmers - but with new 2020s vintage titles? (and renumeration...)
I personally think of "devops" not quite so much as being about "dev" and "ops" collaborating (though that is a noble and worthwhile goal) as about having "developer-operators", people who know how to do operations effectively and who can turn that knowledge into automated, generalized software systems.
The abstract modules and tools can live in their own repositories (or folders, in a monorepo), and your devoperators can work closely with the product teams to use them (and abstract specific changes to meet projects' individual needs to be more generally applicable).
Ayyyy, dios mio.
a) If you need to embed, then actually, you need to embed InfoSec, UX, IT, Customer Success, Product, Compliance, etc. etc. for exactly the same reasons. In today's labor-constrained economy, good luck finding qualified people for every role on every team! And if one of them leaves, who ensured that they documented everything for the next guy? Or that you'll find someone to fill the role quickly? If you have a 30 person company, fine, no big deal. 150+ and it starts to become a serious problem.
b) Particularly for infrastructure, you will shoot yourself in the foot on your production cloud bill. If you share no infrastructure with other teams, then you will find no shared efficiency in sharing the same infrastructure. Conway's Law will burn your runway. If you're 100% serverless then this doesn't really apply, but if you're spinning up eight different Kubernetes clusters for eight different teams then you probably need to collaborate a bit better.
Product teams need to own their product top to bottom. Platform teams need to make that easy for them, because modern stacks are huge, it's not possible to staff a single team with all the necessary experts, and all that expertise is a genuine necessity. The lines are drawn in different places in different companies depending on available labor and technical requirements.
Why? there can be reasonable scenario for that - say 8 reasonably seperated projects run by 100 people?
Also I do not see how being serverless "doesnt apply". It does apply because a lot of your infra is security, especially company-wide security configuration.
I understand the deeper meaning of the message, but at the same time devops is a thing because it likely hurt more than other cases mentioned. But I think the whole thing is often a balance between integrated / standalone.
Every team and project requires breathing room but also requires certain level of integration. Devops was needed and is proceeding - find an engineer who has no docker experience today, compared to the past where often engineers had 0 idea of delivery. Other groups may rise their own requests if they feel, but they will lose some flexibility from being standalone.
If those people are part of your core value proposition, the thing that's supposed to give you your competitive advantage, then yes (though if you need all of them, you probably don't have a very good value proposition). If not, if they're just a cost center doing commodity-level work, then they don't need to be part of the product team - but in that case you should be looking to minimize or outsource them.
> Product teams need to own their product top to bottom. Platform teams need to make that easy for them, because modern stacks are huge, it's not possible to staff a single team with all the necessary experts, and all that expertise is a genuine necessity. The lines are drawn in different places in different companies depending on available labor and technical requirements.
If the "platform team" are doing something so independent from the products that they don't need to be part of the same team, why are they in-house at all? If you're offering a generic platform, either you're doing it better than Amazon and should be in the business of competing with them, or (more likely) you're doing it worse than Amazon and should just use Amazon.
Someone needs to answer to Compliance, to InfoSec, to Finance. Someone needs to make sure that they all understand exactly what production looks like in their language. Compliance wants to know whether we keep EU data in the EU. InfoSec wants to know whether all our code in production passed security review. Finance wants to prevent costs from spiraling out of control and to judge which projects to fund.
Good luck trying to get AWS's "platform" to do any of that as a managed service, out of the box and without any in-house engineering time!
aka, a full-stack engineer! The idea that you have some specialist take care of each role in a team is just fantasy.
Get a smart person, and train them full-stack. Including customer success (aka, sales and after-sales support), compliance (i mean, GDPR is required understanding now, so might as well be the engineer who knows it).
This is exactly the situation I´m currently in. Company decided to migrate from big on-prem kubernetes to AWS. Now every team got their own account and well... good luck, you´re on your own now. We´re a small team of three developers. Although we have three certifications under our belts (AWS Dev, CKA, CKAD) it took us almost three months to configure AWS and set up the Terraform pipeline and define processes like "upgrading cluster". The "enabling" part was basically missing in the whole cloud strategy of the company. It was more like: good luck, you´re on your own now.
In fact we made contact with a neighboring team. Only to find out that their use case was so different from ours that collaboration didn´t make any sense. For them Kubernetes was not a good fit, for us it was the way to go.
Speaking of sharing a cluster or AWS ressources: we figured out that it is not allowed due to billing reasons. Company policy is: One product per AWS account.
If you ask me I see a shift of paradigms happening here. Now you hear a lot about "enabling teams" instead a dedicated team for infrastructure providing services (e.g. the Kubernetes podcast from Google). I´m not convinced yet. I think this is more like kicking down responsibility down the chain. And then it feels more like: Someone needs to do the dirty work but nobody wants to do it.
It might work if you don´t have to provide Service-level agreements (in our case: we don´t). For us it is just more work to do. And our work shifts from dev to ops. Instead of writing software we´re mostly busy with configuring cloud resources. This will ease a bit once everything is running. However: I see this whole change more as ... uh, strong word... ideologically motivated. Cui bono? Neither our team, nor our users nor our infrastructure bill.
That's kinda funny because half the reason why AWS has tags in the first place is to get finer granularity into understanding billing. Not to mention products like Kubecost. Sounds like whoever wrote the policy doesn't understand how AWS works.
> Someone needs to do the dirty work but nobody wants to do it.
There are plenty of people willing to do the dirty work, they're just already working for other companies and their salaries are quite high. The labor market is tight.
> Cui bono? Neither our team, nor our users nor our infrastructure bill.
HR benefits. Having open positions that HR is failing to fill is a bad look for HR.
I think the instinct is that if you’re going to take the moral high ground, you’d better walk up the hill and join us first. And the simplest way to do that seems to be the obvious one, which is to combine them under the same org chart and governance.
> "There were endless complaints about the time taken to get ‘central IT’ to do their bidding, and frequent demands for more autonomy and freedom. A cloud project was initiated by the centralised DBA team to enable that autonomy. [...] Cue howls of despair from the development teams that they need a centralised DBA service"
Author makes it sound like users didn't know what they wanted. This is not true -- I have seen this play in practice, and what author omits is _it was a different set of people_ who were complaining before and after.
If a dev team has at least 2 engineers who are happy working with infrastructure, then the team will benefit from autonomy. If there is no one like that on the dev team, they will cry in despair.
That's something I've observed as well. Seems to me there are (at least) two developer personas, one kind only wants to deliver their task, specialize in what they do well, and generally can't care less if their DB is oversized or has no maintenance windows set or no recovery plans or who has access to it etc. They usually lack the cloud/platform skills as well, and won't develop much in that regard because they don't care to. Even if they did, they're unlikely to get rewarded much for that effort. They are easy to make happy, and you rarely hear from them other than the occasional "thanks" in some Slack channel.
The other kind is either internally very curious about the subject, or already has the experience, or at least they think they have it. They want to have full access, invent things anew in the "right way" they believe. For them there is nothing worse than relying on another team while they could geek out on the subject themselves and they believe they could do it better. Sometimes they're right about it, and other times they're either oversimplifying the work needed, or optimizing locally around themselves/their team/their task. There seems to be no way to make them really happy other than giving them complete freedom to do whatever.
Seems to me most developers (I've worked with) are of the first kind, and they can be made happy after some level of maturity is reached within the company, but the second kind is way more vocal and they won't ever be happy with whatever a central team builds.
More and more I'm getting convinced that the only way to really win both personas is to build two products instead of one. So you build the golden path, the Helm chart or the portal or whatever for the first kind, and give ownership and loosely govern with compliance/policy tooling with the second kind.
This optimizes for the short/mid-term satisfaction, but of course it can also go wrong since team compositions are not set in stone and what one builds may not be maintained properly by the other, and there'll be some duplication of efforts and quality of solutions built might vary between the teams. I guess for some companies this is acceptable, and for others it won't ever be.
tl;dr ¯\_(ツ)_/¯
A lot of time the centralized ops team is very slow or just not very good. Your tickets may take weeks to be processed, or critical requirements are ignored, or maybe central ops team only cares about closing tickets and does minimal possible work to satisfy the letter of the requirement.
If there is no one on your team who can do better.. well you suffer and work with central team. But if your team has someone who can do this and the autonomy to proceed, then your can work much faster -- no need to wait weeks for to allow the other team to access your data, you can grant the permission yourself in under a day.
My current place is just awful. Over complicated architecture born from a platform team that couldn’t be less helpful, so people have worked around it with all sorts of hacks.
It may not be optimal, but it’s almost invariably faster.
Personally I like the way my company does it, where people have (more or less) full access to the AWS account, but there’s a lot of automated guardrails/scanning that alerts you when you’ve done something stupid (public S3 buckets etc.)
For what it's worth this is how AWS operates, and I think it's the mindset with which they build products. You certainly _can_ go your own way and run something like k8s on top of it and build a mini-cloud in the cloud, but it's incredibly expensive.
It's a mistake I've made repeatedly -- "Oh, I'll just add this little abstraction to make it easier for developers!" But now the poor developer has to understand both the tools I built on top of _and_ whatever I was thinking at the time, and inevitably it's an under-resourced area.
Now, at a certain scale for core services, sure, you'll end up with infrastructure specialized folks. But I'm unconvinced that the place you want to start is, "Okay, I need a new service, better go talk to the beleaguered central team that never has quite enough time for anyone."
Sure, this does not excuse most traditional big corps that have huge internal engineering budget yet force a top-down rigid inefficient structure. (Though again, it takes a very principled way of doing things to be able to scale out and keep things sort of consistent and coordinated.)
(This is of course much more true for greenfield projects, early stage stuff. Of course the giant services are large and complicated.)
As for me
1, HashiCorp is forcing enterprise upsales whenever possible, even if it'll hurt Adoption Rates and overall Development Experience
2. Existing TF design issues are ignored, which is causing people some state management trouble irrelevant for TFE. So, yet again, why fix something that will end up in upsales ?
3. MPL requires for the PR's to be available in case someone will really fix something, but it's near impossible to contribute into Terraform with any major design improvements.
4. Existing Providers issues are neglected, and Accepting Working PR's takes around 3-4 weeks...
5. Some Providers (helm) are neglected in favour of the New Product Release (Waypoint provider) and there a Forced Obsolescence Factor alongside with Forced Adoption.
Deficient Relationship Marketing is the Key Factor in deciding who Will actually write Terraform (maybe not even HashiCorp), Who will Wrap Terraform and Into What (terragrunt, terraspace, pulumi, crossplane etc or some custom gitops SaaS), and Who will Support the target providers when Hashicorp solutions will magically turn into an abandonware due to upsales.
Most devs I've spoke to are in this camp they don't want to do any Ops work at all. They want a 9-5 job without evenings are weekends wasted by services failing. No on call rota and all that jazz just writing code that's all.
Tiny organisms like amoeba can be simple. But as organism size increases, so too does complexity. They eventually need a nervous system, circulatory system, extra sensors, a more powerful brain to process sensory information and handle movement, motion tracking for hunting. Suddenly, packs of these animals will hunt together, so they'll evolve communication: signals, sounds, language...
Well, if you're a 4-person start-up sitting in the same room, decisions can be made quickly, you don't need departments, managers. But as you grow your need to be extremely careful that you build a nervous system, circulatory system, sensors ... "management brain".
The biggest failures in ops aren't "who does X?". It's about creating right-sized teams that own functions that are important enough to have specific owners. With further growth, certain functions get more complex, and suddenly you might need dedicated network, database & security teams. And if it gets huge, then you probably need to need multiple copies of those specific functions embedded inside large subsections of the organisation. And they all need to communicate effectively with each other. It's a constant dance. You can't make a single rule and just stick rigidly to it. You need to keep tabs on complexity, workload, morale, lead times. You need to be ready to refactor your teams.
When I hear stores like "it was taking 8 weeks to get a DB provisioned" I think "if that company makes it to IPO and the CTO gets a few $100M, there's absolutely no justice in the world".
"What’s the point of this long historical digression? Well, it’s to explain that, with a few exceptions, the division between Dev and Ops, and between centralisation and distribution of responsibility has never been resolved. And the reasons why the industry seems to see-saw are the same reasons why the answer to the original question is never simple."
It is true that the answer is context dependent. I consult with several startups, I give different answers to different CTOs, depending on what stage their organization is at, and how much they will actually need devops in the future (I recently consulted for Paireyewear.com, a company that relies on Shopify to provide the public facing store through which they sell. As such, they will never need much in terms of devops. Instead I brought in Chris Clarke, one of the best devops talents I know, and he consults with them part-time, and that is as much devops talent as they need.)
Its never been resolved because people try to have their cakes and eat it too. There's pros and cons to both ways, but people refuse to deal with the cons. Dealing with that in my current org, where a decision was made to distribute a specific subset of responsibilities, and as soon as it gets even a little difficult, they start centralizing again (within that subset), even when there's solutions to the problems.
So we end up in this weird kind of Frankenstein organization, and that's the worse of all worlds.
"plus ça change, plus c'est la même chose"
It got me thinking, here at amazon, we deliver "infrastructure as code" using the Cloud Development Kit: https://aws.amazon.com/cdk/
We expect engineers (not devops) to define their infrastructure in typescript and configure it through code. That code gets turned into cloudformation scripts and stands up the how cloud system for the api you're building.
I think this is a great hybrid approach. Knowing what you want is different than knowing all the intricate details of defining, say an API gateway. But the CDK lets me stand up an API Gateway and configure it with a swagger and security policy and be done. This lowers the barrier for devs to do devops work, and lets teams own and move fast when making changes.
https://github.com/pulumi/pulumi-aws-native is nowhere near GA state, just scroll through the issues...
and the respective terraform wrapper https://github.com/pulumi/terraform-provider-aws is somewhat neglected in favour of the tf2pulumi native port above.
Crossplane, on the other hand, does better with the Terrajet codegen, and all the infra drifts are a part of the reconciliation cycle, which is very handy on simpler deployments but doesn't work with more complex ones due to excessive drift polling model.
Had we built this out in terraform, state cleanup and tracking would have been more robust, the ability to retry resource creation would have been more stable, the project overall would have been much more of a pleasure to use. The functional/declarative aspect of terraform in relation to cloudformation is so much more polished.
Edit: declarative
I'm leading the engineering at a startup based in the UK and we actively chose Pulumi over CDK, TF etc....
Been going 12months now and we're really pleased with the decision so far.
2. Both TFCDK and dagger.io can be used for multistage TF deployments, although I prefer dagger myself...
Terraform has a major state management design flaw that had been ignored by hashicorp to force TFE upsales. It's impossible to perform multi stage deployments with a single `terraform apply`. You have to manually identify the deplyoment targets for every stage, terraform providers do not support `depends_on` block and they are not a part of the resource dependency resolution graph. i.e. You can't deploy Vault than configure it with the respective provider - terraform will try to perform both deployment and configuration simultaneously and will fail.
3. This is due to strong Sales Opinion that a Single Plan is of a Positive Product Value for Terraform. While in practice it turned out to be False, the actual Product Value of Terraform is in Single Consolidated Infrastructure state, which can be analyzed by the respective static analyzers (infracost, tfsec, checkov, inframap, driftctl etc). And it's a strong pro compared to both Pulumi and Crossplane...
Having a single state is a blessing for large companies with a tight operational schedule - having multiple states with a single lock can cause conflicts quite often, with volatile outcomes. Yet again, an upsale point for TFE.
Even though Terraform "has more providers" you have to be able to support 'em all by yourself, HashiCorp does not provide a Viable Support Plan for the existing Official Terraform providers (on my xp - maybe someone was more lucky).
That's why I'm often saying that DevOps is not a title, it's a methodology... and every DevSecOps guy should be well versed in golang to be able to support, test and extend the respective tools and operators (k8s automation).
* answered in a bit more detail below https://news.ycombinator.com/item?id=32405064
Early in my career, I worked for a business that still used mainframes, and we had this random bug that caused processes not to communicate and drop messages. Because I was willing to dig into the intricacies of server administration, I was able to diagnose the problem as the IPC queues size being set to the kernal default, and the default only allowed for a few seconds of messages to back up. It was a quick fix, deploy a new kernal parameter to allow for bigger IPC buffers, but if I'd refused to do "devops" work, we'd never have found the problem.
As my old boss said, you're not paid to do only the easy stuff. You're paid to do the hard stuff too.
Indeed. That's what I have done (I do infra stuff + product features). I don't like it, but I do it.
> As my old boss said, you're not paid to do only the easy stuff. You're paid to do the hard stuff too.
This intrinsically states "product features == easy", "infra struff == hard". While I think that certain topics related to infrastructure are hard, certain topics related to product feature are hard as well. What I think your boss wanted to say is "I don't want to pay an extra paycheck to an infra engineer. So, I'll pay you 20% more so you do the infra stuff instead. And you get to 'grow as an engineer'. It's a win-win!".
On the flipside, having a single platform team write the IaC components that do all of this grunt work tends to reduce the degrees of freedom that the engineer has to build the application architecture exactly how they want it.
However, for others on my team coming in without the CF background, it feels a little like voodoo and as soon as they tread off the "standard path" I find myself getting pulled in to do the "hard stuff".
I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud provider and they go off and do so. They still fill out and track change logs, audit changes and seek approvals - but after that's done, it's still the team who presses "the button".
Anybody in a regulated industry where you've hit hard walls that prevent you and your team from going full on DevOps? If so, what rules were quoted that stopped you.
This is what was cited for us.
ISO27001:2013 A.6.1.2: Segregation of Duties. Conflicting duties and areas of responsibility must be segregated in order to reduce the opportunities for unauthorized or unintentional modification or misuse of any of the organization's assets.
The idea comes from finance -- to require collusion to execute a fraud. It's not perfect, but it's something.
But it's not that cut and dried. There are degrees of rigor.
Others here have cited segregation of duties, which is definitely a factor, but the other one less mentioned in finance is the 'one throat to choke' principle: it's simpler from a management and regulatory perspective to have the responsibility for failures in one place rather than across many teams.
I feel like most of the time "compliance" is blamed when really, it's your first point in that section (Absent an existential threat, the necessary organizational changes were more difficult to make) that is the real holdup.
When people keep yammering on about devops as a principle of people and processes they’ve already lost because processes are meant to replace people, so really all that matters are the processes and the services that fit into the process SLA and OLA.
Note that in a big organization what really matters are your particular regulators and arguing with your regulators claiming to know it better than them is probably one of the fastest, reliable ways to get fired I can imagine that won’t result in a criminal lawsuit against you.
This matches my on-the-ground experience. The teams who lived the dream of DevOps were teams which built their software as cloud native (instead of later trying to migrate to the cloud). This is purely because the PaaS tooling let them efficiently be both Devs and Admins.
When you involve many teams instead of just a smallish group of devs, you have momentum to deal with. Plus, specialization - some of these ops people just don't like coding, or at least not the kind of coding you need to be doing to be effective DevOps engineers.
Indeed this leads to SRE - just because "Buying it" is usually easier than "Building it".
Or the total opposite. People that not drink the cloud-narrative and the operational simplicity make devops no-brainer.
ie: You are truly made for the massive overenginering of cloud (because you ACTUALLY need that) or you keep things simple enough to fit in a single head.
What is problematic is when you try to do be first, or present to be second.
> This is purely because the PaaS tooling let them efficiently be both Devs and Admins.
It wasn't the cloud which made them successful, it was the level of automation baked into the PaaS solution(s) they built on top of. I thought I was pretty clear on that, but I suppose I could have been more clear.
You could similarly get success if you had an OnPrem solution with really great orchestration layers. And in fact, I've heard of such successes (but not seen up close) of teams doing exactly that with Pivotal Cloud Foundry.
If you've built your project as cloud-native from day one, you'll have a lot less DevOps work to do. You're basically just writing code or templates that apply cloud-based configuration. That's not to say there's no complexity, but it's not unreasonable.
If your org has already gone all-in on microservices and Kubernetes, there is a much stronger case to be made for centralized Ops. The amount of understanding, care, feeding, and training necessary is much higher. You won't be able to get by with one or two contributors occasionally making changes to Terraform templates as necessary. Clusters are expensive, require occasional upgrades, and centralized metrics and logging don't come for free and require their own access control. It's still better than it's ever been, but it's a lot like building and maintaining your own cloud, which quickly becomes a full-time job.
Security and network teams also built custom terraform modules which the deve teams were forced to use that were guardrails. You didn't use aws_s3_bucket, you used custom_aws_s3_bucket that mandated certain fields and prerequisites. This was the compromise struck to allow devs otherwise to go ham in their own AWS accounts and self-manage their deploys, databases, and so on.
It is a nightmare of arcane copy pasting.
Terraform is an overengineered mess, a complex enemy I need to beat to deploy my simple changes.
How we got here? Business doesn't want to pay for a well designed enterprise and the organization is shitty, so hire people who aren't very good at tech to build an unnecessarily complicated engineering organization that [after they waste millions poorly building cloud tech without prior experience, realize is] still a cost center and tell them to chase fads.
Factor One: Non-Negotiable Standards. Tell everyone they have to do the same thing, even if it makes no sense for what they're building or supporting.
Factor Two: Engineer Capability. Make sure you put unrealistic deadlines in the hands of amateur engineers and then turn up the scope creep.
Factor Three: Management Capability. Make sure your management can always blame somebody else for why your ridiculous initiative and poorly managed company didn't achieve its goals by its stated deadline. Market timing and "I didn't have enough resources" are good stand-bys.
Factor Four: Platform Team Capability. Pay a million in salaries to some middling full-time engineers, put them in a silo, make them build really basic tech from scratch that 50 different managed service companies sell for pennies. Don't Scrum with the teams that will be forced to use it. Make sure everyone is required to use the platform, even when it's not actually ready to go live, so that building any kind of product at all is mostly infeasible, incredibly slow, and painful.
Factor Five: Time to Market. Do everything you can to avoid value chain analysis, training employees on standard practices, unified communications, or getting stakeholders to work with you on initiatives. When your competitor lands a feature a year earlier than you planned, blame the consultants/contractors you never listened to.
Who should write the terraform? An overworked systems engineer in a siloed team. Definitely not someone working on the product. This way they can write 5 layers of unnecessary module abstractions, be unaware of how non-functional the module is from not actually running it on the product [and watching it fail 6 ways from sunday], and still not provide what the business needs.
"we aren't living in 2016 anymore, and the cloud moves fast. Platform teams are expensive and hard to do, offer a mediocre service at best, destroy velocity, and create bad incentives." [1]
[1]https://twitter.com/iamvlaaaaaaad/status/1534489514818686976...
I love the idea of cross functional teams, but from what I have seen of the most recent implementation of it that I'm working in, there are as always, issues of definition around what a cross functional team actually is and should be.
IMO grabbing a bunch of backend SEs and making them handle their own DevOps is a joke of Academy Awards host level proportions. The shit I see as an ex DevOps dude is horrific. The notion that a bunch of people who've never done the role can somehow figure it out without specific training doesn't work, from my experience.
A cross functional team should actually be cross functional, where you have an engineer, whose specialty is the work you intend for them to complete within that team. Otherwise we're just being overburdened with extra shit that we frankly will never get the time to actually complete in a meaningful way, and it just generates more and more technical debt.
So, either way, a large organization is going to punt that terraform/cfn/cdk template down a pipeline with a bunch of automated compliance reviews. Whether the App team or Ops team wrote it.
Then I moved to a team where Ops write Terraform but also expected developers to contribute. They pitched this as "Developers should be able to make small changes". Turned out we had very different understandings of the definition of "small".
I'm currently in a team with no Ops and developers are fully responsible for managing infra all the way to production. The Terraform implementation is an absolute mess. There is, however, an understanding that it needs fixing and Ops support has been promised.
My answer to "Who should write Terraform" is it's the Platform Engineers. A developer can maybe optionally pitch in if they feel confident enough but ultimately Platform Engineers should own the platform.
My personal take is that DevOps doesn't work (for me, and probably many others) because it amounts to context-switching (recently featured on HN: https://news.ycombinator.com/item?id=32390499). By being responsible for both Dev and Ops, my time (and my brain) gets split 50/50 into two entirely different sets of:
- Concerns
- Languages
- Tools
- Mindsets
This is both super draining, and counter-productive, for me. If Ops can be made so simple (by a platform team or otherwise) that it doesn't amount to a whole separate headspace, then great, I'll manage instances myself. But as long as it's a whole separate domain, trying to have one foot on each side of the fence is just not going to be workable.
If you are writing application code with 0% of your brain lent to operational concern, you are writing it in a disconnected dream cloud and your organization will have problems that come from it.
The answer to context switching is not do it several times per day. The answer is NOT to ignore operational concerns and mindsets.
Many engineers pride themselves on having a wide set of skills and enjoy switching contexts, languages, tools and mindsets for variety and to reduce boredom.
Help... me...
Anyway, I've got the members of my dev team writing terraform for their changes now too. It's working, more or less. They are excited to do it because it pads their resumes, because it's new. But we continue to increase demands on devs, they need to get paid for their trouble or the responsibilities must be diffused.
Credit to https://developers.google.com/tech-writing/one/lists-and-tab... which pointed this out and has stuck in my craw ever since.
1. Scaffold your infrastructure with simple point & click in web console.
2. Generate terraform/CDK code by scanning your AWS account with typically available tools.
3. Edit an update said Infrastructure as Code as needed, swapping out the parameters with the vectors you need to change according to CI/CD
The whole "i want to write infrastructure as code from day 1" is not only stupid , its a waste of resources.
I tend to disagree. Depends on the scale... and after you've scaled and grown DevSecOps absence becomes a source of detraction, affect your delivery cycle and indirectly your Sales. Proper DevOps defines some of the business lifecycle operations as well, like BI and A/B testing, which essentially helps in validating pending Business Assumptions. It's something that can help differentiating the market and Validate the actual Product Viability - prove that your MVP actually has any V in it.
Operations wise, First and foremost you have to keep track of the issues that are currently present in AWS solutions and automate workarounds, and there are a lot of security automation and organizational means which can't really be solved with a "Click in Web Console" efficiently.
For instance, setting up a proper EKS cluster by hand, without any hardening, would require at least three hours of clicking through, with all the IRSA roles and EKS specific IAM permissions. While, on the other hand, Terraform automation has ready to use OpenSource modules shipped by both the community and AWS itself (terraform-aws-modules, aws-ia), which introduces some advanced EKS management practices, without any added effort. 10 lines of IaC can easily replace half an hour of click-through.
The cost of Integration is nearly Zero during the product bootstrap phase, but when you're growing integrating proper Organizational Management with AWS Organizations and Control Tower, reordering your AWS Accounts, transferring resources, and hardening security boundaries tends to rise in complexity and cost a lot. Especially if you'll ever want to perform proper security Audits or need some HIPAA/GDPR compliance.
For some Disney companies, for instance, who choose to perform org management by developing custom tools after 5 years of operation, proper integration with AWS Organizations remained a dream, and their unreasonably tight Operational Schedule and On-call deficiency became a source of detraction. The integration cost rose to eight figures.
The cost of DevSecOps hardening basically doubles every quarter, if you're growing fast enough and lack automation.
As for myself, automating everything allowed me manage Kubernetes complexity and develop a fine tuned vertically scalable solution (VPA+HPA on Keda with cluster autoscalers) - about 30 different k8s services deployed in a mix of x86 and Arm instances, with continuous placement and resource limits/requests optimization, completely downscalable. My AWS bill is only 7% of my raw income.
So, if you can hire a DevOps consultancy, and can Actually Measure how much time is wasted during the manual operation compared to the automated one, able to self reflect without a confirmation bias, do that ASAP.
But this is just because companies wanted to put the "ops" work into the shoulders of developers. What should be done is to hire one (or more) specific "ops/platform" engineers per team. Such engineers are the gateway for the team for all platform-related stuff. I'm not talking about SREs here. I think SREs are more about making the products as performant and efficient as possible (while platform engineers per team are more about setting up infrastructure). Sure both roles (in addition to the SWE role) do their job best if they are working together in the same team.
What I see nowadays in small and mid-size companies is either:
1. There is a "platform" team. They own infrastructure repositories, but they let product teams to make PRs to such repos (e.g., the platform team usually creates some kind of guidelines for managing infrastructure, like "How to create a staging mongo db"). The "platform" team is on charge of reviewing such PRs and merge them. Now, there are certain aspects of the infrastructure that only the "platform" team can actually work on (because the product teams either don't care about it or don't know about it). This doesn't work because the "platform" team becomes a bottleneck when the number of product teams starts to grow (the #platorm Slack channel becomes a nightmare with dozens of requests per day. Many platform engineers end up burned out because they see themselves as "customer service" for developers)
2. Developers pushing "product features" and at the same time they do "infrastructure" stuff. Companies usually call this "you build, you run it". In reality it's just cheap management (companies don't want to hire infrastructure engineers and they think the developers are excited to learn "docker/k8s/aws/gcp/terraform", so let them have fun). This is ultimately a nightmare for many developers because they end up burned out ("I want to work on product features! I don't want to fix GitLab pipelines").
I think the original idea of DevOps is totally valid. Just don't force your SWEs to work on infrastructure stuff. Instead, hire one or more infrastructure engineers for every product team you have. This way SWEs (dev) and infrastructure engineers (ops) can work close together and push stuff faster. Obviously almost no company is doing this because it is more expensive than the alternatives stated above.
Would you let your SWEs to design your frontpage? No. They obviously have a voice in the process of designing the frontpage, but ultimately the ones that should design it are your Product Designers (obviously for this to work, both your SWEs and your PDs should work in the same team).
In the future, I'll just link to this piece.
Would love anyone interested to give it a spin at withcoherence.com and please feel free to ping hn@withcoherence.com with any feedback or issues!
We talk to a lot of Terraform users from a lot of different companies. The most popular way of doing things is having your SRE/DevOps team write the bulk of the Terraform modules for your organization. Other members of engineering then consume these modules to create resources for their platform/application/etc. This code can either live in a Terraform monorepo or inside an application-specific repo. We've seen many approaches.
Scaling Terraform inside your organization is incredibly convenient with Terrateam as we leverage many pieces of GitHub.
[0]: https://terrateam.io
B: “Hashicorpse strikes again.”
I presume this self-assured statement comes from ignorance and youth? Can you detail the sysadmins that you've worked with that didn't understand code or the underlying infrastructure they are tasked with supporting better than you or your team?
Seems like this should solve GP's complaint while allowing ops/support to "own" once it leaves dev.
not really an ops thing. This is just the reality of large corps. Microservices are the same way, with devs partitioned off. You put in a Jira ticket and wait a month for a turnaround on a task that would take an hour to do. Eventually management wants it done now and, like always, they bust open and manipulate the process for their own purposes. Rules for thee, not for me.
Good luck. Let me know how that works out.