The development pipeline is a production system
sundry.jerryorr.com
sundry.jerryorr.com
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.
I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.
i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.
It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".
Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.
I also don't find most QA to be worthwhile compared to a good, quality focused dev.
There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.
Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.
Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.
Of course, in these days of vibe coded madness they're probably getting more useful, not less.
A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.
And the whole process improved all across the board.
Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.
The other companies tested "on demand."
Small company in Germany, QA team with about 5 people. But not all the devs are bad, just some are too comfortable with AI and getting lazy.
I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.
The premise is that Software Development is akin to Production which it should be.
And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.
Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR.
This isn’t only for your (deployed) Production environment.
It’s also not software development 101. This has roots in ITIL.
You’re the one drifting all over the place with the terminology.
I get what you’re trying to say. But you’re saying it badly.
You have your dev, test, UAT, production, etc environments. They’re the “software development” envs as you named them.
You then have your DevOps envs. You call them sandbox, but Personally I hate that term because the DevOps envs should be the same as the “software development” envs just not accessible to the software developers. A “pre-dev” env if you will. So it should be the same IaC and the same build pipelines.
Whereas a “sandbox” generally suggests that it’s “ClickOps” / manually provisioned envs with no rigour. A place to play and experiment before you’re ready to commit your infra to code. Which is not how the DevOps envs should exist. The DevOps pre-dev envs should still be a carbon copy of the “software developer” environments.
> ithin your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR
No it shouldn’t. HA, DR, etc costs money. Blue/green is slow and you shouldn’t let DevOps be a blocker for development feedback loops. So you save that stuff for your pre-production envs, and keep your dev and test envs nimble and quick to deploy. The DR for your lowest environments should just be “destroy and recreate” because you’ll have pipelines to deploy that infra.
In short. You can still have the same approximate infra but you shouldn’t emulate production best practices precisely across all development environments.
If you find you cannot have a “production-like” dev but without the HA, DR, etc, then you have designed your pipelines wrong from the outset. Eg You’ve got “pets instead of cattle” (ie the lack of ephemeral infra), manually provisioned services, and other faux pas.
> This isn’t only for your (deployed) Production environment
It isn’t. But it’s also not for all environments indiscriminately either.
> It’s also not software development 101. This has roots in ITIL.
Of course it’s software development 101. If your developers don’t understand the basics of different software development environments then they’re either shit at their job or fresh out of uni. I don’t mean to be harsh here but I don’t know how else to emphasise that what you’re describing here is the absolute basics. 30 years ago, when I was starting out, I learned this before I learned about version control. That’s how basic a topic this is.
So we started being more vocal and management understood its for everyone and their wish got approved recently and its been a big change.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.
If anything, recruitment in our industry was always broken. And yet here we are.
In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.
What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)
And if production is down even if there is just one user, it is all hands on deck.
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.
Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.
Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.
Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.
The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
There is a production eco-system - let’s call it The Hotel. It runs, people work in it and customers pay and it makes money.
But it constantly needs changing. New rooms need to be built, the kitchen rearranged as the chef changes his mind.
So they hire an architect and building firm. Who plan the chnages and then this awesome automated robot comes and builds the extra room.
The architect firm owns and is responsible for the robot and its changes.
Not being able to make a change really really matters, but I don’t call that production because the hotel is still able to operate
Most of the misunderstanding between business and tech is a misunderstanding of are you working on the architect firm or the hotel
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
Deploy was an rsync of a few directories to a server. That server directory also held .env.local, and a content/ dir that a scheduled job writes to directly. Neither is in git. Someone (me) added --delete so the destination would "match source", and rsync faithfully removed everything outside the source set: the env file, the five .bak copies sitting next to it, and a week of generated content.
Two accidents kept it from being worse. The rsync excluded the build output, so the already-running process kept serving; and because the app had read its secrets into memory at boot, checkout kept working off a file that no longer existed on disk. That bought a few hours to restore from git.
What I'd add to the article: treating the pipeline as production should include auditing the pipeline's write scope. Mine had unrestricted write+delete over a directory holding live credentials, and that was invisible until it mattered. Also — backups in the same directory as the original are not backups.
Because long running, well tuned pipelines can deliver on some important goals but be broken by the changes from adopting a new model (ahem opus 5) or rot in some key web automation path.
There’s the system you are building…and the system that builds the system or the SDLC. Both are critical.
Solid read.
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).
It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.
Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.
Sadly, engineer-driven companies are increasingly rare.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
zZzzZ...
2:35 AM
ALERT #42 - CRITICAL
!!!
ALERT #42 - CRITICAL - Joan forgot feir password to npm
Help feir!
The issue is "production is defined by what humans are expected to maintain all night long" and dev ain't prod broskies.