When deployments are easy, code becomes simpler
bitbytebit.substack.com
bitbytebit.substack.com
Shoot, even when it's all first party tooling, I prefer a release-flags-and-binary as an atomic unit. If the flags and binary are going out as a single push, it simplifies a lot of things:
* reproducible state for given time with only one thing to track (running version), instead of two or more things (binary version, flag configs version)
* corollary: rollbacks are much more simple, because it's just one thing to rollback - the code+config+everything as an atomic unit, not "did we need to roll back the flags, or the binary, or both?" (Corollary to corollary: removes the awful problem of "oh shit the code and/or flags/configs wasn't backwards/forwards compatible" where you shoot yourself in the foot while doing a rollback
* removals and cleanups are easier: can remove flags, config bits, and code all at once, instead of having to do a careful dance of "set code behavior to default-on and remove dependency on flag, the wait for full deployment, then make sure everyone is okay with not rolling back after a given point, then remove flags and config logic"
* depending on your tooling, diffs vs prod during code review get cleaner
If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.
This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.
Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.
Forward migration: add date column, set = date(day, month, year) for existing records, drop is valid date constraint, drop day/month/year columns
Down migration: add day/month/year columns, add is valid date constraint, set day/month/year = appropriate datepart of date, drop date column
for trivial example.
It's technically a roll forward, while to most users it's more nature to say rollback/revert
Completely reverting all the state to some point in the past is easy and have been a solved problem for quite a while.
The actual problem is undoing states changes related to the code change, while not losing state changes related to normal system activity.
If I add some new fields to how my web forum records posts, and then find out that it's eating every tenth post and need to revert, it'd be good to not lose and posts made in the meantime.
You're right though. Systems in the social media space dealing with users appending to a database typically don't have this requirement to be easily reproducible, so the local snapshots are the right choice.
Having flags push separately adds another problem on top of data corruption.
I wouldn't necessarily call this approach simple, but it was incredibly safe and rollbacks was always a none event.
We got really good at data migrations and it was no big deal - but we only got serious about this after we had a major DB and functionality update that went wrong and took us down for 2 days.
The typical example is the renaming of a column, which needs to be done in several steps:
1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used
2. Deploy new code that works with both columns, reading from old and writing new and old
3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step
4. Deploy new code using only new column. Old column is deprecated
5. Delete old column
At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.
And these DB migrations, did your team keep a history of them? If so, did you manage them yourselves, or did you use some tools like flyway?
I'm asking because I'm starting a project where we will manage the persistence SQL layer without any ORM (always did it so far with Django's migrations), but might consider some third party tools for DB migrations.
renaming stuff with that gets a little bit tricky, but you can workaround that with database triggers if you really need to rename things.
You just end up with weird, unpredictable DB states that don't reflect production at all. Especially when you're dealing with old versions of MySQL and the character encoding and collation are all over the place.
There shouldn't be any reason that you can't do it with Flyway, but I would be concerned about fighting Flyway a bit. I use Django a fair bit and I honestly don't see a good way to make this approach work for Django, not suggesting that you can't, but you would be fighting Django a fair bit, it's not really how it's designed to work.
If you don't have en ORM, then this is actually much much easier to do right. I'd design the initial schema, either by hand or using some pgAdmin, TOAD or whatever you database has. For there on everything is just hand written migrations.
I'd argue that because you're making the migrations from the model, it's also easier to do accidentally create migrations that are not independent of the code version.
(You could in theory remove the column but not merge the migration if you wanted to show your code worked fully without that column in your ORM model before removing it from the DB as well?)
Here we never drop columns, only add new colunms, also never change column types.
When it comes to constraints (not null or something else) we double check about backwards compatibility.
Changing data is not a robust rollback.
* test of a rollback procedure,
* developers thinking about backwards compatibility and rollback procedure.
The main issue I see with this approach is that the test of the rollback is only partial. Just because the schema is usable by the previous version of the application does not mean the new data that is going to be put by the application will perform the same way.
Another issue I am seeing is that it is not a separate testing event but essentially happening live when the application is turned on on production. Not nice.
On the pros, this is very useful when you want to have more than one version of the application code to coexist at the same time. But I would not rely on learning about incompatibility when I start deploying the application, I would want to know that well before then.
It's a testing event if you make it a testing even. The team I'm referring to regularly makes a dump from production, run it through a PII anonymizer and performs migration test on that.
I guess it goes to the point of the author that hard deployments on mobile makes mobile development harder
Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the part of the process where most effort was saved.
Note that all this was used for a system that would be deployed only at one customer, and the feature would always be on. Yes, a boolean flag would have been a better solution for this.
My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated.
I could be way off base here, but I'd bet a lot of junior devs are just trying to stand out. They want respect, raises, promotions, and new job offers. Invisible good solutions don't bring those.
solutions are only invisible due to a lack of sufficient documentation and communication.
boring solutions are part of the job. they can be fun if the goal is shifted from implementing to automating.
none of this to say you are wrong, but a junior dev that succeeds at such peacocking efforts is signaling the poor health of their surrounding team as much as their eagerness to impress. where was the manager or senior dev to say “great! have you considered a boolean?”
Yup. And that by definition is a junior colleague ;)
That's genius. I'm going to borrow (read: steal) that.
"Isn't this just a Boolean flag? If X else y?"
I've saved a great many man months by doing just this. Sometimes work just vanishes. And it's not just the juniors that do it, sometimes the senior / tech lead types just miss a crucial point to make the solution trivial.
But juniors are often used to wanting to make work, rather than solve a problem.
1. Do nothing, bad state remains 2. Surgically restore to world before bad state 3. Remove all bad state (even if it wasn't caused by the incident)
I'm omitting some details, but I think this strategy can be effective.
How configurable should X be? We have one customer who wants it to be configured like this. All other customers want it configured like that.
We can make X maximally configurable which would allow future customers to have a custom setup - how likely is that?
If it's not likely, here's an alternative...
Even then, I find it increasingly hard to argue people out of the nonsense they pick up on blogs and conferences.
Especially for newer techniques that are not yet proven to be inefficient, such as using micro services in the wrong context, one has to resort to arguments by authority. Some junior and senior devs are not very susceptible to that.
For areas such as web development for user interfaces, or taking in enormous dependency trees with package managers, the problem is even worse. Here, an entire industry had standardised on suboptimal methods. One can argue that this is wrong, but there is no viable alternative.
In the latter case, guiding only one junior developer does not improve things. One has to educate the entire industry.
Thanks for this bit. As I gain more experience, I grow less and less patient with many blogs and conference talks. Often authors are just very excited about a topic, and my general impression is that they tend to propose ideas that are not seriously battle-tested in production.
Just also make sure from time to time that you’re on the same side. Don’t hesitate to laugh at your own mistakes openly. Sprinkle in some sympathetic, honest remarks. Target the thing, the mistakes, never the person.
Bottom line: laugh together at the expense of human fallibility and not a particular person.
You need safety guardrails for any deployment of code or config – a clearly defined unit of deployment that is version controlled, changes going through a repeatable automated flow in which lints/tests can be added, changes can be rolled out atomically all at once, or rolled out with careful control gradually, can be rolled back to a known safe version instantly when needed etc. All of this applies to config and code (binary).
Within code, having feature flags, dynamic config variables, experiments that can be ramped up slowly and ramped down automatically upon regression detection etc – these are mechanisms to make your life easier when you have unknown-unknowns manifest to bite you. And they always do in any non-trivial real-world large-scale distributed architecture application that needs to evolve continuously.
And these unknown-unknowns are not just code bugs, they are mix shifts in usage patterns, data etc causing unanticipated behaviors too. They are old stale stored state interacting with new code that you didn't test for. They are changes in code outside of your control – like remote services or OS/language libraries – that you assume to be mostly stable and don't exhaustively test every single time – that you don't even know when they changed - that change behavior without changing interface etc. These are real-world software systems problems.
If your software is built like an appliance – frozen in time and doesn't change ever after, is stateless, doesn't interact with outside world etc – then you don't need any of those guardrails and you can keep your code simple.
Example: just use a database table. Query and cache in memory for 60s. If you want, build a simple internal web page or tool to toggle flags. This works, and scales (from experience).
Do many apps really need A/B testing or segmented rollout? Maybe, but probably not.
So yes a more complex situation could benefit from all this extra tooling and complexity but otherwise it is dead weight/cost.
I would claim that release a new feature by enabling additional code path across multiple apps at once is a bit of an anti-pattern. It seems rather dangerous and error prone. In that case I'd actually release it in reverse order, so to speak. Release the apps that use the services of others first and have it check is this service is available/functional and if not, skip calling it. The release the feature to the next service down the stack. Then you can always rollback the last service and be confident that the callers still work.
It's way more work and I can see why for certain types of application isn't not really worth the trouble.
Most cases are the simplest cases.
No approach works for every single use case, but OP was specifically talking about the common case. You don’t have to shoot for the most complex use case every time.
.env files are the quick and dirty version for simple sites that don't have an admin panel or database.
Why would anyone replace a literal byte in memory with a full program?
Why?
So it's not just a byte in memory, but often also correlating the status of said byte with a users identity and then tracking and summarising user behaviour based on that relationship.
It's become fairly standardised and requires engineer time to setup and maintain the services behind all that, so it's valid to go third party for less than the cost of said engineer time, if all you want is standard.
Edit:it's also hard to always predict when a standard flags going to become part of a test, so just integrating for every flag and making that a standard process for your teams becomes the simplest approach.
This seems like an incident in the making. If each dev on a team commits unfinished code into prod behind flags, the whole project is going to be littered with flags and unfinished code. Some intern is going to delete a flag or an if check and then everything is going to break.
The idea of the "unfinished code behind a flag" is to be able to work on trunk instead of a long lived branch, increasing the pace of development and reducing integration costs.
This works quite well in my experience, and definitely better than "let's keep a huge branch in sync with the main one for 3 months while we finish".
The problem IME is the opposite: flags do not get removed fast enough, littering the code past their utility.
It's just code.
If the team is so bad that an intern can mess things up, they will, and the mess will have nothing to do with feature flags.
One concern about feature flags is testing, and the added permutations of testing needed to include all the feature flags in testing. You tested with flag A on and off, you tested with flag B on and off, but did you ever test with them both on and both off? Without feature flags, a big change that could have been represented by a feature flag would hopefully have to make its way past some quality gates. With feature flags, the exact permutation that you're going to cause later today by flipping on some feature flags may well not have been tested. Not that forgetting to test is something you can't protect yourself against with tools and processes, but testing all the permutations may be expensive.
You may not have to test all the permutations, if you can predict which permutations are relevant for your flipping feature flags later today. But a lot of organizations have poor discipline in cleaning up old feature flags, so it may not be so predictable. Maybe that's not a feature flag problem but an organizational problem, but the feature flags are gonna get blamed at some point, nonetheless.
We had a lot of tests for each feature flag variation, both unit and ui tests.
The codebase wasn't "littered" but there definitely existed unused code under flags, either to-be-used, or to-be-deleted. We had a grafana view of experiments that were old and not yet removed to manage them.
Overall we never had incidents around this, to my knowledge anyway.
This is how every major tech company works today.
This just seems like taking trunk-based development to an unnecessary extreme.
It's slightly more complex than a code push but still very easy to use.
When you have an `if False:` or equivalent in code, you have to deploy to enable a new feature, so if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature.
Another feature of feature flags is that a non-coder can toggle them.
If you need neither of these, sure, go ahead with commits and deployments instead.
This is a symptom of long release cycles. The more work you “save up” to deploy all at once, the more difficult deployments become. Everything becomes much easier when you minimise unreleased work.
This is not a feature, and it will get you called in during the weekend at some point.
Which means that for 20% you actually do need the feature flags. So you can't do away with the product, and as the author and many of the respondents here have mentioned, there is also a number of benefits to using feature flags.
In the trivial case of a feature for a single user still in development; yeah, a boolean is enough. But that is not the intended use case for feature flags, nor is it what the libraries promote you to do.
There are two interpretations of this, and I find one of them horrifying
1. I'm working on a feature, I need to update multiple components, so I make small changes for each component and feature flag those (this is fine! Totally normal)
2. I commit code that doesn't work because I'm unwilling to use or understand git
The second of these scares the crap out of me. Why? Doesn't it feel like a bit of a footgun to have broken code deployed?
However, I like most of the other comments of the essay - deploys should be easy! That makes rollbacks safe and easy. Using bool flags is fine too - imo feature flagging systems are best at de risking expensive deploys.
If you have an expensive deploy (thinking at least 20m to deploy prod) and you have staging which mirrors prod (probably also 15-20m to deploy staging), then any commit you want to deploy to fix or revert a change in prod is going to cost you a lot of time or hassle. Feature flags give you a mechanism to instantly rollback.
But it sounds like you already have that - it's called your normal deploy.
Most language compilers and JITs eliminate the unreachable code within a constant false branch sooner or later and the code isn't even technically "deployed" even though it exists in source control for that deployment.
My branches are never more than a day behind main (rebase does wonders!)
Code inside an if false block might as well be commented, and commented code is the fastest to bitrot - I would strongly recommend against!
It's not a dichotomy, it's an orthogonal spectrum. Of course it is hugely useful to be good at source control. Of course it is still sometimes useful to have code compiling in CI/CD that "isn't ready yet" but prone to cross-fire in other on going work and refactoring. (Plus source control is greatly useful for answering "Why is this flag false right now? Who was last working on it and for what project?" and keeping that code honest that it isn't just bit rot.)
To avoid this, you will need have a clear policy on what you're willing to support (eg versions up to 1 year old, or "major version - 1", etc.), and stick to it. Otherwise, people will expect you to support all of it, forever.
If you want people to keep up to date with your releases, you either have to provide an automatic update facility that they have to manually turn off (ugh, too hard), or you have to dangle desirable features as carrots in front of them, so that upgrading is exciting instead of a social obligation.
Include the currently running version in every logfile, if not every log statement. Make it clear to you where they are and run your own run-behind that upgrades only after they do. This will be painful, but it will save you overall time.
Dealing with the long tail of customers waiting to upgrade will be a persistent challenge though
You quickly end up with a NIH thing. You make a boolean toggle in the db. Then make an internal dashboard. Then some more advanced toggles. And suddenly you've spent a few weeks of coding time to essentially save yourself from spinning up an Unleash container.
But the OP's real issue is git, and not wanting to work on a branch. I actually think that's a valid goal! I've noticed that many teams have simply moved "one level up" with their git usage. The repo is the new directory; the push is the new commit; etc. You can practice and get fast with it, but it's still too many moving parts and plenty of things to go wrong. I actually think that feature flags are too coarse anyway to block off incomplete code; you should probably use conditionals for incomplete code, even if you also have robust feature flag support. The reason is that you never, ever want to dynamically turn incomplete code on, so coupling the flag to a deployment makes perfect sense.
http://github.com/tinspin/rupy
It also allows for session persistence, so you can test the dev/bug feature without logging in again.
And yes code becomes more manageable if you work like this because it encourages modularity.
« When developer are clever, codes become better »
Maybe it’s just me but having unfinished, dead, or scratch code in a production codebase really annoys me. Either finish your work or delete the unneeded code. More than a few times I’ve sunk time out of my day into investigating some code path only to realize it’s completely unused.
As for experimentation, I like percentage rollouts with segregated control/treatment group metrics. I agree that trivial on/off flags should be replaced by code deployments where possible (at my day job, code deployments happen to be a lot slower).
The exception is experimental work. Obviously experiments aren't being merged into master, so they don't need to be included in the arithmetic for any processes that eventually involve master.
Yeah, you might be really responsible and rebase and test your feature frequently, but what about other people? They are more likely to break the integration with your code since they can't even see it.
> Either finish your work or delete the unneeded code.
It's work in progress. We're working on getting it finished.
So finish it and then I will merge your PR ;) What's the use of putting it in master if it's not finished?
It's the author's responsibility to get it merged successfully. If they're taking too long and have to rebase and re-work their code to integrate, that's on them. Pushing it into master is either wasting a reader's time (per the original comment) or, worse, inviting an uninitiated collaborator to use it and cause an incident.
Since you're here, I'll ask you. What do you think 'integration' means in this context?
If CI passes on your branch but later fails due to lagging behind master, it is on you to get it working before re-requesting review.
PS: The HN guidelines clearly state "Be kind. Don't be snarky." (:
In practice, I have typically seen CI done with either feature freezes, or release branches to allow creating versions where everything is complete.
That’s the problem, which I already stated before the question. That’s not integration, that’s a build. Integration is testing your changes with all the code around it.
Specifically, it’s about testing your changes with changes that started after you began writing your code.
Only small changes land in anything like chronological order. On real codebases, even with low to moderate coupling, the rug can get pulled out from underneath of you without you even noticing until something bad starts happening. CI is about exposing your code to feedback at the earliest possible moment, so you 1) don’t continue to build on violated assumptions, 2) you do lots of small merge operations instead of one big one and 3) the thoughts that began the trouble are still fresh in your head. Consequences of decisions become abstract over time. Nobody changes their behavior based on 1 year old bugs found in their code.
Long lived branches are a crutch. It’s avoidant behavior. If you can’t figure out how to make your code work with everyone else’s, why do you think waiting longer will make things better? It doesn’t. The mess just gets bigger. I’ve seen it over and over again. The guys who won’t merge early are full of delusions about their own work that doesn’t match up with the bug and incident count. They call it bad luck.
Because of the thing I said in the sentence before the one you quoted.
> If they're taking too long and have to rebase and re-work their code to integrate, that's on them.
No, in the organisations I build, we work as a team and if someone is taking too long it's because we don't have the systems in place to help them work faster. There is very clear research on the benefits of working in small batches and integrating continuously. If that research has passed you by then I strongly suggest you go back and take a look.
Visibility, and avoiding duplicated effort. What's the use of keeping it out of master?
> Pushing it into master is either wasting a reader's time (per the original comment)
The new parts they'll have to read sooner or later, and for the parts that are changing it's better for them to read the new version than the old version.
> or, worse, inviting an uninitiated collaborator to use it and cause an incident.
Why would that cause an incident? Having someone see the "upcoming" code and realise they can reuse it in something they're working on is the ideal outcome.
The solution ended up being a rather simple feature flag. Within a few days, he coded a flag that would enable not-inlinging; and updated our test infastructure to look for regressions on unit tests with the flag enabled. Going forward, developers we responsible for making sure their changes didn't cause regression when the flag was enabled; and everyone was able to slowly chip away at everything the feature broke when they had spare cycles.
What would have been a major stop-the-world refactor with our most senior engineers, turned into a slow moving non-issue.
[0] Lack of turing completeness was and remains an explicit design goal, so recursion was explicitly forbidden.
If the code is toggled off for a ticket in progress, great. If it’s toggled off for an epic in progress, okay. If it’s toggled off and the epic is complete/abandoned, then it’s not dark code, it’s dead code. Fire up the chainsaws.
The old code behind a toggle should be deleted before the feature is Done. If it isn’t then someone screwed up.