A development process to ship features fast
growing-products.paralect.com
growing-products.paralect.com
I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time.
I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't.
Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.
Use feature flags to decouple Deployment from Release and you’re fine.
But management sees the capability and says add a flag for customer Y that changes how this page works.
And after it’s enabled for everybody, remove the flag.
This is great until it's not. We're starting to hit this tipping point on our code-base.
The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.
Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.
I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed.
generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time
The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time.
That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter.
The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility.
I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.
E.g. we went with that route with GH Actions, until our tests didn't fit into the memory of the provided machine sizes anymore (shouldn't be a problem anymore as GH provides options for bigger runners now). So we had to spend 1-2 days on setting up a self-hosted runner in a way that it wouldn't be too heavy in maintenance, and then probably another 1-2 days of rework over the next few weeks to iron out the kinks.
The occasional freezes mean future tests written do not need to be optimised (there will be a freeze for important releases anyway), so the unoptimised-tests problem will grow worse the longer you use this strategy, and the cost to rectify it will grow, until your release frequency collapses on itself and everyone complains that "there's no solution to this problem".
These are the sorts of things I think it's very important to nip in the bud. Spend the two weeks now, when it only takes two weeks! Much better than putting everything on hold for six months to fix it four years from now when it becomes untenable.
That does seem like a possibly worthwhile choice though? A "stop chopping the tree to sharpen the axe" moment?
But, maybe spend a day on it. I've been to places where the build took hours and there were obvious 30min solutions to fix it (which I did, in some cases).
Consider it an investment. 2 weeks now gives you how many more features over the next six months due to increased dev velocity?
Bad tests can absolutely make you slower. Good tests can be existential.
The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
Test_OnePlusOneEqualsTwo()
is not as useful as
Test_CanAddTwoNumbersTogether()
Test for behaviour, not implementation.
At an early stage startup with a small app it might not be as big a problem, but where I'm at right now the app is over 10 years old and our test suite with many many parallel workers is still well over an hour cycle time to get that sanity feedback on a commit, and it's painful.
We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
I see that I had assumed we were talking about web apps. This assumption narrowed my perspective, and was an error in thinking on my part.
(I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
I would not judge an individual developer's productivity on commits per day, agree that would be foolish.
1. Pre Series A companies, which didn't have strong engineering team from start and now struggle to ship. 2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget.
1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that. 2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.
This is the important point.
Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.
> But, on another side tests slows you down as you build features. When your team writes tests they need 2x more time to release any feature.
Tests are an investment in fast future development. Last I did a startup, we did not write tests for anything throwaway or experimental. But once we decided some chunk of code was a keeper (e.g., experiment had a positive result), we wrote tests.
With just a team of 6, we were releasing a few times a day, not just once a week. And we had no "QA freeze" nonsense, because our tests were solid enough that we didn't need manual QA to release.
That said, I hired for people who were good at writing tests, and we did pair programming with frequent pair rotation, which gave us a big quality boost and helped keep our test suite lean and effective. I can believe that some startup teams have low enough levels of testing skill that "they need 2x more time to release any feature".
I mean, they are, but what's the roi on that investment? Not all investments are good.
90% of start-ups fail, the chunk of code that's a keeper still has a 90% chance of being thrown away.
Before you have signs of product-market fit, I think people should write throwaway code and then throw it away. Once you have something worth investing in, I think it's time to invest. Having high-quality code in areas where you're confident it's valuable enables rapid experimentation in the areas where you're not sure yet.
At my current job (fast-growing mid-stage startup), any developer can push when they want, but it works a bit differently. Once the release is staged, everyone who has a commit on staging is responsible for verifying their feature. Then the developer goes through a "release checklist," which involves finding a buddy and jointly running through a list of manual tests to validate key features. After that, they can push. They probably push a few times a week.
I feel like code freeze/manual QA/weekly push is something of an anti-pattern. Developers are disconnected from the process of releasing their code, and take less ownership. While in practice manual QA should lead to finding more bugs, in practice it can lead to more defects.
All that stuff can come after you have proven your software makes money / users want to use it.
Just get the damn thing written in the quickest, most crappy way necessary for speed.
The code doesn't need to be organised well. It doesn't need to be documented.
The only thing that matters is to do your best to avoid security holes.
No tests. Stop testing, apart from manual runthroughs that the CEO can do in between trying to raise funding.
My eyes roll when I hear of startups creating "good processes" for development at any level, let alone CI/CD and tests.
Build the MVP so that it's good enough to start solving whatever it is you're solving and get your first customers.
If you can't get customers, what's the point of having a sophisticated codebase with tests, etc?
And in fact every minute spent on writing and running tests (and there's always ALOT) and CI/CD is wasted, wastes money and time, leading to longer development time until the software can be put in front of users.
Therefore CI/CD and tests and all their associated processes are a red flag and increases the likelihood a startup will fail to bring its software to market.
Developers who have always worked in well organised environments have really deep rooted trouble getting their head around this way of thinking. If you say to an experienced developer "no tests! no CI/CD!", it's likely they will immediately decide you are an unprofessional hack.
E.g.: it doesn’t matter what fancy SIEM tool you’re using if you never patch anything and the root password is “password”.
“But the vendor assured us that it would solve our security issues!” is the inevitable reply.
The sum total of time spent on CI/CD and tests is not the time it takes to get it started, it's the time spent executing the tests/CI/CD into the future, thinking about the tests/CI/CD, problem solving the tests/CI/CD.
All these things require ongoing time which takes time away from developing the application.
At my last startup, I wrote totally throwaway code on all our initial prototypes. Once we had something that looked like it would hit product-market fit, I hired a team, we threw away the shitty codebase and started fresh.
If you don't throw it all away after making the kind of mess you describe, then you are going into the company's most crucial time with a boat anchor around your ankle and the water level rising. I'm very glad we did, as starting fresh with tests, CI/CD, and other quality practices let us be extremely responsive to my cofounder's needs and the things we learned from every release.
This is why an experienced developer is sometimes the wrong person for the job to build a startups first version, unless they are able to fully embrace the mindset of prioritising speed over everything.
Beware of experienced developers who get the job of writing a startups first code and who start talking about tests/CI/CD/feature branches and all that stuff.
Avoid security holes and take backups, that's all that matters and anything else should be actively avoided.
As an example, I did a startup circa 2010. We raised money on an idea that VCs were very excited about. So much so that we found out later that other VCs had invested in similar notions. We spent a few months building disposable prototypes to test the value prop. It turned out that users hated the idea no matter what we tried. So we killed it after spending maybe $100k and did something else. Job done! But the other companies persevered, building full apps, and each burned millions before they realized there was no future in it.
So in that case, yes, you don't want to invest a lot. But a few ideas later we had something that looked to have real product market fit. At that point, we threw away the code and started fresh with solid tests, CD, and other quality investments. Why? Because the job we were trying to do had changed. It had gone from hypothesis testing to actually serving users and building a real product. If you want to get that job done well, you can't afford to have a garbage code base.
I like these companies. Share the names and I'll have my sales guys reach out.
...if only I had a cent for every time I've had this conversation with an engineering manager: Manager: "Just live with the mess for now. We'll fix it later when we have money." Later: "Can we fix the mess now?" Manager: "No. We're making too much money now. Blah, blah, blah. Opportunity cost."
That is ludicrously bad advice. Setting up a straightforward CI/CD pipeline can take a couple hours and it instantly provides huge productivity benefits.
I worked for a startup that didn't have an automated deploy pipeline and didn't have any tests. It was a complete shit show from day 1, and I will never work for that chaotic of an environment again.
I am certainly not for over-engineering early in a company's lifecycle, but there are simple, "table stakes" things like use of SCM, automated builds and CI/CD pipeline. If I don't see those I assume your engineering team doesn't know how to build software fast.
Kill CI/CD with fire for the sake of speed.
"Craptacular development mindset" is needed.
Sure, same as everything else that is overengineered. A simple, no-frills GitHub Action (or equivalent) should not take longer than a few hours to implement. If you need more time than that, it means that you have an overly complicated deployment process, which is a time sink when done manually anyway.
Your software should meet the minimums standards required by your target user.
That's the "viable" part of Minimum Viable Product.
In a similar vein, you cannot release a new antivirus product that does not detect viruses, or security product that does not make things more secure. Software development has (or should have) defined priorities - and if accurate financial transaction processing is a base requirement then that's the minimum viable product you need to deliver. And if it takes tests to do that, then that's what you need.
Being a startup doesn't mean your tech has to be garbage. So many miserable devs hoping that "after series A" they'll get to refactor that spaghetti code mess (they won't) and seeing their velocity slow to a crawl because they can't make a change without worrying the whole thing is gonna break.
2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
3. You might be right in some circumstances. But your assumption elsewhere in comments that CI causes over-engineering is incorrect: any team that has an over-engineering problem will generally be in trouble, which is not caused by CI.
4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
[1] I’m using this definition for startup http://www.paulgraham.com/growth.html — personally I dislike PG’s definition of a startup (paraphrased: designed for fast growth). Should a business that is trying-but-failing-to-grow not be a startup? A business designed for slow growth but large market is not a startup? PG gives Google as an example of a startup, yet that contradicts his own essay: didn’t Google have a lot of luck, and it isn’t obvious to me that Google was designed for growth, more that the opportunity knocked and Google jumped on that opportunity, successfully. Some of PG’s definition feels tautological; 20/20 hindsight.
Edited: made politer.
Agreed. I have made an error there.
>> 4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
Agreed. I have made an error there. I should choose a better tone. I do work hard to get the tone right but not successful today obviously. Apologies if this has offended anyone. I usually delete things that I feel are too negative, adversarial, preachy or otherwise not the right tone.
>> 2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
The comments I make are about a company pre product/market fit, doing the first build of its software before opening the doors to customers. I do agree that the startup period continues beyond this perhaps for a number of years, at which time development priorities might change.
>> [1] I’m using this definition for startup http://www.paulgraham.com/growth.html
For reference, in the link PG says "A startup is a company designed to grow fast.". This seems overly narrow because it seems self evident that there is lots of startups that have different objectives to fast growth.
My definition for a startup is more like Steve Blank's: “A startup is a temporary organization designed to search for a repeatable and scalable business model.”
I think you (a) have been burned (and I'm not denying testing can be a waste of time) and (b) not being a dev yourself you don't fully understand that this isn't a black and white issue or that you are actually paying a massive velocity cost by doing zero testing. Your POV is akin to advising people to not drink water on a hot, strenuous hike because "every time you lift your arm to drink, you are wasting energy that could go to your legs". Uh yeah, in both cases it's a little more complicated than that. Definitely don't waste an hour making a gourmet meal (and yes I get sometimes devs do this), but there are a spectrum of options and there is a sweet spot. How about "drink some gatorade and eat a power bar without slowing down"? Getting that balance right isn't trivial but it is possible and the payoff is huge. Which goes to the other point you made about not hiring expert devs, man, you've been hiring the wrong experts if you think fresh, unsupervised bootcampers are a better choice at any stage other than "can you make me a framer prototype over the weekend?".
Why do you think I'm not a developer?
Here is a list of OSS developer security tools: https://github.com/boxyhq/awesome-oss-devsec
Some reasonable advice in here otherwise.
You’re still waiting to release features that are done and turning on a whole week’s worth of features at one time, but it’s even worse because you spent a bunch of extra time building it to have feature flags.
The modern "feature flag" is more of a knob that can be slowly turned up, along with ideally a way to send shadow traffic through the feature even before any % of real traffic is allowed through.
In that proper implementation of feature-flags, the overall benefits (easy rollback, graduated rollout, selective rollout based on user characteristics, better ability to manage scale) outweigh the rather-trivial engineering cost of wrapping a few if-clauses around the feature.
> rather-trivial engineering cost
I personally disagree with this. To implement feature flags your engineers have to reason about all the places where the feature needs to be on/off, implement those flags, write and maintain test cases for both the on and off case, rollout the code, then go back and refactor everything to remove the flags afterwards.
It’s not insurmountable. But it’s not trivial in my experience.
I respectfully disagree, because I'd say that a "release" ought to be defined from the viewpoint of a given individual user. From that person's perspective, they are indeed seeing things change only weekly, and the details of how that's done (e.g., feature flags vs giant deployments) are mostly irrelevant to them.
> It’s not insurmountable. But it’s not trivial in my experience.
You are right. Honestly, I'm so used to having to sell the idea that I just reflexively wrote it that way.
Thank you for listing out the different pieces. You're absolutely right there is a lot of extra "essential complexity" that gets added in. Even if the "actual" coding work is "just a few" extra if-clauses and tests (which is how I'd sell it ;), everyone has to hold a lot more in their heads.
Personally, though, I would still insist that it's way better than the essential complexity that gets added everywhere by coupling "releases" with "deployments." In my experience, once an org lets product-release-tempo dictate engineering-velocity-tempo, the bureaucrats essentially have a chokehold on all engineering decisions.
The organization slowly grinds to a halt; high-value customers start getting picked off by nimble, faster-moving startups; and the circle of life continues.
Feature flags are IMHO a key technique for a startup-that's-getting-bigger to hold off this particular mechanism of decline. Non-technical stakeholders at all levels can be easily waved off with a simple remark of, "Well, we'll feature-flag it." And engineers can stay much more empowered, even as the company grows much bigger.
The argument in favour of this advice only checks out when a feature branch is kept out of sync with the main branch. In my experience, the process that works the best is to rebase the feature branch on top of the main every time there is a divergence, resolving all the conflicts right then and there.
> Use a monorepo
Definitely - if you have a monolithic implementation that is deployed as a single unit. Otherwise, monorepos start getting in your way pretty soon.
> Use feature flags
There are pros and cons for this. In my experience, feature flags work best in the front-end, where you want to hide features from certain users at certain times.
> Deploy weekly
The deployment strategy depends primarily of the type of the product you're delivering. For a standard, vanilla SaaS, the continuous deployment is the most efficient.
If every branch has multiple “merge master into …” commits, and they are also merged back into master (without rebasing or squashing) you end up with a huge mess. Reverting anything becomes impossible, and mistakes when merging almost irrecoverable.
Why would either of those be harder?
What you mean to say is you don't like the way the history looks. That's fine, it's a valid criticism. Calling it a necessity is over the line.
Hard release cadence is amateur hour (outside of major software pieces like operating systems and languages).
One release method that I had to install by pulling rank (after non-stop discussions) was the "release when ready" approach.
Imagine your scrum board. There is the QA column - all the stories that are currently being QA'ed.
Now imagine that during a standup, the QA column is empty - everything has been tested and there are no unstable features in QA. This is when you cut a release.
It can be that you release every day, or you may not release for three weeks.
It worked without fail, and was insanely flexible, resistant to change in variables.
Apart from "team size", every single item on that list is part of the definition of "type of the product you're delivering".
These are at odds with each other. In my previous role at a startup we deployed on average twice per day per developer. The automation necessary to make this hands off and reliable wasn’t actually that much. It rarely caused issues.
> merging the feature branch with the main branch takes a lot of time
Why would that be the case? `git merge` is not a slow and expensive process. Is the problem the per-branch code review process that teams often engage in? They didn't even discuss how code is reviewed (feature demos are not code reviews).
> when the feature is done, merging branches back might lead to new bugs and instability
Feature branches should regularly be pulling in changes from the main branch. This seems like it would be a problem whenever your local main gets behind the upstream main, either way. Lots of teams will squash changes before merging feature branches, and run the test suite on it before it's merged.
> it is more difficult to do incremental refactoring
Smaller PRs and feature branches.
There have been a few posts lately espousing the removal of workflows that have evolved over time to solve real problems in development practices. To me, it seems like a lot of cowboy coders not wanting to be constrained by sustainable development methodologies.
This is why you should be merging main back into your feature branch daily, so imo this is just a strawman.
That's assuming it's not reasonable to have smaller features. Sometimes it's not. You should strive for feature branches that live for a day or two, but if you MUST have it longer, just merge back in.
This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure.
It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.
At that point most products are basic enough for developers to test the product themselves.
And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down.
Maybe it is better to only begin automated tests when you reach product-market-fit.
Tests force you to write better code. Code that is easier to change. Code that is more robust.
You don't need 100% coverage or perfect tests to gain this benefit.
A good test suite enables you to make large refractors safely and quickly.
The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests.
Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started.
Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively.
Tests speed up development velocity and improve your codebase.
Just don't get stupid about it and you won't negatively impact startup performance.
I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve.
Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in
2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
People who don't test don't test because they don't know how. They are hacks.
Also hopefully interesting, the opposite of “don” is “doff”.
Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks?
If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it?
There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
* BE tests are easy to write, catch lots of important bugs, and generally speed up development.
* FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct.
* E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead.
----
Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.
I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest.
Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.
But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.
In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead.
>the codebase and/or the database schema changes dramatically it will often break the tests
That shouldn't break integration tests [much].
It also requires considerable skill to build and maintain, and not every team has that...
...in which case you might have a less good and useful test suite.
I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.
Something that's missing from this list that I find extremely useful:
An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount.
That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.
the latter is the only hope, but practically no one is willing to commit to it
now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.
It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding.
By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle.
That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.
The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool.
Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.
On the FE, we noticed:
* Most of the bugs were in UX/UI that are very hard to capture in a test suite.
* CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team.
* It took forever to write FE tests.
* It took even longer to run FE tests.
* Even well-written FE tests miss a lot of subtle issues.
* We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.
What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error.
It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended.
Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.
We've found that Typescript and a bit of boilerplate has almost entirely eliminated these types of concerns for us.
In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.
Both Google and Facebook use monorepos
If you're not successful, then you won't have any scaling problems.
Stacking is a wonderful workflow to accomplish this: https://graphite.dev/blog/post/N8SOs49y4bYdV1Vjf5qE
Also what happens if during the review of PR #1, the reviewer suggests something profound that would fundamentally change the direction of future commits?
I think this is mistaking cause and effect. The problem is having long lived branches, not the fact you have a branch.
IMO the best advice is to avoid long-running branches. This is easy to achieve by slicing features into smaller units, and using feature flags, both recommendations already featured in the list.
The clearest helper "rule" for a team I came up with, was "at the end of the day, there is no branch in a repository except main". Takes time to get there, but simplifies a lot.
We also found feature flags slowed us down a lot in the early stages. I feel these should only be done for some features (e.g. an explicit AB test, a large feature with a lot of technical risk). Feature flags create a ton of complexity, and are often not worth it.
That essay was written based on 3 three pre Series A companies, which didn't have right processes in place and was struggling to ship fast enough. Weekly deploys is a first step. It's difficult to impossible to deploy more often if you don't have good tests coverage.
I think the pandemic broke us.
Sigh, such a misguided quota.
Monorepos don't make sense at face value if you want your code to be modular. How can bringing together multiple modules into a single monolith possibly help to improve modularity.
Plato said it best:
"You see, it’s not, I think, the function of heat to cool things down, but the opposite."
"Nor of dryness to make things wet, but the opposite."
Not to say 'spaguetti' code is not an issue, but these are a little bit orthogonal to each other. It's very easy, for example, to grow complexity two/three-fold by trying to make a component more generic and reusable before the actual need arises. And when it does, the original interface you predicted is rarely a good fit.
At some point the business needs will change. At that point, someone will wish that the Apache server didn't talk directly to the database about last year's invoice minutiae.
I actually do think that code maintainability probably doesn't matter much in the context of many big corporations since they often have market monopolies and can afford to hire huge numbers of developers; they can afford to waste a lot of developer time... That's fair enough, but I don't see why they should be lecturing the rest of us about what good software development practices are.
That would be like a rich person lecturing a group of starving people about money not being important. This lesson is completely irrelevant to that audience. Anyone who listens to them will starve to death.
The things the article talks about are not bad advice but I do not think any of them are critical for being fast besides getting and incoperating feedback as early as possible. This is of course not limited to feedback on an implementation, you of course also want early feedback on your design before you even start implementing something.
Also tbh feature branch is not really bad? How long does it take for one to create branch and open PR for reviews? It was a nightmare for everyone to merge in 1 branch then merge back to master when we want to lol.
FF 100% helps, but we shouldn’t compromise tests and PRs I believe (it also helps when we need to rollback stuff obviously)
This is a big one. It seems the industry (at least to my awareness) has almost totally rejected Gitflow for trunk based development in the last few years. It’s a beautiful idea in theory, but a complete headache in reality.
> Use feature flags
Be careful here. Adding feaure flags means you now effectively have N^N builds (where N is the number of flags) in production, each with their own possible quirks and integration issues. Keep them to an absolute minimum, and remove them as soon as possible.
2. Feature flagging is pretty key to make the described process possible, and about halfway down it turns out of course this post is really an ad for a feature flagging SaaS product. Nothing wrong with that necessarily, but worth knowing that the motive behind this is to sell you a product.
I write apps (which may include some fairly significant backend stuff).
Some of these suggestions apply, but not all.
I ship weirdly fast, anyway,
Ideally the client won't let them occur. But you should still verify the server doesn't give up the goods.
I guess you could "manually" test by using the command line? Sounds like a lot of work. In my experience, more work than automating it.
I think better tooling around git will help a lot with speeding up development teams.
saves 48 hours on releases
> Avoid Feature Branches
> Break features into small, incremental changes
This makes efficient & organised writing of code easier but makes product development harder or at least very artificially limited. If you want to ship fast this will help. If you want to ship quality & value, this will likely hamper you. This needs to be compromise - feature branches can be tricky to manage well, but it's better to learn how to than avoid it. It will allow you to manage complexity and meet your target audience's needs more effectively.
> Use a monorepo
Absolutely arbitrary and has zero bearing on speed of delivery. Monorepos vs monoliths vs multi-repo microarch is as efficient / in-efficient as your choice of app architecture, team structure, tooling, software ecosystem, etc. It depends on far too many variables to be a blanket positive.
> Automate CI/CD pipeline
> will make it easy to release changes multiple times a day
> Deploy weekly
There's a lot of contradictions here. Deploy/release much more often than weekly. Please.
> Freeze the code before release [...] This is not to improve the speed
OK.
> Write tests (or not?)
Make up your mind.
> Request at least one demo per week
This is generally reasonable advice, but the cadence seems too frequent. Demos are expensive w.r.t. developer's time & energy - they should be frequent but still leave some time for actual dev in between.
> Use feature flags
This is nice if it's simple (if you're following the "Break features into small, incremental changes" advice I guess you have no large/complex features).
If you do have any large features, feature flags can sometimes come with side-effects, impacting performance / stability / security of your overall application.
Follow this with care.
> Define owners
This sounds like good advice until you read the author's description:
> avoid the shared responsibility trap, when no one is responsible
This sounds like a bad manager or dysfunctional company - shared responsibility is a good thing actually. If your engineers aren't motivated to own code, that's a different problem.
> The feature owner knows explicitly that no one else is going to complete their work.
This sounds like a very dysfunctional team. If you have this problem, work on your culture, don't try and fix it with technical solutions.
Overall, code ownership is good as long as it's shared. Bus factor is a thing.
---
This all seems like the common mistake of mis-attributing success to a large but irrelevant change that coincides with many small beneficial incremental learnings.
Imagined example of the above: say the author moved from a disorganised mess of many disparate repos to a fresh-start carefully-planned monorepo: things didn't improve because it's a monorepo, things improved because someone sat down and planned out the organisation of code to fit their needs (needs which probably evolved to that point of maturity over many small repo-structure changes before). Correct connclusion here isn't "monorepos are good", it's "sitting down and re-thinking/re-organising/planning code structure periodically is good".
If branching/merging is giving you this much trouble, you’re doing something very, very wrong.
Pause, actually learn git, then resume.
Also, testing is so easy, if you’re doubling your dev time writing tests, that’s another thing you need to dedicate time to for learning, because it should add maybe 10% more if you’re competent.
I actually tried to simply have my team commit to the main branch, and it worked out quite well - we never had problems with code divergence. I guess it doesn't work so well for larger teams not working on MVPs, though (we were 5 developers).
The thing is, I want my developers to code, not talk all the time about what/where they would like to code and that the others have to wait until they're finished or suck up the merge conflicts if there's no time. Some communication is always necessary - but this solution with feature branches requires constant communication and still, the developers would rather wait that try to synchronize this way.
The more you comment, the more it’s clear you work with a bunch of coders who want to “go dark”, and there are a ton of problems with that.
Don’t go dark.
Well it wasn't trivial at that stage of our development. It became much more trivial once we and the market were happy with our product-market fit and at that point we indeed started with feature branches again.
Now I'm working at a consultancy and since there's much more time for everything, we are doing feature branches during our MVP development too. But there's a stark difference in velocity this current team and the team I told you about achieved.
This just isn’t a real problem. Try working more incrementally, or try working with colleagues who don’t rewrite large swaths of code without coordinating with their team.
You have a people problem, not a tool problem.
Your team is suffering badly from a lack of even a baseline understanding of how to develop software as a group, and learning even the most basic usage of git would substantially alleviate that issue.
When your business changes wildly every 2-3 months for a year (and sometimes few times a month too), it's hard to not change the code base a lot. Our place was not to change this situation - our place was to learn to deal with it, which we did satisfyingly.
Also I guarantee I’ve used git branches in more dynamic situations than what you’ve described here and it caused absolutely zero issues.
I'm glad that your preferred branching strategy works out well for you.
Seems like by your definition there's no way to not do feature branches. A little weird take, IMHO - I know it as a very specific strategy that you have to consciously perform e.g. by naming your branches correctly in sync with your issue tracking system, by not merging until you're done with that ticket, etc - not something that happens automatically just by cloning.
https://sillevl.gitbooks.io/git/content/collaboration/workfl...
There is nothing whatsoever that requires a feature branch be long lived or named anything in particular. This is what I mean when I say you don't understand the technology you're using.
What you have linked is a "flow" that uses branching, and isn't relevant to this discussion.
So yeah sure, we were doing 2 hours-long 1%-of-a-feature branches.
https://www.atlassian.com/git/tutorials/comparing-workflows/...
https://www.decodingdevops.com/how-to-create-feature-branch-...
https://docs.gitlab.com/ee/topics/git/feature_branching.html
Could you please post at least one article which describes what you're talking about and what the differences from "just commit/push to main" are?
BTW this whole thing isn't about keeping a branch up to date with the main branch - sure that's easy. This is about integrating changes in a small code base where collisions happen often due to small but rapidly changing featureset on which 5 hyperactive developers work on. The only solution to that is to integrate the code very often on a common branch - in our case, around every 1-2 hours, regardless of features being done or not.
> Some people refer to Git’s branching model as its “killer feature,” and it certainly sets Git apart in the VCS community. Why is it so special? The way Git branches is incredibly lightweight, making branching operations nearly instantaneous, and switching back and forth between branches generally just as fast. Unlike many other VCSs, Git encourages workflows that branch and merge often, even multiple times in a day. Understanding and mastering this feature gives you a powerful and unique tool and can entirely change the way that you develop.
It's really not hard, a "local copy" is a branch.
If you can't manage what you just described trivially, you need to learn git better, because I, and everyone else who knows how to use git, can manage those things trivially, and have successfully for many years now.
Your solution was the wrong one, homegrown to avoid dealing with the reality that you and your team didn't know how git works.
All of this would be fine, by the way, had you not proposed others listen to you. Exactly zero people should do as you have described.
[0] https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-N...
> It's really not hard, a "local copy" is a branch.
For sure. Not necessarily a "feature branch" though. That's a more specific thing described in the articles I linked, building on top of the basic Git feature you linked.
Please understand that "feature branch" is a technical term describing a very specific branching strategy - and you're not going to find its definition in the Git docs; I think it was originally defined by Atlassian (but not quite sure). A local copy of the main branch with a few additional commits that you're going to push to the origin way before the feature is complete is not a "feature branch" as commonly defined.
You originally replied talking about how hard branches are to merge, but then revealed you also use branches to develop features, thus revealing a fundamental lack of understanding about how git works.
In our case we stopped using the `git branch` and `git checkout` commands whereas previously it was an integral part of our workflow, hence "not branching out" - again, sorry for not saying explicitly in that one comment that of course we worked with the local copy of the main branch which is a branch too, since that's how Git works I thought that's a given, this is a professional forum and I didn't think I need to call out everything so explicitly as if I was talking to a junior dev.
I never said that branches are hard to use nor that we didn't use them - it's only your strawman where you (since I explained what I meant so many times, it must be) intentionally misinterpret what I said about feature branches. You even sent a link to Git docs when I asked you about workflow strategy and in reply to articles talking about workflow... :-D
Feature branches are a very common workflow at many companies big and small, one could reasonably say it's the standard, practically every single team where I worked has done it this way and any change to this workflow was much discussed and temporary - and I've seen that happen only 2 times ever in my 12 years of SWE consulting career. I'm really surprised you are acting as if this was the first time you heard the term.
And - we got it already, you're a Git ninja. You can stop trying to show us your superiority now. The discussion would be much more pleasant - and I'm not saying this because you know more about Git than me, I'd be happy to learn. ;-)
Agree on the testing, though, unless you're doing horrible generated cruds it is usually faster to write tests than not.
It honestly sounds like you don’t know what a branch is, since you’re talking about libraries and multiple supported branches, so I very much doubt your “git-fu”.
It sounds like you’re in the “merge conflicts are hard” camp, which is a camp of ignorance.
Also you’re pushing code into this main branch from… where? Another main branch.
That’s literally how git works, you’re just removing any oversight and collaboration opportunities by insisting on calling this branch you’ve created locally “main” as well.
Code reviews are awesome, and not doing them is a fantastic way to ruin your team’s ability to support work they didn’t themselves write.
The difference lies in size of the diff and the manner of introducing changes. Other developers have your changes instantly. This is core of the discipline called „continuous integration”.
I’ve been in multiple teams doing mandatory, pre-merge, double code reviews and getting crappy results. The longer someone kept their precious feature branch isolated, the worse.
And keeping branches „up to date” only solves a small part of the problem with long lived branches.
But even if it did, regularly remerging main back into it (or rebasing) does solve like 90% of the issues being described here, and further comes with a whole host of other benefits.