Hacker News Reader

I think this is really important and under-examined viewpoint.

Anecdotally, at work I usually contribute to 2 codebases. The first, we've invested into optimizing CI heavily, CI is generally not flakey and almost always takes less than 10 mins to run. It's also engineered to generally fail early most of the time, so if it's going to break, you often find out in < 2mins.

The other codebase takes about 20mins, often takes >15mins to fail and is generally quite flakey.

Subjectively speaking, the code quality on the first codebase is much higher, since making those small changes is generally quite painless.

The causality could be reversed: the high quality causes faster less flakey builds. There could be a hidden variable like developers on the better repo are better developers or the better repository solves a less complex problem.

However, I do agree with you. For me it's so much more pleasant working on the faster repo and it feels like it's because of the faster build. But the above criticism again applies to my feelings.

I'm going to just put it out there that developers who tolerate a flakey CI are, by definition, worse developers than those who don't. Possibly because lower competence, but more likely less experience of much damage a flakey CI causes to productivity and morale.

I’m in an organization with tens of thousands of people, and the CI/build team is very far from me in the org chart. The CI system they’ve devised is extremely flaky, not because of anything in my code, but because the CI infrastructure itself is extremely unreliable.

Am I by-definition a worse developer because I find myself in this scenario? If I were a better developer, what would I do? Change teams to the CI team so I can fix their shit? Just quit and join a different company?

Not everyone is in the position to fix the things that are causing them pain.

You would leave and join a company that has their shit together? Or you would convince your manager to provision a host just for your team and put a CI server on it? Or you would run it on your desktop?

Everyone is in a position to fix it because everyone can quit. Life is too short.

Flakiness isn’t always because of the code under test. CI systems can have any number of flakiness issues way before they ever get a chance to run your code. Spinning up VM’s/containers, installing build dependencies, SDK’s, huge amount of prerequisites to even running your code, clusters of test runners that can fall over, bad connectivity, etc etc etc.

Take a look at the kubernetes/kubernetes repo and look for kind/flake to get an idea of what a project looks like that has a tremendously complex integration testing, and how often the integration tests aren’t 100% reliable. I’d wager any organization that’s “all-up”/“end-to-end” testing software past a certain level of complexity (especially anything that touches cloud service integration) is going to suffer this problem. If you’re a developer on such a project, flakes are just sorta the way of things.

> If your build and test suite takes hours... you will often decide it's not worth it and move on.

I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right?

> tests will take hours to run again. Do you bother...?

I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time.

Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.

> Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right?

There are things like ci.skip option in git for this.

Right, but that leads to a culture of skipping CI for things, which can easily turn into skipping CI for things that are a bit less obviously-fine, which I’ve seen time and time again. Once force-merging over incomplete CI becomes a thing, people start using it more often and suddenly you get actual bugs slipping in.