Hacker News Reader

Comment by gampleman | Hacker News Reader

I think this is really important and under-examined viewpoint.

Anecdotally, at work I usually contribute to 2 codebases. The first, we've invested into optimizing CI heavily, CI is generally not flakey and almost always takes less than 10 mins to run. It's also engineered to generally fail early most of the time, so if it's going to break, you often find out in < 2mins.

The other codebase takes about 20mins, often takes >15mins to fail and is generally quite flakey.

Subjectively speaking, the code quality on the first codebase is much higher, since making those small changes is generally quite painless.

View on HN

The causality could be reversed: the high quality causes faster less flakey builds. There could be a hidden variable like developers on the better repo are better developers or the better repository solves a less complex problem.

However, I do agree with you. For me it's so much more pleasant working on the faster repo and it feels like it's because of the faster build. But the above criticism again applies to my feelings.

Flakiness isn’t always because of the code under test. CI systems can have any number of flakiness issues way before they ever get a chance to run your code. Spinning up VM’s/containers, installing build dependencies, SDK’s, huge amount of prerequisites to even running your code, clusters of test runners that can fall over, bad connectivity, etc etc etc.

Take a look at the kubernetes/kubernetes repo and look for kind/flake to get an idea of what a project looks like that has a tremendously complex integration testing, and how often the integration tests aren’t 100% reliable. I’d wager any organization that’s “all-up”/“end-to-end” testing software past a certain level of complexity (especially anything that touches cloud service integration) is going to suffer this problem. If you’re a developer on such a project, flakes are just sorta the way of things.

I'm going to just put it out there that developers who tolerate a flakey CI are, by definition, worse developers than those who don't. Possibly because lower competence, but more likely less experience of much damage a flakey CI causes to productivity and morale.

I’m in an organization with tens of thousands of people, and the CI/build team is very far from me in the org chart. The CI system they’ve devised is extremely flaky, not because of anything in my code, but because the CI infrastructure itself is extremely unreliable.

Am I by-definition a worse developer because I find myself in this scenario? If I were a better developer, what would I do? Change teams to the CI team so I can fix their shit? Just quit and join a different company?

Not everyone is in the position to fix the things that are causing them pain.

You would leave and join a company that has their shit together? Or you would convince your manager to provision a host just for your team and put a CI server on it? Or you would run it on your desktop?

Everyone is in a position to fix it because everyone can quit. Life is too short.

Reply on news.ycombinator.com