Comment by solatic | Hacker News Reader

Comment by solatic | Hacker News Reader

Please, step me through this. In a start-up:

  a) Developer builds locally
  b) Developer tests locally
  c) Developer pushes to a repository
  d) CI starts
  e) CI ends
  f) Wait for human code review and approval
  g) Merge and deploy
  h) Observe that nothing broke / no need to revert.

Is it even possible for (e) - (d) to be short enough, let alone (f) - (d), to keep the developer's attention instead of context switching? Most devs I know just context switch immediately after (c). If you care about developer productivity, you're much more likely to get results from focusing on (a) and (b), by hooking development laptops into caching and restricting system/test scope, than you are by reducing CI times, unless your CI takes some ungodly amount of time to run (several hours).

For point of comparison: article examines how using monster machines can reduce the build time of Fedora to 27 minutes (not really a comparable example to most companies, but OK). My devs complain about an (admittedly unoptimized) CI time of 20 minutes (on a much simpler project than Fedora) that introduces context switching. Is the article really trying to get me to believe that Fedora developers wouldn't context switch on a 27 minute build, twiddling their thumbs for 27 minutes, but that a 35 or 55 minute build would? Something about getting under the 30 minute bar gets developers to keep their focus? I call bullshit.

To me, it's not about the context switching, but the gradual "learned helplessness" that comes from CI tests taking so long that it's no longer worth making small changes or fixes.

Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right?

If your build and test suite takes hours, and the infrastructure is flaky (sometimes there are not enough job runners, sometimes the test setup fails, etc), you will often decide it's not worth it and move on. Or you'll decide to roll that change into the next PR, at which point PR's start to have a lot of unrelated work because "well, I gotta run CI for <new feature> anyway, may as well fix some linting errors while I'm at it", which is often not great practice.

Or, somebody leaves a comment in your PR that is a nitpick about something which isn't that important, but you agree you should probably change it, but tests finally finished/passed. If you make a change to address the nit, tests will take hours to run again. Do you bother, or do you "leave it for a followup"?

Or worse, do you force-merge the amended PR because there's no way that little nitpick will break anything? Leading to a culture of merging over incomplete tests?

Tests taking a long time can gradually degrade the quality of a project in this manner. Little things never get done, or the culture starts being around skipping tests for changes that "shouldn't break anything", etc. I've seen it happen in $dayjob and it really kills the culture of quality for a project.

> If your build and test suite takes hours... you will often decide it's not worth it and move on.

I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right?

> tests will take hours to run again. Do you bother...?

I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time.

Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.

I think this is really important and under-examined viewpoint.

Anecdotally, at work I usually contribute to 2 codebases. The first, we've invested into optimizing CI heavily, CI is generally not flakey and almost always takes less than 10 mins to run. It's also engineered to generally fail early most of the time, so if it's going to break, you often find out in < 2mins.

The other codebase takes about 20mins, often takes >15mins to fail and is generally quite flakey.

Subjectively speaking, the code quality on the first codebase is much higher, since making those small changes is generally quite painless.

The causality could be reversed: the high quality causes faster less flakey builds. There could be a hidden variable like developers on the better repo are better developers or the better repository solves a less complex problem.

However, I do agree with you. For me it's so much more pleasant working on the faster repo and it feels like it's because of the faster build. But the above criticism again applies to my feelings.

I'm going to just put it out there that developers who tolerate a flakey CI are, by definition, worse developers than those who don't. Possibly because lower competence, but more likely less experience of much damage a flakey CI causes to productivity and morale.

I’m in an organization with tens of thousands of people, and the CI/build team is very far from me in the org chart. The CI system they’ve devised is extremely flaky, not because of anything in my code, but because the CI infrastructure itself is extremely unreliable.

Am I by-definition a worse developer because I find myself in this scenario? If I were a better developer, what would I do? Change teams to the CI team so I can fix their shit? Just quit and join a different company?

Not everyone is in the position to fix the things that are causing them pain.

You would leave and join a company that has their shit together? Or you would convince your manager to provision a host just for your team and put a CI server on it? Or you would run it on your desktop?

Everyone is in a position to fix it because everyone can quit. Life is too short.

Flakiness isn’t always because of the code under test. CI systems can have any number of flakiness issues way before they ever get a chance to run your code. Spinning up VM’s/containers, installing build dependencies, SDK’s, huge amount of prerequisites to even running your code, clusters of test runners that can fall over, bad connectivity, etc etc etc.

Take a look at the kubernetes/kubernetes repo and look for kind/flake to get an idea of what a project looks like that has a tremendously complex integration testing, and how often the integration tests aren’t 100% reliable. I’d wager any organization that’s “all-up”/“end-to-end” testing software past a certain level of complexity (especially anything that touches cloud service integration) is going to suffer this problem. If you’re a developer on such a project, flakes are just sorta the way of things.

> Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right?

There are things like ci.skip option in git for this.

Right, but that leads to a culture of skipping CI for things, which can easily turn into skipping CI for things that are a bit less obviously-fine, which I’ve seen time and time again. Once force-merging over incomplete CI becomes a thing, people start using it more often and suddenly you get actual bugs slipping in.

Here's a single bit of anecdata:

I'm currently working on a computer vision pipeline. Some pipeline components test and build in 3-5 minutes, which is the absolute upper limit on my attention. Other components take 10-20 minutes, and I always switch away to something else.

Agreed that there are categorical tiers of CI times (instant, check something, grab a coffee, grab a lunch, see you next day) and optimizations within a tier don't matter.

I will say blocking review on CI is unneeded and, depending on your system, dramatically slows down down your whole process. PRs being posted is the main notification people get. If people instead poll for CI being done, then you'll get significant lag. This shouldn't negatively affect things. The biggest risk is CI pointing out a change big enough to invalidate the review but that has been rare in my experience. CI should just be treated as another reviewer.

the problem is how much can you context switch. My next story really needs the previous to be through at least (g) before I can start it. I keep a collection of unrelated stories, but right now I'm focused on one feature and so it is really hard to find something that is unrelated to the previous feature. I can make a snapshot at (c), and start the next feature from there, but now if code review finds a critical problem requiring some re-work I have to merge that into the previous code which may now not fit in.

I'm currently working on a low priority story in a completely different area from the important feature my boss needs done because there is too much WIP from other stories. (I need to poke some people to look at my reviews right now - but I know they are working on critical must be done yesterday fixes)

In my current project I'm unable to do step b) because running the tests on my workstation would take longer than 8 hours. So I can only run the tests on CI, where it takes about 1 hour. So i have a 1-hour feedback loop when I'm fixing broken tests.

you could also skip f) by pair-program in a) and b), and then have the CI decide whether or not the code is deployable and then automatically merge into master or another pre-production branch

What about e.1) rerun failing CI build?

Reply on news.ycombinator.com