What Is Proper Continuous Integration? | Hacker News Reader

What Is Proper Continuous Integration?

semaphoreci.com

73 pointsmarkoa48 comments

> As with all ideas, everybody does their own version of it in practice.

Yes.

> Continuous integration (CI) is confusing.

At core, no. Not really. See the 1st paragraph of https://en.wikipedia.org/wiki/Continuous_integration :

> In software engineering, continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day.

So the 2 words are defined as follows:

Continuous: at least several times a day.

Integration: merge changes to master.

It's not a tool: it's a practice, a workflow that builds, tests and tools can help you do safely.

Some of these "CI Server" tools also support "build and test the branches" which is useful but is not CI. This is a common mistake.

Really does sound like "Let me shame your practices by redefining what this means in order to sell this service."

As long as those branches are merged in daily, it's still CI.

Your working copy is just an informal branch. Sticking it in a feature branch while your working on it is just convince.

Yes. I meant that "build and test the branches" is not itself CI. But it can be a useful practice before merging in a CI workflow. Or it can be a painful set of long-lived branches.

But some people have the idea that the build and test of any branch, i.e the automated production of a binary that passes checks, is the "integration" itself. Look out for statements like "We do CI of branches on our CI server".

Someone introduces $buzzword.

As software is a popularity contest industry driven by hype and career building blogposts, now everyone needs to do $buzzword so that they can play SEO with their resumes. This is a highly competitive market after all and you gotta put food on the table.

smug-faced guru enters the scene: "Oh it looks like you're not actually doing proper $buzzword. Shame on you for raising the hand!"

Now you're trapped. You either a) do what the guru says or b) remove $buzzword from your resume before you kiss your children goodbye and prepare for your new homeless life.

> "Of course our build takes long, we have over 10,000 lines of code!"

Well, my project misses the 10 minute target by a factor of four, but does include tests and have 670kloc of C++. I claim continuous in this context, thanks very much.

Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.

Indeed it could, and you're looking at the result, down from 2.5h-ish when I arrived. Our pipeline runneth deep ... (Though of course we'll never stop whittling!)

Aw man!

I've been on projects that did this - just linking everything back together if you touch a widely used subproject can blow that 10 minute budget too, for a single build flavor. When porting things for game development, I routinely have 30+ build flavors (say, 5 platforms, 6 configs per), and want to test at least a few of them to have even a remote chance of the checkin going green when touching core platform abstractions or config-specific stuff.

A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky.

Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...

I quite agree with you!

I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.

500kloc here! Produces a 200MB binary in debug, 8MB in release.

We managed to get it down under about 15 minutes for the most common platform by using concatenation builds. It's much much faster to build half a dozen files than a thousand.

At a previous employer we were up to 1.25mloc by the time I left and we had a different solution: ccache+distcc distributing the build across a room full of computers. Few files took longer than 5m to build all by themselves, and the link phase also took about 5m.

(Splitting up large C++ projects that have grown organically is really hard. It requires re-architecting and commitment to spend time on this technical debt.)

Do you experience any real consequence from this, as the author suggests?

I don't deny that can happen, but I have seen teams with build times that exceed 10 minutes that don't have issues. People aren't generally dumb, so they find a way to work that doesn't involve constantly sitting around waiting on the build.

Moved from a team that had build times of 15 minutes to a team that had build times of hours, it felt crippling :(

Hours, sure. But your original 15 minute build time exceeds the author's 10 minute cutoff. I'm questioning whether that seemed to be a huge issue.

We do, yes. The main one being developers will often have to make a judgement call regarding only doing partial builds and/or coverage so that the commit is not always certain to work during CI (despite what they pwomise to do regarding pre-commit testing). It also means that they have a chunk of mental load that lingers after the commit, the bot might DING them and they'll need to switch back. I don't think the distinction between 10 and 40 minutes is as much as the 40 to 2.5 hours difference -- any problem worth solving is going to take you half and hour IMHO (but let's not bike-shed that), but 2.5 hours you can start a whole investigation from scratch etc...

If you switch to a modern build system like Bazel or Buck with remote caching, you will likely get that down to < 5 mins per change...

They look interesting! I keep an eye on these re: Windows support (we already use IncrediBuild there, and distcc and ccache on Linux ...) We have some monster links that can take upwards of 8 minutes on our fastest machine with SSDs etc, so the 5 mins seems keen.

I think this really depends on the code base. If I had a website where a change would trigger more than 5 minutes of integrating and testing I would go mad.

I'm working with a weather model and it takes around 1 hour to build and test everything: We build against 2 super computers, 3 different compilers, 2 different architectures (CPU, GPU) and single and double precision. Building alone takes 30 minutes (more in some cases). For testing we reserve 1x node with 8 GPUs and 9 CPU cores on one machine and 2x nodes with 1 GPUs in a 30 minutes debug slot.

With a pull-request based workflow we are able to push to the master multiple times a day. 40 minutes might be achievable by massively revamping our build mechanism and getting jenkins to store gigabytes in build artifacts for each run. However, I do not think it is worth it, because changes can take multiple days, or in some rare cases months to implement. If people need to wait for an hour for their tests to validate that's not so bad.

Our tests can take hours to days and need multiple hardware configurations. I am already happy if all tests run once a week or even once a month. In the past a lot of tests would be run maybe once or twice during a year long release cycle so once a month is already huge progress.

I think this would be too expensive for us. Why do they take so long? What area are you working in? Would it make sense to scale down the tests to a sensible size and do the longer tests over the weekends?

This is an okay and mostly correct introduction for anyone who has never heard of CI, but lacks any sort of depth. The author sounds like he just discovered the concept and wants to share it, but lacks practical experience.

And yet is a blog post from a CI platform!

Yes, I realized after commenting. It's not a bad post, but I'm not sure why they tried their luck here.

Hackernews provides a relevantly sized audience. "Roughly 2.6M views a day, 300K daily uniques, 3 to 3.5M monthly uniques. It depends on how you count, of course."

And that was two years ago...

https://news.ycombinator.com/item?id=9219581

Yes, I suppose I am aware of their product now...

It's an advertisement for their CI software in the form of an informative blog post to hide that it is an ad so they can post it on websites like hacker news.

The original poster is a cofounder of Semaphore.

The domain is "semaphoreci.com" and the service is linked very early in the article, I don't think they're trying to hide anything. It's content marketing, just like 80% of the blog posts that end up here, except for Medium which seems to be exclusively marketing.

Hi, author here. I do come from a team making a CI service, and it took us a lot of time to realize that it'd be useful to have a time-based hard limit for what's good enough. And for anything over that you're obliged to take action to make it better.

E.g. one of our own builds took longer than 10mins at some point. You just assume that it's a function of size of the code and it's normal to rise over time. So my goal is to share that idea & why I think it matters, and get feedback. :)

How is the hard limit enforced? By the CI tool as a failed build?

It would be perfect if one could somehow keep the build quick using tools. A hard limit on a build might not be the best way. If the test is taking Limit minus one millisecond, and I add one new test that takes one millisecond, I break the build. But the culprit was really commit yesterday that added a test that takes 9 minutes. The worst thing one can do in CI is somehow flag the build as "not good enough" and point the blame the wrong way.

A tool that treated test quality/perf like any other asset and fails builds based on time limits etc would be perfect. It's really hard to do that especially on shared CI servers because of fluctuating performance.

Yeah sorry for not being clear there, by hard limit I didn't mean that CI should fail the build. But that the number should be a fixed threshold for the team to optimize for. Thanks for your feedback.

If it's not a function of the size of the code, then what does build time map to? I understand that dependency management, effective testing, proper build configs, etc. also factor into that function. I also understand that size of the code does not have to be a linear scalar, but it seems that a constant hard limit is very arbitrary.

I'm really not trying to crap on your decisions. I just want to know the rationale behind it. There are projects where everything is written in house with 10K lines of code. There are behemoths with 500k+ LoC and numerous dependencies. How does this factor into the decision?

Oh, I meant that "size of the code does not have to be a linear scalar". Also by hard limit I didn't mean that the build should auto-fail or anything (sry English is my 2nd language).

But if we have a rule of thumb, we can know when it's time to start thinking about optimizations.

Code size is certainly a factor but it doesn't have to imply constant time increase. Speaking from experience in web apps, optimization of setup (whatever needs to happen before first test runs) + more parallelism + thinking more about decoupling is usually the right answer.

The point of these discussions is mostly to help people avoid ending up with a single 500k+ LoC behemoth.

Hi! Unfortunately, compilers must take at least linear time to compile code, because they must examine every character. Ditto for pretty much any serializer. Also, you'd think that the number of LoC matters, but the choice of language matters too; look how long it takes to compile terse high-level languages like Haskell. Finally, as you probably already know, many steps of a build are non-parallelizeable, and as others have already pointed out, even a task like linking a binary can take multiple minutes at scale.

I really don't think build time is very important. With a separate CI server it doesn't much matter if takes an hour or <10 minutes for a build to complete, the gain for me is that it's no longer "your" time. It's some other server that's doing the work with a saved state (master).

I'll typically just merge into master on GH, let the CI server take it from there and not bother checking the site manually when the build is complete, it's not necessary. Our QA and CI process handles that.

I consider it a major advantage (benefits far outweigh risk) that I can typically merge my ticket into master, close my laptop and head home.

Well a major problem in many organisations is perceived sensitivity of source code, and therefore a reluctance to outsource many tasks which rely on access to such.

I like to think "we'll get there".

More important than build speed is the confidence that code which passed to build really works.

I think one big reason why organizations don't integrate and deploy often is that people are afraid they will break something when they do this. Then you end up pushing this big and scary thing forward and forward. This of course does not take away any of the risks, but at least you need to close your eyes and wish for the best only once a month.

For those using Selenium or something similar, do you have any tips for making test suites fast? Usually I just test the main flows with Selenium, then go more indepth with unit-tests where parts of the application are mocked out - but it's a trade off between accuracy and speed.

Run tests in parallel, only way too make it fast.

That's a good trade off though, and not just for speed. Selenium tests are more difficult to write and take more effort to maintain. You should definitely test as much of your application as you can with unit tests and integration tests. The end to end tests with something like Selenium should then focus on the type of bugs that you couldn't catch with the lower level tests.

I've seen situations where people had Selenium tests for every possible validation error on a form. A lower level test would have been sufficient to test all validation permutations and making sure they return the right error message, and then a single high level test can ensure that when an error message is returned it is properly rendered on the screen.

I'd advise definitely covering all usage scenarios with Selenium. And then parallelize.

Overall unit tests should greatly outnumber UI/integration/acceptance tests. And when you catch a bug with Selenium test, replicate it at the right place with a unit test. Look up "test pyramid" if you're perhaps not familiar with it.

- Test cases should be designed to allow parallel execution (they should not interfere with each other)

- Do not use sleep() for synchronisation - use explicit waits (e.g. wait until element is visible, then proceed immediately)

If you haven't encountered the concept before, I recommend looking at the testing pyramid[1]. Basically the idea is that UI/Functional tests take a long time to run so you want just enough to test the main, important flow in the application while you want a ton of unit tests because they run quickly and give you very targeted feedback about what might be broken.

[1] https://martinfowler.com/bliki/TestPyramid.html

Well, what if the CI is first building xcode project and then deploying it on device and then running tests? 10 minutes?

Is that now a thing to put your css at the end of the page?

What exactly is the significance of 10 minutes? Who came up with that part? These kind of purity tests can get a little silly, I think.

The "ten-minute build" is an idea from Extreme Programming, and you can find it in Martin Fowler's article on continuous integration[1]. (That's probably where OP got it from.) The reason, as explained in Fowler's article, is to provide rapid feedback. So the person who came up with it is Kent Beck.

[1] https://www.martinfowler.com/articles/continuousIntegration....

Quite a lot of words in the comments and article, but not the word "automation" for some reason :P

Reply on news.ycombinator.com