How CircleCI Processes 4.5M Builds per Month | Hacker News Reader

How CircleCI Processes 4.5M Builds per Month

stackshare.io

146 pointsemidln83 comments

Is this the only big success story for Clojure or other companies using the Lisp family of programming languages? I know there is Paul Graham's Viaweb. [1] Does anyone know of other examples?

I like the fact that they are a practical company, using Go when needed where "static dependency compilation and fast start-up are more important". I wonder if the ClojureScript's annoucement on integration of NodeJS modules [2] changes that? Also, Lumo [3] is definitely a move in the right direction for this, addressing the slow start-up times for Clojure/ClojureScript, making it suitable for shell scripts and CLI binaries.

> Having a lingua franca also helps reduce overhead when engineers want to move between layers of the stack.

The way I see it, Clojure allows you to use a single language and syntax from the super heavy backend stuffs, to the front end and now to small, fast CLI tools and scripts. A candidate for the "Business English" of the technical world as it were.

[1] http://www.paulgraham.com/avg.html

[2] https://news.ycombinator.com/item?id=14754614

[3] https://github.com/anmonteiro/lumo

There is a list of company success stories. [1] And a list of companies using Clojure. [2]

[1] https://clojure.org/community/success_stories

[2] https://clojure.org/community/companies

Good list, would love to hear more details about their success stories and use cases.

Most of the success stories have links for details below them. i.e. [1]

[1] http://blog.cognitect.com/blog/2015/6/30/walmart-runs-clojur...

Urban Dictionary uses Clojure. We're really happy with it

Nice! You guys rock. Any blog posts to share about why Clojure was chosen and how it's working out for you guys?

We're a pretty small shop, not enough time in the day to write these things usually. I'm sure Aaron would love to if we had the time.

We use Clojure and ClojureScript heavily at Peerspace. Best tool for the job for scaling without adding too much complexity.

Great, would love to hear more about this!

Yeah, I also found it interesting. Curious what's their hiring pipeline for Clojure devs? I know a couple of startups that picked Clojure ended up having troubles finding good devs.

Well, I found it found it hard to hire Clojure developers too. But I addressed it by hiring good to competent developers, asking if they are willing to learn Clojure and then getting them to implement a test project in Clojure as part of the interview process. Since Clojure is syntactically not huge, it is easy to pickup, for good developers. That's my hiring pipeline. ;)

Out of interest, roughly what percentage of candidates agreed to take on the project?

A fairly high percentage. I guess filtering for candidates who are polyglot developers comfortable in more than one language helps. The fact that Clojure is used acts as a filter for self-driven and "better programmers" [1].

[1] http://www.paulgraham.com/pypar.html

> or other companies using the Lisp family of programming languages?

Off the top of my head:

Naughty Dog's games (Crash Bandicoot, Jak and Daxter, Uncharted) are powered by their own Lisp implementations. Google for "GOOL" and "GOAL".

Grammarly uses Common Lisp.

Oh, and if you're searching for air travel connections, your query most likely goes through a huge Common Lisp system (search for ITA Software).

I don't think Node solves the problem of static dependency compilation and fast startup.

With nexe [1][2] and pkg [3] you can now package ClojureScript on Node.js as a static binary, and startup time is now a lot better compared to Clojure on JVM. Go would definitely be better in terms of speed and performance. So it's a tradeoff to make between relative size, performance and having a single language stack, increased team productivity and reduced team overhead.

[1] https://github.com/nexe/nexe

[2] https://clojurescript.org/guides/native-executables

[3] https://github.com/zeit/pkg

It solves fast startup relative to the JVM.

JVM start-up time is below 50 msec. Is that so slow? (Measure a simple hello world, and you will see.) Any slowness above that comes from your code, not the JVM.

Maybe more accurately JVM based Clojure, as opposed to node based ClojureScript (Lumo).

Excuse my brevity. Definitely meant "Clojure on the JVM" vs other hosts.

That's a bit of a straw man though - Go is significantly faster to start than either.

They also mention static binaries which Node does not solve.

It's not a strawman at all- many of us who run Clojure have had to switch to other langs for parts of our infrastructure that require faster start-up time. Think Lambda / "serverless" computing, etc

Agreed it doesn't help with static dependencies, which is why I didn't mention it ;)

It is a straw man - you're justifying your suggested solution by comparing it to something that is already considered bad in the area. That's exactly what a straw man is.

You're suggesting the answer to "we need better startup time" must be "we need the best startup time possible"? In that case, Go is also bad. Everything should be hand-rolled assembler.

Many of us who actually run Clojure in production have been bit by this, and Clojurescript has been fast enough to alleviate the issue. Not a straw man, a community meme.

With pkg [1][2] and nexe [3] you can package ClojureScript on Node.js applications as a static binary. Of course, Go would be better in terms of size/performance, so it's now a matter of making tradeoffs between relative size/performance and team productivity. But at least for teams using Clojure/ClojureScript, this gives you that an additional option, where none existed.

[1] https://github.com/zeit/pkg

[2] https://clojurescript.org/guides/native-executables

[3] https://github.com/nexe/nexe

CircleCI rocks. Our tests ran almost as fast as they do on a MacBook. And their pricing is great.

Of course, that was after trying to get shippable to work.

> Our tests ran almost as fast as they do on a MacBook.

This doesn't seem like glowing praise to me? If I'm paying 50$ per month for a container, I want it to be faster than a laptop...

> This doesn't seem like glowing praise to me? If I'm paying 50$ per month for a container, I want it to be faster than a laptop...

Maybe I am doing it wrong but do people create and destroy a container thingy every time they run a test?

I agree with you though. I am no expert by any means I'd expect Circle CI to be faster than a Macbook at building things like Google Chrome from scratch. But if the macbook already has a head start, then coming a close second is acceptable I suppose?

To be honest, I have no idea what I am talking about and hoping to learn more. My experience with CI is mostly with hobby projects with Gitlab CI and Travis CI. I've only ever "used" Atlassian Bamboo at work if you can call me pushing code to trunk/origin "using Bamboo".

Yes, CircleCI creates a Docker container for every run. In the CircleCI 2.0 config style it is more explicit about this behavior (one specifies the Docker image they want to use) while in the 1.0 API, the config was more inferred / implicit. It can get more complicated than this, but at minimum every build on CircleCI is creating and destroying at least one container to run the unit test suite. It can do this pretty quickly though: a barebones Python repo (1 function, 22 parameterized unit tests) for me takes ~12 seconds from startup to teardown to displaying the completed build using 2.0 with virtual environment caching.

Sorry, I meant to ask about locally. I am still trying to figure out how to write code in docker. It is very confusing and send like too much work to do this locally.

I assume the disk is pretty much always the bottleneck? I mean even with system CTL, how much memory can a rest API flask web app take right?

I'm not sure how it worked prior to 2.0 or if it was possibly, but you can build locally in 2.0 ($ circleci build) which I think is just a thin wrapper around a docker build + docker run + run unit tests (or whatever you want).

That said, you don't have to use Docker and for a small project with no dependencies or no system level dependencies, I think it's overkill personally, so I'd just test outside of Docker or use the new base images from Circle.

https://circleci.com/docs/2.0/local-jobs/

> Maybe I am doing it wrong but do people create and destroy a container thingy every time they run a test?

I create and destroy a container every time I:

* Run a test

* Start a command line environment for debugging (Ruby mostly, so usually "pry" with all my apps libraries loaded)

* Build anything (e.g. update the installed gems via bundler for Ruby)

* Run any scripts that are part of the app.

It means I can be sure I always run things in precisely the environment the apps will run in, including the right interpreter or compiler, right dependencies etc. - nothing "leaks" from my laptop. Nothing "leaks" to my laptop. I can trivially test with multiple different interpreter/compilers etc. without convoluted "environment management" tools that tend to be language specific. Bugs in my script are also reasonably well contained (though I won't trust Docker for isolation as sole protection against hackers, I trust it for isolation against my own stupid mistakes most of the time).

Bind mounts of the source directory ensures I can do live reloading of code etc. when suitable, so I don't need to restart those containers all the time.

Typically starting the Docker container is one of the fastest parts of the above - often my makefile will not just start a Docker container, but for simplicity run the build whenever I do anything (like run the test suite) to make sure I run it in the newest version of the container, and thanks to extensive caching that too rarely adds more than a few seconds.

In a CI setup like CircleCI, though, the "cost" in time of this when building remotely is pulling the relevant source images from their registry, and the images can be fairly big, especially if you're not specifically going to some effort to keep it down in size.

On shippable, the same tests took 100 times longer to complete, if they finished at all.

And I'm typically not sitting there waiting for them to finish. All I care about is that they eventually do and the code deploys. Just don't take forever.

I also understand that the company is going to be serving many other customers at the same time.

And, no, the first server is free. The second server is $50.

  https://circleci.com/pricing/

- I'm from Shippable and we have many customers who've moved from CircleCI to us because of many reasons like better Docker support and the ability to create simple to complex pipelines.

- I'm not sure if you contacted us to improve build times but I'd love to jump on a call and discuss your experience. This is definitely not the typical feedback we get so would like to dig in deeper and improve our handling of your scenario. manisha@shippable.com

- the first build server on circleci is only free up to 1500 build mins per month.

macbook pro's are faster than any VM you can pay $50 a month for.

~50$ a month buys you a dedicated server with a Quadcore Skylake (i7-6700).

The fastest Macbook available right now has about the same CPU power, and comes with less RAM. So no, the Macbook isn't faster than what you could get, but it makes sense if rented CI infrastructure isn't faster for that price.

Just want to point out that there's also the fact that if they want good unit economics, only a fraction of the $50 you pay the is going to paying for the hardware.

Where?

Everywhere?

First provider I checked has a multiple server offerings with quadcore i7-6700 and 64Gb DDR4 RAM for 39€/month [1].

1: https://www.hetzner.com/dedicated-rootserver/ex41s?country=g...

This post makes no mention of the macOS part of the CircleCI infrastructure, which is one of the trickiest parts of infrastructure IMHO. Tricky because Apple's EULA prevents virtualizing on anything other than its hardware, never more than 4 VMs per host, and not for commercial purposes...

Apple's policy on VMs feels so backwards and it really bothers me. Apple expect you to have dedicated hardware if you want to do CI? It makes Mac/iOS projects a real pain and that's not including how much hassle Xcode only functionality is to script.

Apple are a hardware company… they don't make money unless you buy their hardware.

You'd think Apple would be interested in making the workflow for developers easier though so better apps would be made.

It's slightly better now but I've done iOS+Android mobile projects before where CI for Android has been easy to set up and CI for iOS has been a complete nightmare.

I think the workflow is pretty easy: buy a MacBook & an iPhone, and load your app onto the phone & test on it for things which the iOS simulator is not good at.

Don't CI your apps, CI your libraries.

> I think the workflow is pretty easy: buy a MacBook & an iPhone, and load your app onto the phone & test on it for things which the iOS simulator is not good at.

> Don't CI your apps, CI your libraries.

That's not scalable for complicated apps that you can write automated UI tests for though.

Could you take the complicated bits and put them in a library? How complicated is your app?

To me, what you described is an app that was not designed to be easy to test component-by-component & will have a high overhead on maintenance. Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

> To me, what you described is an app that was not designed to be easy to test component-by-component & will have a high overhead on maintenance. Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

We have complex commercial apps with fully automated user interface tests (e.g. it'll test that you can enter your username + password, actually login and you'll be able to see content). You could test all the individual components as much as you want but you're still going to get bugs at the UI layer that you can automatically test for.

We do the same for Android and CI for that is so much simpler.

> Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

You'd think they'd want to support workflows that led to better quality apps being created. Surely it's their responsibility to support developers?

> you're still going to get bugs at the UI layer that you can automatically test for

Surely you can prove the correctness with integration tests of your modules, and then your acceptance tests can be simple 'did the thing not show an error when we clicked on the thing, and it went to the right page'? We have 'simulate the world' 'acceptance' tests at work and they're terribly flaky, and I look back at the layered approach to testing we used in a previous job where the go-live test was to hit the landing page & search results page & check that the status code was 200.

To be fair, the kind of tests I'm proposing won't catch layout issues. But, neither will the tests you're using.

I'm not saying it's always a good idea to write exhaustive UI tests but it's nice to have the option. There's apps that I've worked on where you simple must test several business critical UI interactions haven't broken (e.g. logging in, submitting data) before you deploy. Apple getting in the way of automating this without a decent reason is annoying.

What made your UI tests so flaky? As long as the code for them isn't much beyond "enter text into text input X, click this button, you should be able to see this text" they're not too bad to write but the initial setup can be tiring.

> To be fair, the kind of tests I'm proposing won't catch layout issues.

Screenshot comparison based tools can be amazing for this as long as the way your app looks is fairly stable.

Automated testing is always a tradeoff with effort to setup vs the time you save (I'm against e.g. TDD for everything) but for complex apps and big teams it eventually pays off.

But most of the hardware sales are iPhones. Mac sales are dominated by portables, and I doubt that many serious providers are hosting anything off of MacBooks.

Apple just needs to handle the build process themselves. Let me upload my app code and you do all the hard work. It's not like I can deploy to other app stores...

Apple supports enterprise distribution outside the app store. Like the idea though.

Email Tim and ask him for this: tcook@apple.com

Not quite sure if I understand the question, but we've been using their iOS product for a couple years now and have appreciated their support. They upped their game last year with a key hire or two, although there were growing pains.

Overall we are happy with it. I don't think CircleCI 2.0 is available yet for iOS, though, so nothing new to discuss yet.

My comment wasn't a question, but rather highlighting that a major part of their stack wasn't so much as mentioned in an article about their stack.

A provision they have never enforced.

On the topic of QA testing/automation, it really kills me that Apple servers won't sign older iOS releases which means QA teams need to protect stock devices from accidental upgrades. The AWS Device Farm iOS devices are all jailbroken.

What's the commercial usage limitation? Is that specifically for virtualization of OSX to prevent offering a "Mac On Demand" by not letting you run the OS itself?

Turns out it's two, not four VMs. From macOS Sierra's EULA[0]:

> (iii) to install, use and run up to two (2) additional copies or instances of the Apple Software within virtual operating system environments on each Mac Computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using macOS Server; or (d) personal, non-commercial use.

[0]: http://images.apple.com/legal/sla/docs/macOS1012.pdf

Wouldn't it be covered by c, though? It's an or condition, not an and, the way I read it.

With c) you're allowed to run up to 2 copies. It doesn't really help you run a hundred containers on a powerful server... (Not that you easily legally could, as I guess the moat powerful Mac hw available is the Mac pro - not exactly great for rack deployment).

[ed: i guess you mean running os x server is an alternative to "non commercial use" - I agree with that reading. The option for nc use seems like a nod to not completely make experimentation and creative development/research entirely illegal.]

That's interesting. I always wondered why Macminicolo was so expensive relative to a Linux host (and how they did it).

We are using GitLab CI, and it seems majority of the features of CircleCI are already provided in GitLab CI. Can someone with experience with CircleCI and GitLab CI to talk about some differences?

To be clear, CircleCI doesn't support GitLab. So if you're using GitLab for a VCS host, you might as well use GitLab CI and the integration will be nice there.

If you're using GitHub or Bitbucket, you can't use GitLab CI as they only support their own product. So in that case, CircleCI will be the best choice.

So whether or not you want to use GitHub or GitLab is a different discussion. :)

note: There is a hack to get GitLab CI to work with other VCS hosts but that requires mirroring everything over to GitLab which isn't ideal.

disclaimer: I work for CircleCI

Gitlab ci works best with gitlab. Circle ci is is better suited for github

That pretty much sums it up. I've used both on different teams and like both. But everything in GitLab integrates so nicely that it doesn't make sense to break out of the box IMO. With the 2.0 API that just rolled out, CircleCI has flipped from "convention over configuration" to everything being explicit. I find that it takes a bit longer to write the config files in 2.0 but they're easier to read and understand what's going on because they're more explicit.

Since 2.0 just launched and they have solid example projects across languages, it's a good time to give it a shot. It'll be easier if you've never used the 1.0 version of the product IMO.

Circle works great with Bitbucket too, by the way.

CircleCI has been faster and easier to use than TravisCI. Their macOS builds ran 10x faster and it only took me an hour to rewrite the config. My only complaint is their macOS pricing is based on minutes which I was unaware of. Ended up using 80% of my minutes this month so needed to rethink my build strategy. Not huge, but just a consideration.

I find circle to build significantly slower and it can only build xcode projects. Not impressed at all.

Some of our builds are running on CircleCI but have been trying out drone for a more container focused build. Been enjoying the flexibility and increased speed from custom build images which are trivial to make.

If you don't mind tinkering, I highly recommend Buildkite. They provide the management & cloud UI and you run the actual build agents on your own infrastructure.

When you get it going it blows everything else away in value for money. We routinely have hundreds of containers on-the-go and as many concurrent builds as we need; this all runs on a large AWS spot instance at a cost of about $50 a month. Added bonus: our docker cache is always available.

Buildkite provide a CloudFormation stack, but we just opted to run the agents as containers via ECS to make the setup quicker and managing them easier.

FWIW, GitLab CI also has an agent architecture so you can run the build agents (which we call runners) on your own infrastructure, on Linux, macOS, Windows, FreeBSD, Linux ARM, Docker, etc.

I'd love to hear more about using Buildkite. It appears to to offer greatly flexible agents with out a heavy cluster to manage or poorly written plugins to deal with...

Happy to answer any questions about it. We've been using Buildkite for a while - it has been a pleasant experience and has let us craft a CI setup whose cost would be prohibitive to us otherwise.

Another similar option we tried is AWS CodeBuild which has per-minute billing and provisions the build machines for you. However, it's very bare-bones and because you always get a from-scratch instance for each build you have to distribute your docker cache which is not ideal.

You are using a large spot instance on ecs?

Yes, it works very well. Earlier this year AWS automated the process of setting up a cluster backed by a spot fleet[0]. It's a very cost effective way to run lots of containers.

We actually use a combination of on-demand and spot container instances in production to keep costs down; we have some logic to provision more on-demand instances in case of multiple spot outages.

[0] https://aws.amazon.com/blogs/compute/powering-your-amazon-ec...

CircleCi 2.0 left beta last week and is fully docker based

They also provide a set of official pre-built Docker images for each language so 2.0 is easy to use even if your project doesn't use Docker.

https://circleci.com/docs/2.0/circleci-images/

Wercker is great, but the price jump above the free tier is a bit steep.

I feel like http://concourse.ci/ might be the sweet spot here.

Have you tried Shippable? It's very focused on Docker based workflows for CI and also for building images, provisioning infrastructure, etc and then connecting all these jobs into a dependency matrix with configurable triggers.

https://www.shippable.com

Main downside is paid-for parallelism. Ideally (and I'm sure they'll get there) is per second billed builds.

Other that that, great job and nice write up.

I think each container can have up to 8 cores. So using multiple containers to parallelise your tests is your choice.

That's pretty impressive, given that a build typically hogs CPU as well as IO.

Reply on news.ycombinator.com