Hacker News Reader

Comment by clarkdave | Hacker News Reader

If you don't mind tinkering, I highly recommend Buildkite. They provide the management & cloud UI and you run the actual build agents on your own infrastructure.

When you get it going it blows everything else away in value for money. We routinely have hundreds of containers on-the-go and as many concurrent builds as we need; this all runs on a large AWS spot instance at a cost of about $50 a month. Added bonus: our docker cache is always available.

Buildkite provide a CloudFormation stack, but we just opted to run the agents as containers via ECS to make the setup quicker and managing them easier.

View on HN

FWIW, GitLab CI also has an agent architecture so you can run the build agents (which we call runners) on your own infrastructure, on Linux, macOS, Windows, FreeBSD, Linux ARM, Docker, etc.

I'd love to hear more about using Buildkite. It appears to to offer greatly flexible agents with out a heavy cluster to manage or poorly written plugins to deal with...

Happy to answer any questions about it. We've been using Buildkite for a while - it has been a pleasant experience and has let us craft a CI setup whose cost would be prohibitive to us otherwise.

Another similar option we tried is AWS CodeBuild which has per-minute billing and provisions the build machines for you. However, it's very bare-bones and because you always get a from-scratch instance for each build you have to distribute your docker cache which is not ideal.

You are using a large spot instance on ecs?

Yes, it works very well. Earlier this year AWS automated the process of setting up a cluster backed by a spot fleet[0]. It's a very cost effective way to run lots of containers.

We actually use a combination of on-demand and spot container instances in production to keep costs down; we have some logic to provision more on-demand instances in case of multiple spot outages.

[0] https://aws.amazon.com/blogs/compute/powering-your-amazon-ec...

Reply on news.ycombinator.com