How to set up continuous deployment using free hosted tools | Hacker News Reader

How to set up continuous deployment using free hosted tools

simonwillison.net

411 pointssimonw88 comments

I can't believe the author doesn't mention gitlab. Setting up full CI on gitlab is amazing. They have the easily integratable gitlab runners i.e. travis-ci functionality + different build stages for automatic deployment via ansible/chef/whatever and unlimited free private projects to boot. If you're working on anything that can't be public I highly recommend gitlab. Its an amazing service.

FYI: I am in no way affiliated with Gitlab.

I haven't tried it :) I'll definitely give it a go.

I think I saw that GitHub now supports non-gh_pages branches, but Gitlab pages are also much easier to setup.

I've moved all my sites to Gitlab pages generated via Hugo or Jekyll.

Uptime (via Uptime Robot) just doesn't seem as good as GitHub, but it's not bad.

> , but Gitlab pages are also much easier to setup

uuh, what?

i think gitlab pages are wonderful, because you're not restricted to jekyll, but how is it easier to setup? you only need to toggle the checkpoint in the repository settings...

Sorry, easier to setup because you're not restricted to Jekyll/can easily use Hugo/etcetera.

Edit: And what I said in original comment. Gh_pages branch was required, but Gitlab could just use the built assets.

GitLab is great and has come a very long way over the last several years. However, I still gladly pay GitHub $7 a month for my private repos because I feel that GitHub truly does offer a better code review and repository experience. GitLab in certain places is still very rough around the edges, not to mention performance has always been a concern.

To me, GitHub still is the gold standard and will likely be for quite some time.

Not sure how, it's got the same here is your changed files and the changes.

I find GitLab is better on a few point. You can theme it from about 4-5. I prefer the dark theme as it matches my IDE and easier on my eyes. You can also have comments marked as addressed. By default the changes are shown in the page but you can also choose to show side-by-side like in GitHub.

You can host it yourself. It ties in very well to it's own CI/CD service and comes with a docker registry. So your code can be tied easily to your deploys and you see this in the issues.

> can't believe the author doesn't mention gitlab.

I absolutely agree. We use Gitlab in our company for continuous deployment and it works like a charm.

Apart from the other tools I miss some performance monitoring. Sentry is great, but if your application suffers from bad performance after the latest deployment, you won't notice that soon enough.

I recently found Datadog for performance monitoring (it's like statsd++++) and I'm already a big fan. I haven't managed to get it working with my Heroku projects yet though.

We're working on adding more performance monitoring to GitLab https://docs.gitlab.com/ee/administration/monitoring/perform...

A great self-hosted CI alternative is Drone.io (https://github.com/drone/drone).

To set it up is simply a matter of starting a Docker image and creating a .drone.yml file (very similar to .travis.yml).

Note: I'm not affiliated with the project in any way. I'm just a happy user who think the project deserves way more buzz.

Once you need something a bit more heavyweight than Travis (e.g. more CPU) I would highly suggest Buildkite (https://www.buildkite.com). It’s a “bring your own machines” solution, which means you can have beefy machines (e.g. via buildkite on EC2 spot autoscaling), or even machines that run inside production without handing buildkite itself any secrets. We’ve been using it at Ladder and it’s been phenomenal.

Also it has a lot of emojis.

+1 for mentioning Buildkite, also works great for machine learning

Is there a particular feature that's suited to machine learning? I'm having trouble understanding the connection.

the way one process can feed into the next. you can setup data pipelines that are nice for ML projects.

Another very happy buildkite user here. It's extremely flexible and the crew just keep adding feature after feature.

Heroku does have a free tier, but it's intended for experiments. It's very limited and wouldn't be acceptable for any serious website, certainly not a 'world class' setup. Unless you're very careful to have one free dyno per account, and avoid using features like staging apps, they will power down your server for several hours a day.

You probably should not use the free tier for any serious website or 'world class' setup...

Yeah, the “world-class” hook was more about the quality of these tools than their applicability for use at scale for free - I encourage people to pay real money at various points in the article.

I’m currently running my blog on a single $7/month Heroku Dyno (plus $9/month for Heroku PostgreSQL) behind a free Cloudflare account - with the right caching headers I can easily handle huge amounts of read traffic. I was going to mention Cloudflare in this piece but I decided it didn’t fit the theme of continuous deployment as well as the other tools.

> Once you’ve set up Travis, it will automatically test every commit to every branch

Small correction: It will automatically test the latest commit on every push to every branch

Thanks - I updated that sentence to say push instead of branch.

You can manually test all commits in push though. Or automatically run bisect on failure, etc.

I'm the developer advocate for Codeship.

Codeship (https://codeship.com/features) allows forever free builds for public repos. It has support for both Continuous Integration and Delivery/Deployment. Maybe you can give it a spin as well?

I can second codeship - even though its a bit more clunky than Travis and with less integrations and command-line tools, its has way beefier machines, easily several times faster than what travis offers even to paid customers. And their tools are improving. Keep up the good work.

We did not pursue Codeship due to lack of a "codeship.yml"-style configuration so that we could track config in source control.

Codeship founder here. We support .yml file config for our container/Docker based product (called Codeship Pro). We're looking into offering a UI + yml file approach for both products because we see a 50-50 split when it comes to what people prefer. Thanks for the feedback :)

>The integration of Travis with GitHub runs deep

At the moment there are 10+ companies that do the exact same thing.

http://blog.codepipes.com/hosted-ci-comparison/hosted-ci-com...

If you don't have time to click the link:

  - Buddy Works
  - CircleCI
  - Codefresh
  - Codeship
  - DeployBot
  - Distelli
  - SemaphoreCI
  - Shippable
  - SolanoLabs
  - Vexor
  - Wercker

https://buddy.works/ is great, fast builds, slick UI, and a free self-hosted enterprise version.

Only comment I would make is that you can use CircleCI instead of Travis to get for free running tests on private repos. The configuration is similar and the benefits as well.

Thanks for the tip - I just added a note about Circle CI to the article.

Doesn't show any automated verification of code in deployed fashion, no load tests or verifications which would be part of a world class pipeline. Also no canary or red-blue tree type rollout hooked to sentry. At least it has a staging but don't know what they're using it for other than to test migrations before promoting to prod. Nice little write up but far from world class.

Exactly - continuous deployment is about moving some/most testing to production, not eliminating it altogether.

Acceptance testing the system once deployed is the cornerstone of organisational acceptance of the CD doctrine - errors will always happen, it's just that they can now be fixed much faster in production.

Using an acceptance test suite as a deployment safety harness should dramatically increase the quality of tests, and hopefully the application too.

> Need more? Pay for it!

or be willing to move vendor and re-implement the stuff.

For comparison my magic combination(and not all free or realistic for everyone):

  - gitlab
  - jenkins/looking into gl ci
  - linode/cloudafrica(ah-hah hosting is not free)
  - sentry is a tough one. We rolling a custom mqttwarn/zabbix solution
  - irc(fine I will take your slack and use the irc-bridge pfftt)

irc is fine but if you want an alternative, you can look into riot/matrix (irc bridge), mattermost, rocket chat, etc.

For hosting, I am looking to host my own server for fun and learning.

I really wanted to use Gitlab for my next side project, but Circle CI doesn't support them, and I really don't want to pay Travis for my shitty proj. Hopefully this changes some day soon.

Gitlab has great built in support for CI [0], and you can easily add and run your own CI runners [1].

[0]: https://docs.gitlab.com/ce/ci/README.html

[1]: https://docs.gitlab.com/ce/ci/runners/README.html

We use gitlab's built-in runners and pipelines for continuous deployment to kubernetes and it works very well. We self-host our repos but the same tools are available through the hosted version.

GitLab's built-in CI is very good, and simpler than Travis for some things.

You also have the option (unlike Travis) of registering your own runners in your project settings, and you can do this either on gitlab.com or another deployment, without needing to be an admin of the deployment - you can get a key for it in your own project's settings. I'm using this at work, where we have an internal GitLab that I'm not an admin of (and where the GitLab admins probably have no interest in running a CI system), to do a small bit of CI on some things using one of my team's servers as a runner.

- sentry is a tough one. We rolling a custom mqttwarn/zabbix solution

Sentry is open source, you can self-host it...

A tough one, in the sense of suggesting a better/other alternative. so sentry... gogogofireintehole

> - jenkins/looking into gl ci

Yup, definitely look into it. We just killed our last Jenkins instance last week after we found out that everyone eventually moved to the GitLab CI.

> - sentry is a tough one. We rolling a custom mqttwarn/zabbix solution

Why a tough one? I found Sentry easy to self-host. They have pretty good docs: https://docs.sentry.io/server/installation/

> - irc(fine I will take your slack and use the irc-bridge pfftt)

We ditched Slack for Mattermost. It's not entirely up to par (yet), but works for most use-cases. And there's an IRC bridge for it, too (https://github.com/42wim/matterbridge). Oh, and Mattermost comes bundled with GitLab in their omnibus.

> Why a tough one? I found Sentry pretty easy to self-host.

your right, sentry! It was tough thinking of alternatives.

Heroku is awesome for this use case, but there are also great alternatives based on Cloud Foundry. Cloud Foundry is an open source PaaS with support for Heroku buildpacks + docker containers. At a median of ~5€/GB RAM per month they will also be a bit cheaper than Heroku.

Some providers that offer Cloud Foundry including a free tier for small Apps:

- Meshcloud (hosted in Germany, 2 Locations) https://panel.meshcloud.io/#/register (Disclaimer: I'm a founder there)

- Pivotal Web Services (hosted on AWS US): https://run.pivotal.io/

- IBM Bluemix https://www.ibm.com/cloud-computing/bluemix/ (various locations, a bit more expensive than the other two though)

Shameless plug: I wrote an open source incoming webhook server that can run your build & redeployment scripts: https://github.com/adnanh/webhook

Thank you very much for this!

We use this to listen for Github webhooks and execute jobs with Saltstack.

I have a similar setup, but using Docker images instead of Heroku buildpacks. Buildpacks are horrible if you want to do anything custom.

To make this work, I use Codeship instead of Travis. Codeship builds a Docker image, runs the tests against the image, then pushes the image to Heroku if the tests pass: https://documentation.codeship.com/pro/continuous-deployment...

You can also do every step listed, for free using a single platform: VSTS (Visual Studio Team Services).

Online, free for small teams, Git source control, CI/build/release, supports every major build stack, not just MS. It even has a top quality agile work tracker, that I've found better (or at least more balanced) than Jira/Trello equivs.

EDIT: link: https://www.visualstudio.com/team-services/

Not sure bitbucket pipelines are mentioned here. But for free I manage to commit Hugo source files to a private git repository, generate the static files by spinning up a docker container and generating them using Hugo and finally rsyncing the static output to my Webserver. All for free. You only get 50 minutes of time now (for free) in the docker instance but for simple continuous deployment cases like mine it’s very effective.

You might want to look at Gitlab if you want to drop the need for a web server. Their pages support supports Hugo much better than GitHub (which leans toward Jekyll), and is pretty well documented.

The last time I tried using pipelines, it wouldn't let me use any arbitrary Docker image, I had to use one of the Atlassian "blessed" images. :/

Add Netlify and it will continue to be free. I use Hugo as well, Netlify compiles Hugo on it's own, so I don't need workflows, but it is cool to have them.

Why is that better than generating static locally? Seems like a lot of fiddling for no benefit.

It’s fully automated. I do a commit and push from any machine that I have cloned the repo to and it gets published automatically.

Now I don’t need to download Hugo, remember the Hugo commands, remember my web server login details, etc. I think it’s a lot less fiddling as all I need to remember now is how to add a file to a git repo to get a post published.

Don't you miss out on Hugo's archetypes doing it that way, or at least need the templates locally? Also you shouldn't need your web server login details for rsync... Hey whatever works, it just seems like substantially more work to me.

I just set up CI tests for some of my Python libraries on GitHub, and another tool to note is Appveyor -- very useful for running tests on Windows (which Travis doesn't do yet). Example: https://github.com/benhoyt/scandir/blob/master/appveyor.yml and https://ci.appveyor.com/project/benhoyt/scandir

I have started using visual studio team services, it is really good and it's free. https://www.visualstudio.com/team-services/

I used to use team city, octopus and jenkins in previous jobs. Personally, it is really easy to use and you don't have to use the microsoft stack. It supports Python, Ruby, node etc... It feels to me like they waited to see all the good bits people have done and then just implemented them.

I didn't want to manage a server so it is perfect for me.

GitHub and Travis are not self-hosted tools.

It would be nice to see a setup with something like Gitea[0] and/or Jenkins[1] edit and some other slack alternative like mattermost[2]

[0] https://try.gitea.io/

[1] https://jenkins.io/

[2] https://about.mattermost.com/download/

"Free hosted", not "self-hosted".

The title says 'hosted', which is the opposite of 'self-hosted'.

> Travis are not self-hosted tools.

Pretty sure you can self host travis.

A bit that's missing: bors-ng https://bors.tech to only merge PRs which passed testing.

Does anyone have a solution for a situation where I have a Django backend and a VueJS front end and want to run the Vue build step before deploying? Currently doing this with Git hooks and committing the resultant build, but that is bad for a number of reasons. This is running on Heroku. Would their release phase allow for this without significantly slowing things down? Would it deal with npm installs?

We have this with GitLab CI, although with React and not Vue (but I don't think this matters). We have two stages, `build-assets` and `build-app`, first one runs Webpack etc, and the second one copies the build artifacts into a Docker image with Django and finalizes the image with collectstatic.

We're using Docker, but I'm sure the same approach can be used with simple tarballs, OS packages (.debs, etc) or deploy-specific git repo (for Heroku)

You do not need to commit your build to git if you are using heroku.

If you add the nodejs buildpack to your heroku project, it should detect and build your package.json automatically on heroku. Works with both yarn and npm.

You just need to make sure to set NODE_ENV=production in your heroku config.

https://devcenter.heroku.com/articles/nodejs-support#build-b...

You might be able to solve this using a bin/post_compile script - most standard Heroku buildpacks will run these as part of the build step, and they can be used to run commands that generate files which will then be included in the compiled app that gets deployed to Heroku.

Here's an example that might be useful: https://github.com/nigma/heroku-django-cookbook/blob/master/...

I've done this for several apps. We have a large Jenkins 2 cluster and use declarative Jenkinsfile's. For Jenkins, you can accomplish this by simply setting up a Jenkinsfile that has an agent defined in each stage. Build your VueJS app in the first stage with a NodeJS container, and build the second stage with a Python container. Bonus points for building an actual Docker image as your artifact that has Django inside of it with Apache or nginx serving the VueJS bits too.

For Java/Postgres/React/Google App Engine, here's a skeleton project using CircleCI:

https://github.com/stickfigure/gstrap-skeleton/

Deployments are git checkout production; git merge master; git push. I love being able to use git log to see what changes are live.

An alternative to travis-ci which also works well: circleci.com. It has a free tier (whatever the open/close status).

I use gitlab instead of GitHub and Travis and tend to use the free tiers of GAE and Heroku indistinctly. But man, I wasn't even aware of something like Sentry existing, I'll have to get me some of that.

I also need to think of the messaging system. I'll probably start using mattermost instead of slack, though; because GitLab.

I still have my stuff on AWS instead of Heroku and use Rollbar instead of Sentry (not for any specific reason), but overall I'm really happy to hear that I'm tracking well with this setup. I find it smooth as butter most days. Thanks so much for sharing!

Did Travis replace Jenkins?

Thanks for posting this. I'm looking to get any application and start deploying through a CI/CD process, to learn how this process works. Any application suggestions for a CI "Hello World"?

Have a look at gitlab ci[0], the quick start was easy. Got it working online and self-hosted.

https://about.gitlab.com/features/gitlab-ci-cd/

I think you mean CD “Hello World”, not CI. You can make a “hello world” web app (or even just an API endpoint) in your favorite language and set it up to deploy on each commit.

CD means each commit that passes tests is deployed

CI means your code is regularly integrated (merged) vs having long QA / code review cycles.

CI and CD work great together but you can have one without the other.

Yes, I want to better understand the process of committing code to the repository of an application, have it run automated tests, report via a communications tool the result, and automatically deploy if successful. Understanding this efficient process is high in demand and thus highly profitable.

I've created and worked on several projects that have involved APIs, reading from Zendesk, and writing to Slack or HTML, among other places. I've supported Jenkins and related tools multiple times in production. Now I want to fill this gap in my knowledge.

Travis is a lot more user-friendly than Jenkins and they run it for you, whereas Jenkins you usually self-host. (Jenkins is a lot more flexible and has a lot going for it, but Travis is much easier to get up and running)

Travis is a third-party service. Jenkins is (usually) software you run yourself. Travis will run the infrastructure for you and will not charge you if your project is public on GitHub. So that makes it an extremely popular choice for open-source projects: sign into Travis with your GitHub credentials, click a checkbox on your repo, and add and commit a .travis.yml file, and now you have CI that is well-integrated with GitHub and you don't have to run any infrastructure or pay for any server hosting.

Of course, you have much less control over Travis than Jenkins. Two cases where Jenkins remains obviously better are when you need custom environments that aren't easily portable to Travis (Windows support is a big one, although there is a company called AppVeyor that is something like a Travis for Windows; licensed software is another big one), or if you have strong security/confidentiality requirements (e.g., you're in a regulated industry) and using random cloud services isn't a good option.

You're missing Concourse

Two hours after posting, and this is the only comment about Concourse? jacques_chester must be on holiday.

Is it possible get get Concourse running entirely on free tier stuff? I don't think there's a hosted Concourse service yet - airplane.ci is coming, but still in beta - and i imagine you'd struggle to get a whole setup on a t2.micro.

I like Bitbucket better for my personal private repo's. Also would like to mention few more tools I like, Netlify for static site hosting, SemaphoreCI as alternative to Travis, I think both are good.

When they say that AI is not going to take over dev jobs, show them this thread and tell them that the time cut down because of automation is going to be spent in something more meaningful, like coding!

I would prefer Jenkins instead of TravisCI. And bitbucket provides private repos and I always prefer BitBucket or even Gitlab but no GitHub.

This Talks about Freehosted tools on Code that is Opensource - for Closed source - this does not work.

any suggestions for self hosted stack?

Instead of Slack -> Mattermost or Rocket.Chat

Sentry also has self-hosted option available (and it's also opensource).

Instead of GitHub + Travis CI, you can use GitLab (it also has bundled CI capability, based on Docker).

with GitLab, you can also have Continuos Deployment, wired up with the CI solution. The best experience will be deploying to Kubernetes, as it will give you even monitoring, automatically, but you can code custom deployment logic as a step in the CI pipeline.

  [ 'GitLab', 'Jenkins', 'Vagrant || OpenShift[.org]', 'Errbit', 'IRCd || ejabberd' ]

Just keep in mind that "self-hosted" means "self-supported". I hope you pay yourself well.

In my world, its not world-class if i don't get some SOX compliance around it. Its a cool example though.

So you equate "world-class" with "enterprise"? I disagree. Tools that scale down can also be world-class, and in fact that ability to run cheaply is a big part of what allows startups to get off the ground and bootstrap their way to eventually challenge the big boys. Heck, even if they stay small, they can still have world-class tech. Once you get big and purchasing decisions are made far from the front lines I'd argue it gets harder to really have the best tech.

>>So you equate "world-class" with "enterprise"

Well as part of my current world, sure I do. I have to satisfy the needs of not just dev, but operations, compliance, security, business, the people overseas that i've never met, etc. What I'd really like to see are startups that bootstrap this stuff to think that way prior to growing.

My original point was that this was cool for the dev crowd and perfect for the blog use case, but "world"-class seems hyperbolic. But if you are working for other people: people walking around who have strung together various automation stacks are a dime a dozen -- I've literally seen 4 different takes this month. Execution matters, and execution for your audience matters more.

>> Tools that scale down can also be world-class

We are of the same opinion. I love me some elegant free solutions, and I need to because then I can judge what I'm buying when I need to buy something.

>>Once you get big and purchasing decisions are made far from the front lines I'd argue it gets harder to really have the best tech.

I guess I see where you could be coming from, but it doesn't HAVE TO be that way does it? Isn't the whole point of work to empower yourself to affect the company decision-making. Failing at doing so is sort of, without offending you, you not doing your job. At least that's how I look at it.

[edit: formatting]

Reply on news.ycombinator.com