Hacker News Reader

Comment by solatic | Hacker News Reader

Hi Mitchell,

Could you please elaborate further on how you see Waypoint integrating with CI systems?

It seems like you expect developers using Waypoint to deploy straight into production from their laptops, which is an anti-pattern, particularly for any company larger than a few people. Companies really need to make sure that code is checked into version control and that tests are run before code is deployed into production.

How do you see Waypoint being used in companies where there are more structured pipelines to build code?

View on HN

We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution

Generally speaking I like to describe Waypoint in CI similarly to Terraform for infra. Prior to Terraform, all CI environments had this messy step that was like "create/manage infra" and it was mostly homegrown scripts that are cloud-specific.

Today, CI systems often have "deploy" which are similarly: home-grown logic that is platform-specific (kubectl for K8S, packer/terraform for VMs, etc.). We believe the "deploy" step in CIs can be replaced with "waypoint up" or "waypoint deploy".

The link above shows you how to do this.

I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak.

However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong.

I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists.

If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?

> I'm not sure how much that's the case with application development though.

Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated.

And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible.

I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.

Hey Mitchell,

It's an interesting idea, but as a Devops Lead, I'm not sure this is any where near our biggest pain point. We've been in the midst of a Devops transition from manually managed EC2 and chef towards CI/CD and infrastructure as code for two years. Terraform solves one of our biggest problems - consistently representing infrastructure as code. But Waypoint won't solve the other. In fact, Waypoint doesn't seem to touch on anything that's actually much of a problem.

The deployment step isn't the problem - it's everything surrounding it. See, we're not on CI or CD yet. We're working our way towards them from a legacy system. In that intervening time, what we need is a generalized build tool that provides the ability to do CI or CD, but also allows you to just write general build and deployment pipelines that allow you to gradually automate that process and run them manually outside the context of an automatically triggered pipeline. We've been in this stage for two years, and the only tool that actually does this coherently is Jenkins. But Jenkins is an absolute nightmare to work with on many levels. Our only other real alternative would be to cobble together a mess of bash/python/nodejs scripts or use chat ops. There isn't a good solution in this space.

Gaia Pipeline, an open source project in development by a small team lead by a Hashicorp engineer, looks like it could be our long wished for Jenkins killer. I'd really love to hear your thoughts on whether Hashicorp might be willing to throw its full weight behind it: https://github.com/gaia-pipeline

Totally agreed ! I still haven’t found something other than jenkins that would work for our use case which is very similar to what you stated.

You can check out Drone.io. It's based around plugins, which are basically Docker containers which accept env variables for configuration and a volume with the code (which Drone clones from git). You can easily write custom plugins in any language you want, test them, and run them locally outside of Drone with little configuration.

Depending on where you are in the transition, Garden (https://docs.garden.io/basics/how-garden-works) might be a good fit. It definitely touches on all of the surrounding problems such as managing dependencies, integration tests, and running arbitrary tasks (think DB migrations)

You define one part of your system at a time and Garden compiles that into a graph of modules that it knows how to build, test, and deploy. This means you can adopt it piecemeal and use it to run an entire CI pipeline from any given context, e.g. your local machine, VM or actual CI for that matter.

Full disclosure, I'm affiliated with project but your comment describes a problem we see from a lot our users so I couldn't help but replying.

I'm not Mitchell but I think this page already answers your question:

https://www.waypointproject.io/docs/automating-execution

So basically you can integrate waypoint to your existing CI/CD solution.

Kind of using it as a Makefile for pipelines, if I understand correctly.

Ooops, burned :P

Reply on news.ycombinator.com