Comment by mitchellh | Hacker News Reader

Comment by mitchellh | Hacker News Reader

Parent Full thread

Hello HN! I'm the founder of HashiCorp.

Waypoint is our 2nd day HashiConf announcement and I'm excited to share and talk about it! Compared to Boundary, Waypoint is definitely weirder, it's trying to do things differently. I'll be around here to answer any questions.

I think the most common question will be what is this and why? I cover that in detail on the intro page here so I recommend checking that out: https://www.waypointproject.io/docs/intro

Here are some major things Waypoint is trying to do:

* Make it easier to just deploy. If you're bringing a Ruby app to Kubernetes for example, Waypoint uses buildpacks and an opinionated approach to deploying that app for you to Kubernetes. You don't need to write Dockerfiles or any YAML. You can `waypoint up` and go. If you have existing workflows already, you can use a plugin that just shells out to `kubectl`. The important thing here is you get a common workflow "waypoint up" and your future apps should be much easier to deploy.

* Provide common debugging tools for deployments. Waypoint gives you `waypoint exec`, `waypoint logs`, etc. and these work on any platform Waypoint supports. So while K8S for example may provide similar functionality, you can now get identical behavior across K8S and serverless, or your VMs, etc.

* Build a common workflow that we can plugin other tools around. This is similar to Terraform circa 2015. There wasn't a consistent way then to think about "infrastructure management" outside of a single target platform's tools. With Waypoint, we're trying to do something similar but for the application deployment lifecycle.

As always, a disclaimer that Waypoint is a 0.1 and there is a lot we want to do! We have an exciting roadmap[1] that includes promotion, config syncing with KV stores, Vault and others, service brokering for databases, etc.

And also, lots of jokes about Otto in the comments here. :) I think Waypoint and Otto similarities end at "they both deploy" (and Otto with HEAVY quotes around "deploy"). They're totally different tools, one didn't inspire the other, though we did make some major changes to avoid Waypoint hitting the same pitfalls as Otto.

Super excited to share Waypoint today!

[1]: https://www.waypointproject.io/docs/roadmap

Hi Mitchell,

Could you please elaborate further on how you see Waypoint integrating with CI systems?

It seems like you expect developers using Waypoint to deploy straight into production from their laptops, which is an anti-pattern, particularly for any company larger than a few people. Companies really need to make sure that code is checked into version control and that tests are run before code is deployed into production.

How do you see Waypoint being used in companies where there are more structured pipelines to build code?

We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution

Generally speaking I like to describe Waypoint in CI similarly to Terraform for infra. Prior to Terraform, all CI environments had this messy step that was like "create/manage infra" and it was mostly homegrown scripts that are cloud-specific.

Today, CI systems often have "deploy" which are similarly: home-grown logic that is platform-specific (kubectl for K8S, packer/terraform for VMs, etc.). We believe the "deploy" step in CIs can be replaced with "waypoint up" or "waypoint deploy".

The link above shows you how to do this.

I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak.

However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong.

I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists.

If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?

> I'm not sure how much that's the case with application development though.

Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated.

And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible.

I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.

Hey Mitchell,

It's an interesting idea, but as a Devops Lead, I'm not sure this is any where near our biggest pain point. We've been in the midst of a Devops transition from manually managed EC2 and chef towards CI/CD and infrastructure as code for two years. Terraform solves one of our biggest problems - consistently representing infrastructure as code. But Waypoint won't solve the other. In fact, Waypoint doesn't seem to touch on anything that's actually much of a problem.

The deployment step isn't the problem - it's everything surrounding it. See, we're not on CI or CD yet. We're working our way towards them from a legacy system. In that intervening time, what we need is a generalized build tool that provides the ability to do CI or CD, but also allows you to just write general build and deployment pipelines that allow you to gradually automate that process and run them manually outside the context of an automatically triggered pipeline. We've been in this stage for two years, and the only tool that actually does this coherently is Jenkins. But Jenkins is an absolute nightmare to work with on many levels. Our only other real alternative would be to cobble together a mess of bash/python/nodejs scripts or use chat ops. There isn't a good solution in this space.

Gaia Pipeline, an open source project in development by a small team lead by a Hashicorp engineer, looks like it could be our long wished for Jenkins killer. I'd really love to hear your thoughts on whether Hashicorp might be willing to throw its full weight behind it: https://github.com/gaia-pipeline

Depending on where you are in the transition, Garden (https://docs.garden.io/basics/how-garden-works) might be a good fit. It definitely touches on all of the surrounding problems such as managing dependencies, integration tests, and running arbitrary tasks (think DB migrations)

You define one part of your system at a time and Garden compiles that into a graph of modules that it knows how to build, test, and deploy. This means you can adopt it piecemeal and use it to run an entire CI pipeline from any given context, e.g. your local machine, VM or actual CI for that matter.

Full disclosure, I'm affiliated with project but your comment describes a problem we see from a lot our users so I couldn't help but replying.

You can check out Drone.io. It's based around plugins, which are basically Docker containers which accept env variables for configuration and a volume with the code (which Drone clones from git). You can easily write custom plugins in any language you want, test them, and run them locally outside of Drone with little configuration.

Totally agreed ! I still haven’t found something other than jenkins that would work for our use case which is very similar to what you stated.

I'm not Mitchell but I think this page already answers your question:

https://www.waypointproject.io/docs/automating-execution

So basically you can integrate waypoint to your existing CI/CD solution.

Kind of using it as a Makefile for pipelines, if I understand correctly.

Ooops, burned :P

Hi Mitchell!

A problem I had with Gitlab Auto CI/CD was that even though it fully abstracted away Kubernetes and Helm, when something went wrong I was suddenly confronted with having to learn and understand all of it.

Then when I fully understood it, writing my own simple .gitlab-ci.yml was orders of magnitude easier and simpler than using the big abstracted auto ci/cd system.

Waypoint seems to be similarly abstracting away a lot of complicated details. Is there a philosophy about what waypoint expects of its users, especially when deployments break out of the cookie cutter standard situations?

The level of Waypoint abstraction is the same as Terraform. You are not expected to use Waypoint and not understand the underlying platform. This is similar to Terraform: you still have to know something like AWS well to use Terraform.

With Waypoint, we aren't doing any magic, you specifically opt into a Kubernetes plugin for example. If something goes wrong, you'll have to understand Kubernetes. Waypoint isn't trying to replace that, we're just trying to provide a common workflow on top of these platforms (same as Terraform with infra).

Product Leader from GitLab here - In support of what Mitchell said. One lesson we learned from Auto DevOps is that composability and transparency were key. Just like you mentioned tinco - users struggled when things eventually broke - or they wanted to customize beyond the out of the box customizations that were available. That's why we evolved to have composable Auto DevOps[1] and Helm installs via CI/CD pipelines.[2]

[1]https://docs.gitlab.com/ee/topics/autodevops/customize.html#... [2]https://docs.gitlab.com/ee/user/clusters/applications.html#i...

Hey, yeah sorry I didn't mention that. We were a pretty early adopter I guess, we've been on it for well over a year now and it's evolved a lot since then. That it's now split up in separate modules makes it much easier to work with. Something that I couldn't find back then was tutorials or blog posts on how to really work with it, not just deploy happy case apps. The product wasn't very popular yet so there was really no community producing that kind of blog post.

Also that the Gitlab issues have now merged into one repository has made reporting an issue so much clearer. You guys are constantly improving at such a rapid pace, definitely kudos for that.

I'm quite curious about the URL service when deploying the server onto Kubernetes. How is the public `waypoint.run` able to access deployments in a Kubernetes cluster? I know it uses Horizon. But, are the requests somehow proxied through the deployed Waypoint server?

We talk a bit about how this works here: https://www.waypointproject.io/docs/url

Our entrypoint reaches back out to the URL service. This feature is optional. That’s how it works. On our roadmap page you can see that we are planning various improvements to continue to make this more secure as well.

Is the source for waypoint.run open source? Can we run our own?

I've not had a chance to dig for the source and it wasn't obvious when I skimmed the project.

Yes it is: https://github.com/hashicorp/horizon and https://github.com/hashicorp/waypoint-hzn

Not we haven’t tried to make that easier to self-run, but we didn’t purposely make it difficult either. It just wasn’t a priority for an initial launch. We’ll continue to improve this going forward.

Could you be more transparent with the roadmap for Waypoint and Boundary?

I couldn't find a roadmap anywhere. I'd like to know a rough timeline if I am to consider these even for non-prod for the projects that I do.

I understand these are both v0.1. I'm just humbly asking for some transparency in regards to the future.

Here you go: https://www.waypointproject.io/docs/roadmap

And boundary: https://www.boundaryproject.io/docs/roadmap

Maybe I'm an idiot but there's no timeline anywhere in those links.

That's likely intentional? My employer doesn't give timelines either because of how hard it is to predict software development times.

Yes this is intentional. There are a couple reasons:

(1) It really depends on how much community feedback we get. If we’re planning feature A but suddenly loads of people want feature B, we’re gonna reprioritize and change timelines.

(2) People get really, really, really upset when you don’t hit your timeline. Its sort of a lose/lose for us. So, we use keep them close to heart.

Curious if you are familiar with Habitat (https://community.chef.io/products/chef-habitat/) and how this compares?

great work! just curious - could you describe your product development timeline for Waypoint at a high level?

how you came up with the idea, how you iterated on it, and how you decided what features to include and not include in the initial release

is there a complex build example, say non-trivial java (e.g spring,jetty,etc) or rust?

I work at Product Mgmt at HashiCorp. There is a Java Spring example here that uses buildpacks from either Heroku, Cloud Foundry Paketo (VMware Tanzu team that works closely with Spring), and GCP Buildpacks.

https://github.com/hashicorp/waypoint-examples/tree/main/doc...

You don't have to use buildpacks, you could use the Docker plugin with a Dockerfile as in this reactjs example: https://github.com/hashicorp/waypoint-examples/tree/main/doc...

I think these are still trivial though in that they don't use any external resources like a database. Are there no examples that do that?

Reply on news.ycombinator.com