Show HN: Gaia – Build pipelines in any programming language
gaia-pipeline.io
gaia-pipeline.io
Also, very minor nitpick, but your programming language icons shouldn't be in <a> tags. They appear to be clickable, but do nothing, which gives the appearance that the site is not working as intended.
I think we have answered those questions on our github readme page (https://github.com/gaia-pipeline/gaia) but somehow missed that on our webpage. We will work on that!
Edit: Oh and thanks. I will fix the links of the language icons. :-)
Every web page I've ever seen that looks like yours has had the same problem.
And after looking at the gif, i resolved myself to take a look in a few weeks for a personal project.
I'm however missing an excerpt about the developer/-s of this project.
Who is building it and why? how likely is development to be abandoned? is there paid support available or planned in the short term?
from what i could tell from the contributors its pretty much a solo endeavour from you, with some help from @skarlso in June/July. Are you planning to create a GmbH and sell this as a product, or is this just a personal project you're doing on the side?
>What is a pipeline? A pipeline is a real application with at least one function (we call it Job). Every programming language can be used as long as gRPC is supported. We offer SDKs to support the development.
What is this and how would I use it? More importantly, why do I want to use it? Every application is "real", but it's not clear what the function takes as input and outputs, or how it's dispatched, or... well, anything.
The individual steps are jobs and the whole thing is a pipeline. Since it executes arbitrary code, it has the flexibility to do anything from ETL to CI.
I’m curious what differences/trade offs there are in Gaia vs something like Argo (https://github.com/argoproj/argo) or Buildkite (https://buildkite.com/). It looks like at least one difference is an actual API for steps rather than bash commands. Is there anything interesting in terms of cross job caching (e.g. saving npm install data) or persistent runners? And obviously - what else am I not thinking about?
Buildkite controller and UI is closed source.
Argo is pretty cool but I think it's different. Imagine developing an automation workflow with 20-30 different steps. In Argo you basically have to develop 20-30 different applications, compile them, build a separate docker image and push all these images to a registry.
In my opinion that's too much overhead and a configuration management nightmare as well. Additionally, what if one step needs to share information with another one? What if when this information is not trivial like a binary for example?
Also, make the page display something without 2 different Javascript sources having to be enabled.
To answer your question, a pipeline is a real application which consists of at least one function (we call it job). You can compare it with a Jenkins Pipeline (https://jenkins.io/solutions/pipeline/). Therefore, a pipeline is one flow of automation task which consists of one or more steps.
So.... every application is a pipeline? This really clears very little up for someone not already familiar with the concept. The Jenkins link is about testing and deploying code. Is that the intended use case for this tool as well?
Edit: gRPC for events is dope too.
We are already working on "agents/build slaves" (we call them worker): https://github.com/gaia-pipeline/gaia/issues/107
We have already defined a "Q&A" section in our github README file (https://github.com/gaia-pipeline/gaia#questions-and-answers-...). Does this briefly answer your question or is still too vague?
To quote the github readme: "any programming language as long as gRPC is supported"
You can use gaia without an official sdk. You just need to handle the gRPC integration yourself.
From looking through the front page of the site, I see a bunch of code examples, but no explanation of how this differs from using Unix with a bunch of programs that each read from stdin and write to stdout.
I'm sure there's actually something useful about this product, but the website really doesn't give any indication of what that useful thing is.
> integration of services is hard because not everything uses grpc.
> this assumes everything uses grpc
https://github.com/gaia-pipeline/gaia#motivation
https://github.com/gaia-pipeline/gaia#how-does-it-work
On the main website.
While being able to write in any language gives you flexibility, I do wonder about maintainability down the line. How would this be addressed? Or it doesn't matter as this sort of code tends to be thrown away?
Meanwhile if I give you a 500 line python function you have a lot of tools at your disposal to refactor to improve maintainability (you could actually write basic tests for your CI pipeline config instead of having failures when the config gets pushed out)
I usually compare Gaia with tools like Jenkins and Spinnaker. For example, many people use Jenkins Pipeline (https://jenkins.io/solutions/pipeline/) which allows you to write CI/CD tasks in Groovy. In my opinion, Gaia fulfills this job way better because it doesn't force you to use a specific language. It's also super fast and provides features like the automatic (re)build of your pipelines.
CI typically seems much more dependent on bash/unixy sorts of tools to get things done. This seems to not really support that workflow, requiring code to define pipelines instead.
If it's intended to do CI, how do you deal with CI-style tasks, like shuffling around files between pipeline steps? Or the corollary, what does this do that makes it easier to do CI in practice than with typical unix command based workflows? Inherently, it seems like "Create a golang script that can start a subprocess that runs a test suite" is more overhead than "run a test suite".
At first glance, it looks like more of a competitor to, say, AWS step functions, but it doesn't sound like that's what you're targeting.
Have a look at the Kubernetes deployment tutorial, this might clear things up for you: https://medium.com/@michelvocks/automatic-kubernetes-deploym...
In my opinion Gaia is perfect for programmers which are "forced" to write automation tasks. It allows you to write automation tasks in your preferred programming language and makes it super easy for you to schedule them because Gaia comes with an automatic build feature (just provide the git url of your source code and Gaia does the rest). Additionally, Gaia is super fast, lightweight and open-source.
It looks like the project has progressed quite a bit!
In my opinion it's comparable with airflow except it's more lightweight and supports basically any programming language and not only python.
perhaps a cultural thing but I find people gushing on themselves suspicious and distasteful.
yet another but what about X
In my opinion Gaia is perfect for programmers which are "forced" to write automation tasks. It allows you to write automation tasks in your preferred programming language and makes it super easy for you to schedule them because Gaia comes with an automatic build feature (just provide the git url of your source code and Gaia does the rest). Additionally, Gaia is super fast, lightweight and open-source.
In my opinion Gaia is perfect for programmers which are "forced" to write automation tasks. It allows you to write automation tasks in your preferred programming language and makes it super easy for you to schedule them because Gaia comes with an automatic build feature (just provide the git url of your source code and Gaia does the rest). Additionally, Gaia is super fast, lightweight and open-source.