Hacker News Reader

>If I need a for-loop, I'll write my own script to generate the JSON.

That's how you end with extreme mess in logs, UI and metrics.

> But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call.

Or you can use PythonOperator with hooks, that generally integrate external APIs with Airflow connection system.

https://github.com/apache/airflow/blob/main/airflow/provider...

I think the bigger problem with Airflow's Operator concept is N*M problem of integrating multiple systems. That's how you end with GoogleCloudStorageToS3Operator and stuff like that.

What you're asking for is basically Argo Workflows, I think.

Not that I recommend it. It's quite lovely in principle, but really flawed in practice. It's YAML hell on top of Kubernetes hell (and I say that as someone who loves Kubernetes and uses it for everything every day).

Having worked with some of these tools, what I've started to wish for is a system where pipelines are written in just plain code. I'd like to run and debug my pipeline as a normal, compiled program that I can run on my own machine using the tools I already use to build software, including things like debuggers and unit testing tools. Then, when I'm really to put it into production, I want a super scalable scheduler to take my program and run it across dozens of autoscaling nodes in Kubernetes or whatever.

The only thing I've come across that uses this model is Temporal, but it's got a rather different execution model than a straightforward pipeline scheduler.

Check out Toil: https://github.com/DataBiosphere/toil

Looks Python-specific? I don't use Python.

Prefect might work for your use-case:

https://docs.prefect.io/

IMO the issue with things like Prefect (and the sinking comment suggestion of Toil) is the inextricable coupling with Python.

We use Argo at my work, because we made the “unit of work” just a container that’s run by K8s. There isn’t a Python runtime around to run this stuff, and that’s a good thing.

Same as with Pulumi, I’ve seen what devs do when you let them write arbitrary code anywhere-you end up with code that lives everywhere, confusingly cuts across different infrastructural layers with wild-abandon. It becomes difficult to on-board, difficult to maintain, the blast radius becomes unknown, and just because it’s Python, it will randomly break in 6 months because nobody pinned a transient dependency.

YAML is awful. Argo could do with some trimming in its configs, but I’ll take it, and other “constrained” alternatives over arbitrary-code-in-Python-that-controls-resources-that-run-more-arbitrary code.

I don't see the attraction of a language-specific framework. I don't use Python, and adding yet another language into the mix would just add complexity.

That's one of the things I really like about Spark (though it's probably a lot less general than what you're after - running a pipeline of steps that are Scala code rather than arbitrary commands). I can run, test, and debug my pipeline locally in my IDE where it's just plain old Scala. Then I can change one line and run my same pipeline on the company Hadoop cluster.

The flexibility of code as configuration is indeed somewhat of a footgun at times. That’s why with Orchest we went with a declarative JSON config approach.

We take inspiration from the Kubeflow project and run tasks as containers. With a GUI for editing pipelines and managing scheduled runs we come pretty close to what you’re asking for (bring an image and run a command). And it’s OSS, of course.

https://github.com/orchest/orchest

> If I need a for-loop, I'll write my own script to generate the JSON.

OK, so you are using some service that expects JSON to config. You have a script to generate the JSON. Now you gotta make sure to run the script on changes. You also need to check in the JSON into your VCS despite it being a generate artifact. This leads to the perpetual "somebody fixes an issue in the artifact, and it gets wiped after someone uses the script" issues.

What you're saying is theoretically very good! But it doesn't play well with the combo of assumptions made everywhere in hosted services and some PL infrastructure:

- every project is 1 git repository, and vice versa

- commit security is an all or nothing proposition (so it's hard to have permissions like "this file can only be edited by this user" or "this commit can be made directly to `main`, _if_ it only touches this file")

- configuration as configuration

- version numbers have to be written into code

And obviously this is a problem, because everyone keeps on inventing YAML-based programming idioms for each of their services! Because static configuration doesn't map to even the simplest of setups very well.

People come and them mention stuff like Dhall. But really people want to be able to write programs that read files, make HTTP requests, and then make decisions about what to do based on that. And things like "if this test suite fails, do this other orchestration". The ideal CI flow is not staticly definable at the start of execution!

There is a killer protocol out there, and it's ... something like Skylark, with a standard library that gives file reading, checking of statuses, probably HTTP and the like.... and an executable that allows for configuring permissions in certain ways. The bonus would be just re-doing Bazel but with less guardrails. If someone showed up with CI/CD that was more in this vein they would become the dominate CD host very quickly in my opinion.

Strongly disagree.

IMO I think GUI variables are a big weakness of Airflow.

I, simply, never want to be responsible for a typo in a prod config variable. Ever.

I'm all for version controlling every, single, variable.