rubenfiszel

Founder of Windmill here, and d-k-p answer is spot-on. If you want to manage your entire workflows as a typescript code, then trigger.dev would be preferable. Windmill supports typescript, python, go and bash, and handle all the advanced features such as retries, approval steps, pre-made integrations to hundreds of API at the workflow level in a UI-based editor. Then for the steps, themselves. you would use either your IDE and sync to Windmill, or use our web IDE.

We support both typescript and python at windmill [1]. But our model is doing low-code for the workflows, and actual scripts/code for the steps themselves so it's quite different than trigger.dev. We handle OAuth, refresh tokens and secrets being managed by the platform. It's quite different from trigger.dev considering they are doing workflows entirely in code which has some pro and cons.

[1]: https://github.com/windmill-labs/windmill

We do not provide a desktop app, but we provide the FaaS and instructions to self-host wherever: https://github.com/windmill-labs/windmill

We do this at windmill https://github.com/windmill-labs/windmill, including self-hosting instructions and pre-made integrations with stripe

We also do this at windmill: https://github.com/windmill-labs/windmill

We noticed that too and thought scripts with simple main functions and their parameters could be the universal layer for integrations, so we (Windmill) built an open-source hub for it https://hub.windmill.dev

Pipedream (which is probably Automatisch main competitor) also has a repo full of integrations: https://github.com/PipedreamHQ/pipedream. Their format is a bit more specific than Windmill.

Hopefully, we all agree on a format and differentiate ourselves on the quality of the orchestration, flow capabilities, performance etc ...

I can see where this would be inconvenient. I write the migrations first so it doesn't bother me, but for prototyping it sure seems cooler to not even require the schema to exist.

I find that the best of both worlds is to use something like sqlx for rust [1], which does compile time checking (using postgres explain analyze!) of the queries and macros for static typing, but still let you write the query raw (and can be turned off completely by removing the trailing macro `!` as an escape hatch)

We use it extensively for Windmill (e.g: [2]) and couldn't be happier.

[1]: https://github.com/launchbadge/sqlx [2]: https://github.com/windmill-labs/windmill/blob/d60a770eb710e...

I saw your post a few months ago and already thought it was awesome. Congrats on quitting your job to devote yourself to this project. I would assume that you have considered joining YC?

I'm the founder of the OSS project windmill [1] that, among others, separates the code logic from the management of secrets. We are in the same boat of being a small team doing too many features and hence have covered very lightly the secret management. I would love to see if we could write some integrations between our projects so that scripts and workflows could leverage advanced secret management.

[1]: https://github.com/windmill-labs/windmill

One thing that high-performance polars/arrow enables is to build ETL without data sharding because in most cases, all your data fits into one node. Jupyter and local scripts become sufficient and you avoid a lot of complexity.

We hesitated a lot before mentioning ETL in the OSS project windmill [1] but since we support any python including polars, passing data between steps through shared folders and since polars is blazingly fast, it is a suprisingly good system for ETL waiting for the need of using spark/materialize directly.

[1]: https://github.com/windmill-labs/windmill

Thanks, not yet but that's mostly because we don't have the throughput. If you're serious about this, hit us on our discord which you can find easily and let's make it happen!

As the founder of one of the open-source alternative [1] to Retool, I thought of doing PWA for our apps to be fit for mobile but this is next level. Congrats to the whole retool team.

I legit think with your new workflows you are gonna kill the non open-source competition (Airplane, Superblocks) and now the question remains if being an open-source alternative is strong enough of a differentiation for the rest of us. Our case is slightly different as we bet on our open runtime and advanced workflows to serve different enterprise needs but needless to say, I wish you guys were not so good.

[1]: https://github.com/windmill-labs/windmill

If I'm not mistaken, looking at the docs of: https://pypi.org/project/pip-run/ it doesn't parse the AST to infer the dependencies, you still have to declare them (either in script with # Requirements comment (which we also use as an escape hatch https://docs.windmill.dev/docs/reference#dependency-manageme...)) or externally declared.

So it's close but not exactly the same.

Indeed: https://github.com/windmill-labs/windmill/blob/3f318b4ac4170... :)

It's a topic close to my heart since that is one of the key thing that we solve in the OSS project windmill [1], letting users define simple single-file scripts in python/deno/go and handle the dependencies for them.

For deno, as the article points out, it's easy, deno has this feature baked in and we just have to deno run the scripts, similar for go where the imports are pretty much a fully qualified pointer to their source and version.

For python, we have to do a bit of machinery: we parse the AST, look at the imports, use the heuristics that most import names correspond to their pypi package (we maintain also a list of exception-mappings), pip-compile [2] all the imports, and then get a requirement file that we attach to the script, pip-install it before running the script (then we do a LOT of magic to cache the dependencies so one doesn't actually have to install them 99.9% of the time).

Note: if you're interested in a collection of single-file scripts to many APIs, that you can just copy-paste and call the main function of, that's what windmill's hub is for [3]

[1]: https://github.com/windmill-labs/windmill [2]: https://github.com/jazzband/pip-tools [3]: https://hub.windmill.dev/

https://windmill.dev also has upfront pricing

https://windmill.dev has support for SSO (GitHub/Google/Microsoft/GitLab) and any custom ones in the open source version https://github.com/windmill-labs/windmill#oauth-for-self-hos...

Take a look at https://windmill.dev

Our open-source project [1] was built with nothing but Rust + Postgres on the backend side of things and it allows us to iterate quickly and have very simple deployments for self-hosting. The queue which is the core bottleneck of our stack will eventually get replaced by redis but for a scale of less than 10k/s jobs, it scale incredibly well.

[1]: https://github.com/windmill-labs/windmill

Yes, inputs/outputs is likely the most interesting problems for our diverse specs of flows.

Because data pipeline is not the primary concerns of Windmill, we took the stance that Inputs and Output of steps were simply JSON in, JSON out. For all the languages, we simply extract the JSON object into the different parameters of the main, and then we wrap the return into the respective language native serializer for the output (e.g JSON.stringify in Typescript). Then each step can use a javascript expression executed by v8 to do some lightweight transformation between the output of any step to the input of that step.

A lot of the simplification we made is actually parsing the main function parameters into the corresponding jsonschema, supporting deeply nested objects when relevant.

That works great for automation that do not have big input/outputs, but not for data. So what we do for data is to use a folder that we symlink to be shared by all steps if a specific flag for that flow is set. It also force us to have the same worker process all the steps inside that flow when otherwise flow steps could have been processed by any workers. It is very fast since it's all local filesystem but not super scalable.

I am not pleased with that solution and believe that if we were to expand on the data problem, we would certainly rely on fast network and HDFS/Amazon EFS/etc to simply share that mounted folder across the network.

Anyway, sorry for the rambling but I do feel like we're all taking different approach to the same underlying problem of building the best abstraction for flows and believe we might learn from each other's choices.

ps: congrats Patterns on the launch, the tool look absolutely amazing.

I am working on something adjacent to this problem. We focus much less on data pipelines but on automation, but in the end also have an abstraction for flows that one can use to build data pipeline. The locking-in issue was something I thought a lot about and ended up deciding that our generic steps should just be plain code in typescript/python/go/bash, the only requirement is that those snippets code have a main function and return a result. We built the https://hub.windmill.dev where users can share their scripts directly and we have a team of moderators to approve the one to integrate directly into the main product. The goal with those snippets is that they are generic enough to be reusable outside of Windmill and they might be able to work straight out of the box for orchest for the python ones.

nb: author of https://github.com/windmill-labs/windmill

I've been following up your progress and I'm so impressed with your commitment to open-source, the swiftness of your execution and the quality of your software. This is the start of a revolution in KYC which has been plagued by closed gardens.

Windmill.dev

If you want to play with deno in the browser with LSP support and instant preview, no need to install anything.

Signup on https://app.windmill.dev -> New Script -> Next, that's it you can now play with deno and get a feel of the language and auto npm imports.

It's an interesting discussion. I of course agree with the sentiment and am convinced of the need for an open-source "operating systems" for enterprise and that's where we draw our inspiration from. However there is a big risk of both bloat and doing everything but not very well.

Foundry relies on Spark to do the actual ETL so they can focus on doing the products on top, the most interesting one is the integration with data lineage imho. But in practice, many business did not actually need big data since most of their ETL could run on top of a few non sharded sql queries on postgres. On the other hand, if you care about ETL there are a few amazing competing tools, dbt, airbyte, snowflake, the databricks platform and so on. And last, not being open-source is in my opinion a big risk for a large enterprises to bet and implement all their internal processes and golden tables on (or "ontologies" as palantir love saying). Even though Palantir would love to be product-led growth, their moat is strongly in their forward deployed engineer, half-consultant, half-software engineers that can push Foundry in big old-school companies and governments.

There is space for a new wave of less bloated, open-source tools and I for one am pretty excited about the new players in the field.

I tried quickly the app builder with Firefox and it seemed to mostly work fine. I would bet it's just that the devs use chrome and didn't have time to test extensively all their advanced features on all browsers.

fixed, thanks :)

I'm an ex-palantir (left 3y ago, things might have changed) that worked on Foundry and building Windmill [1] which is an open-source framework that would actually be closer to Foundry except we do workflows and not data pipelines.

I do not think the comparison stands between Retool, or this tool and Foundry. There is indeed a sub-product in Foundry called 'Slate' which is an UI builder but it's a small part of Foundry. Foundry is mostly about data pipelines, to do spark transforms on large ETL, and then having lots of product on top of it to make it easy to make Spark work in an enterprise environment such as a UI builder (the slate mentioned above), a graph viewer of the ETL (monocle), a report builder, RBAC, a timeseries processor, data lineage, versioning of the code, a webeditor and so on.

[1]: https://github.com/windmill-labs/windmill

Congrats on the launch! The app builder looks amazing and seems indeed very inspired by Retool.

We are building a tool in the same space with Windmill, also open-source [1]. The goal is not to hijack this thread since I do not believe we are that competitive.

We focus on workflows and more backendy/complex logic by converting automatically Python, Typescript, Go, Bash into endpoints and workflow modules that you can run at scale on workers that are deployable on one's own infra. We are currently also finishing an UI builder [2] but because it is not our prime focus, it will always be a much simpler alternative to what you and Retool are building. Hence, I see a lot of potential for integrations/collaboration for users with need of UIs similar to retool while needing more complex logic than REST or raw database queries.

In any case, congrats, I have played with the tool and you guys did an amazing job.

[1]: https://github.com/windmill-labs/windmill [2]: https://github.com/windmill-labs/windmill/pull/886

We bet big on deno with Windmill (OSS platform to turn scripts into internal UI and workflows) [1][2] and are extremely pleased. The language design choices and the performances/caching strategies are a perfect match for building a distributed workflow runtime at scale. However, the compatibility with npm was the most requested feature. Before the compatibility mode with npm was released, we were about to build a completely separate mode of execution based on node. This solves it very elegantly.

Also super excited about the lsp improvements. I believe we are the only project to have a webeditor that features lsp integration with deno in the web (to test, go on [3], New Script -> Next, it works really well). It's very pleasant to work with and make the webeditor experience match closely a local dev setup. We are super excited about the improvements and I am about to try tonight if we can have the inline hints work with monaco. As we are fully open-source, one can see how we are able to pull it off [4][5].

[1]: https://windmill.dev

[2]: https://github.com/windmill-labs/windmill

[3]: https://app.windmill.dev

[4]: https://github.com/windmill-labs/windmill/blob/main/frontend...

[5]: https://github.com/windmill-labs/windmill/blob/main/lsp/pyls...