rubenfiszel

We have worker groups that would allow you to do exactly this: https://docs.windmill.dev/docs/core_concepts/worker_groups

And you would do exactly what you suggest in the last paragraph

Figma :)

You would run your CI in github like you would for normal codebase, the draft aspect is something that is only for the frontend editor. Every time you push from the the CLI (which you could do upon merge to main), it is akin to do a deployment of the new versions.

Webhooks require to be authentified through tokens passed as bearer tokens or query args. This is because we have a tight permissioning system around visibility of scripts/variables/resources/flows, etc.... But nothing prevent you to add a proxy on top that call every webhook with a sufficiently permissioned token.

We do automatic parsing of the parameters in typescript/python/go/bash, we infer your dependencies automatically.

And we have workflows and a retool alternative/app builder baked-in. And granular permissions and a lot of other things... The UI of script-server looks a bit bare.

But like with a lot of those tools, if you're happy with script-server then that's probably the most important!

We provide a generous free-tier of 1000 executions a month and as a very small team I'd rather rely on github/google bot abuse prevention than reinvent the wheel there. Email verification is not strong enough.

Yes, you can do both. We actually have a guide about keycloak: https://docs.windmill.dev/docs/misc/setup_oauth#keycloak

For the second part, you can just iframe the whole public app and we will provide a React component soon.

Yes: https://github.com/windmill-labs/windmill-sync-example

Makes sense. I understand the synchronous part now. I think there are ways to achieve the same benefits with an asynchronous system but if what you have works, no point changing it!

1. Yes there are no time limits for workflows and there is no cost to keep a workflow suspended. It will just be scheduled to be reactivated in the future

2. The webhook can contain a payload, and you can but the suspend step into a for-loop. You could literally say, for-loop forever the suspend until the result of the suspend step (which can be the result of the evaluation of the webhook payload by your rule is true). For that, you will need to use another primitive that we expose which is "Break" that will break out of the for-loop

3. I'm a bit unsure of the term synchronous call. We have a long list of APIs https://app.windmill.dev/openapi.html#/ and you fetch the status of the flow or trigger a flow in a synchronous http webhook to wait for the result. As to force the state of the workflow. We do not expose something like that but it could be added easily.

Happy to chat if we could be a good fit for your needs: ruben@windmill.dev

That's because you're looking at the TaskFlow syntax and a trivial example of passing all the results from a Task A as all the inputs of a Task B that follows it immediately. TaskFlow works mostly in trivial case. The true syntax for it is in the tab "Traditional syntax". For a complex graph where you're passing data from step A to a node that is far down the tree and deeply nested and you will see that it's quite cumbersome. Even in their trivial example:

    res = requests.get(url)
    ti.xcom_push(key="cat_fact", value=json.loads(res.text)["fact"])

    ...
  
    cat_fact = ti.xcom_pull(key="cat_fact", task_ids="get_a_cat_fact")

You're absolutely right. We're pretty weak in docs for developers that only want to use the CLI and the reason why I wanted to wait for the V2 is that we wanted to do a full cycle on showing the potential of the CLI for local dev and airplane-like kind of development.

For the temporal thing, that's absolutely fair and completely understand the worry but since most workflows are about the steps and not the glue between them, you will be able to reuse most of your code. However, temporal is quite an heavy format for writing workflows so of course it will still be a bit of work.

Our views support our retool like builder but also full react apps, or svelte, or vue, or anything that can compile to an iife. The example provided above can be iterated on with just a simple 'npm run dev': https://github.com/windmill-labs/windmill-react-template

Sorry that's obsolete, we will remove it from the Readme!

The updated paragraph:

> In addition, a commercial license grants you a dedicated engineer to transition your current infrastructure to Windmill, support with tight SLA, and our global cache sync for high-performance/no dependency cache miss of cluster from 10+ nodes to 200+ nodes.

Windmill Hub is not open-source, not because we want to keep it private but because I wrote it a bit quickly using svelte kit and it's not ready for the world to see yet. It does not use windmill to operate but every actions trigger a windmill webhook and alert the discord.

I like to think that we target hybrid teams made of low-code/semi-technical people AND engineers that expect to have the same characteristics seen in temporal/airflow. Abstraction without compromise, or zero-cost abstractions as the rust community like to say

Passing information between steps is a pain in Airflow (https://docs.astronomer.io/learn/airflow-passing-data-betwee...) whereas it's quite trivial in windmill, the performances/latency is better, and it supports typescript. Overall, the biggest improvement is the support for typescript/go/bash and the low-code builder that will allow you to build extremely complex flows that would be hard to visualize with just code annotations but with exactly the same featureset and more (retries, error handlers, suspend/sleep). We also make it really easy to test individual steps and the whole flow since web apps allow you to build advanced UX fit for big graphs. But Airflow is a great framework that we took a lot of inspiration from.

It's similar to airflow/prefect/dragster in that it can run workflows, and those workflows can do ETL and ops and can run arbitrary code.

It's different in those fashions. Windmill can run python but also typescript, go and bash. Airflow/Prefect etc uses code annotation which mean the code is actually specific to the platform. We actually have the steps be directly scripts/code with a main function, and then have a low-code builder to compose them into powerful workflows. That low-code builder produces a yaml like that one: https://github.com/windmill-labs/windmill-sync-example/blob/... which you could theoretically write by hand.

The low-code builder is intuitive and similar to n8n except it has the feature-set of the above framework. It's easy to test the individual steps or to test the whole flow. The workflow engine is very efficient and running 50 steps will essentially have about 1s overhead total, and the steps are executed bare so it will take as much time as if you were running them without windmill on your node directly. They can be hardware accelerated easily in that fashion. We also support some features borrowed from temporal like the ability to suspend at no-cost flows until they are resumed by an external event such as an approval from slack.

I'm very tempted and agree that Zulip sounds much better than Discord. The only thing I'm a little bit worried about is that I have noticed that most people hate having multiple chat clients open and that discord seem to be the de factor standards for open source projects. I apologize to be contributing to the network effect and will strongly consider switching to Zulip.

I love open-source and wanted it to be open-source but given that's our whole product and not just a part of a bigger ecosystem, and given that we intend to commercialize mostly the large enterprise audience, we needed to have a reason for the legal departments of those enterprises to push to buy our commercial license. So otherwise said, we do not want to employ dark patterns and want it to be as easy as possible to self-host but also need to be sustainable as a company. Most features that you would expect to be premium plugins are baked-in and open-source (sso, audit logs, smart autocompletion on the webeditor, etc etc)

We have someone from our team working on exactly this. We will release that feature in the following months.

Thanks a lot for that contribution :)

1. yes (that's exactly what the run page of each execution is)

2. yes (our flows have optional error handlers)

3. yes (TIMEOUT is an env variable of the workers)

Yes, we expose webhooks for all scripts and flows so you can integrate it for anything that can send webhooks, including Gitlab, and our workflows can be used as CI/CD.

We could but decided against it because we have no way to run those efficiently given our architecture and prefer to support a few languages and ensure we can run them blazing fast than all languages but with no guarantees of performance. In windmill, you have the guarantee to have no cold-start thanks to leveraging the great design of the deno runtime. For python, we had to do a LOT of engineering to make it work well and did not have the bandwidth to do the same for all languages. Ultimately, we could decide that some steps could be slow and it doesn't matter but it wasn't requested enough to prioritize it.

To be clear, the workers can run in docker, but do not run docker jobs.

In opposite order

2) yes, https://docs.windmill.dev/docs/advanced/local_development

1) since you can run them locally, you can also run tests as part of your CI. You can also deploy a full windmill as part of your CI, deploy to that local instance, run some e2e tests by calling the webhooks, and then only deploy once those work.

Note that we have a full drafting system baked in into the webeditor as well: https://docs.windmill.dev/docs/core_concepts/draft_and_deplo... and have support for folders with different permissions and you can create workspaces so you may implement dev/staging/prod in any fashion you like (different folder, workspace, instances)

We will add powershell support soon (within the next month), it's very easy to add given our current architecture and has been requested a few times.

No, just don't modify the software itself without making your changes open-sources and you will be good to go. Having scripts on the platform is not modifying the software.

Our users requested it, so we build it. We use scripts where we could just say "code" or "serverless functions".

Hello, founder here. Thank you for submitting Windmill. We're really close from hitting v2 (adding multiplayer with yjs is our last milestone) and would have waited a bit before submitting but here we go.

We are fully open-source (AGPLv3) with just one ee plugin for syncing the cache of the workers at large scale (more on that below). You can deploy it on a small ec2 instance with our docker-compose or on very large clusters with our helm charts: https://docs.windmill.dev/docs/advanced/self_host

We are an onion of 3 layers, which are usually separated verticals and the closest services we replace in each vertical are: Lambda + n8n/temporal + Retool:

- Workers implemented in Rust that can run any scripts from source in python/typescript(deno)/go/bash. They are extremely efficient, with most typescript scripts being able to run in 40ms e2e. This can be seen as a self-hosted lambda except it works very differently. Lambdas spawn microvms using firecracker and launch http servers within those micro-vms. We run the scripts bare (with isolation using nsjail on our multi-tenant instance) but cache very aggressively the dependencies. For instance, in python we analyze the dependencies using the AST: https://github.com/windmill-labs/windmill/blob/main/backend/..., infer the dependencies from there, have them go through pip-compile to generate a lockfile (and cache the result), then create a pathset similar to how virtualenv behave with the version of each dependency locked. For typescript, we leverage the very well made immutable deno cache. The cache works well but if you scale your cluster to more than 10 workers, the likelihood of having your worker see dependencies for the first time increase, which is why we reimplemented a pip scheme backed by s3 that is extremely fast (we do not compress, just tar individual wheels and s3 is very fast on local networks). For all the supported languages, we parse the signatures of the main function to infer the corresponding json schema of the payload to trigger them, and from that jsonschema we can then generate automatically the UI and infer the input shape required in the flows for that particular step.

- A workflow engine re-implemented from scratch which can be seen as a distributed FSM and whose state is stored in postgresql and every transition done through postgres transactions. Every worker is picking a job and progressing the fsm at completion. The full spec is at: https://docs.windmill.dev/docs/openflow. We support branches, for loop, suspend/sleep, approval steps, sharing the same folder between steps and everything you would expect from a temporal like engine (retries, error handlers).

- A dashboard/app builder that is very similar to Retool but much faster because it's implemented in Svelte and which can also run python scripts directly leveraging the 1st layer. We also support full react apps: https://github.com/windmill-labs/windmill-react-template

We have a CLI that allows you to sync from github directly: https://github.com/windmill-labs/windmill-sync-example or your local filesystem. Our scripts are absolutely normal scripts so there is no lock-in, and they are executable locally https://docs.windmill.dev/docs/advanced/local_development so you do not have to use our webeditor if you do not want to but our editor supports pyright and deno lsp through our dedicated backend communicating to monaco using websockets.

We also support worker groups to run some jobs on hardware accelerated machines (GPU) and a lot of other features (oauth syncing of the resources, schedules, granular permissioning, groups and folders).

Every script/flow deployed has a dedicated webhook endpoint to run synchronously and asynchronously. Scripts are never overwritten, they keep a lineage of hash for each version and have per-hash dedicated webhook to ensure their behavior never change.

We also have a hub for sharing re-usable scripts https://hub.windmill.dev to be used in flows to have some of the same convenience you can find in Pipedream or Zapier.

Our closest alternative would be Airplane.dev, but they're not open-source, not self-hostable air-gapped, less performant, less-featured and much more expensive. You can self-host us fully free.

We are used in production at scale in clusters of 50 nodes+ and thanks to the reliability of Rust and Postgresql have had no downtime to report.

I'm a solo founder, YC S22. The MVP got some attention at the very beginning: https://news.ycombinator.com/item?id=31272793 but was very bare. Now we're ready to bring this to the next level and enable all engineers to spend more time building and less time reinventing the wheel thanks to it :)

How do you define open-core ? We and I suppose it's the same for supertokens literally only have features in ee that large enterprise would care about like scaling to hundreds of workers. Would you prefer that we never develop those features ? What difference does it make if we develop those plugins in the main repo versus another one ? Would you prefer that those features to be hidden from sight so you cannot even read the source ?

I might be wrong but open-core is for me products that expose the minimal amount of features as open-source but are only really usable with the proprietary glue. That is not the case for us.