tonyhb

You might want to look at https://www.inngest.com for that. Disclaimer: I'm a cofounder. We released event-driven step functions about 20 months ago.

You can apply on Ashby here - https://jobs.ashbyhq.com/inngest (we use that as an ATS). If you want a quick chat beforehand you can reach out to me - tony [at] inngest.com

I'd like to learn more about JuiceFS, but from their architecture diagrams I'm struggling to see what benefit they provide if they're a layer over blob store systems like Ceph, MinIO, etc.

It looks as though you need to set up the underlying storage engine that has its own recordkeeping (one point of failure), then layer JuiceFS on top — another point of failure, and I don't know what it gives you.

It's good that it's fast, but if it's pointing to another blob store you'd... expect it to be fast? Is this only needed if you want properly faked file system primitives over blob stores, if you can't use blob stores directly? Definitely need to spend some time reading.

So you have to give your 3rd party providers access to _code you write_, _secrets to your own datastores_, _open firewalls to 3rd parties_, and then allow them to run any code which could access your system, hoping that they do in fact run the blob you upload?

I agree, but still find it weird — Sony use LG (or Samsung, depending on the model) panels in their TVs, so you'd assume that "going to the source" would be better.

This is my biggest gripe against Equinix. Other than that, they're 10/10.

Fair means that we should be giving users roughly the same capacity, or at least roughly the same chances to be worked on.

In the case of this queue, where each letter represents a user:

[A, Bx10000, C, D, E]

We're being unfair to C, D, and E. Realistically, while working on B's jobs we should have some mechanism to know that latency for C, D, and E are increasing and that we can start assigning them to workers.

Without that, latency for any step function you run is highly variable and impacted by other users. With multi-tenant fairness, latency is purely driven by how well we auto-scale and your own concurrency capacity.

The post here is about multi-tenant fairness in particular, so the intent is that fairness is viewed from a multi-tenant / PaaS lense.

Cant control rods can be lifted or inserted to meet demand?

In that case why not use JWTs which do this for you? In your case you end up building a subset of JWTs yourself.

Hard agree, they're a great team who have more knowledge in this space than almost anyone. The spec is great and makes a ton of sense.

You're literally describing what we've built at https://www.inngest.com/. I don't want to talk about us much in this post, but it's so relevant it's hard not to bring it up. (Huge disclaimer here, I'm the co-founder).

In this case, we give you global event streams with a durable workflow engine that any language (currently Typescript, Python, Go, Elixir) can hook into. Each step (or invocation) is backed by a lightweight queue, so queues are cheap and are basically a 1LOC wrapper around your existing code. Steps run as atomic "transactions" which must commit or be retried within a function, and are as close to exactly once as you could get.

This is great! It reminds me of a (very lightweight) Elixir specific version of what we built at https://www.inngest.com/.

That is, we both make your existing code available to serverless functions by wrapping with something that, essentially, makes the code callable via remote-RPC .

Some things to consider, which are called out in the blog post:

Often code like this runs in a series of imperative steps. Each of these steps can run in series or parallel as additional lambdas. However, there's implicit state captured in variables between steps. This means that functions become workflows. In the Inngest model, Inngest captures this state and injects it back into the function so that things are durable.

On the note of durability, these processes should also be backed by a queue. The good thing about this model is that queues are cheap. When you make queues cheap (eg. one line of code) everything becomes easy: any developer can write reliable code without worrying about infra.

Monitoring and observability, as called out, is critical. Dead letter queues suck absolute major heaving amounts of nauseous air, and being able to manage and replay failing functions or steps is critical.

A couple differences wrt. FLAME and Inngest. Inngest is queue backed, event-driven, and servable via HTTP across any language. Because Inngest backs your state externally, you can write a workflow in Elixir, rewrite it in Typescript, redeploy, and running functions live migrate across backend languages, similar to CRIU.

Being event-driven allows you to manage flow control: everything from debounce to batching to throttling to fan-out, across any runtime or language (eg. one Elixir app on Fly can send an event over to run functions on TypeScript + Lambda).

I'm excited where FLAME goes. I think there are similar goals!

The research wasn't done by them — it was paid for by the US gov't and developed at US labs. ASML were given the research knowing that it could work.

Plenty of people invest on the side, founders or not. It's not uncommon.

I don't know Python too well but I think this is covered in TFA. There's a comparison between a Result type using https://github.com/rustedpy/result and an ADT/sum type — and the sum type looks like it's way more ergonomic with Python's current typechecker.

While I like Rust — and Result works really well with `?` — it doesn't actually look like that's the best pattern for Python?

Check out https://dagger.io/. Write declarative pipelines in code, reproducibly run anywhere.

I'll throw a vote for https://reflect.app/. It has everything you need, is well polished, and Alex is a great person.

Chiming in as a founder of Inngest.

We released our TS SDK a year ago so had quite a head start in this area. The approach in their V2 release makes them more similar to us, though there are still some large differences, including the fundamental architecture, queueing technology, scale, etc. Some differences you'll notice as a developer:

- Concurrency and parallelism. We allow you to run steps in parallel, configure concurrency (per function, or with sub-groups for custom concurrency limits), and automatically fan jobs in, eg https://www.inngest.com/docs/functions/concurrency and https://www.inngest.com/docs/guides/step-parallelism.

This is a pretty big point, as you often don't want to run functions sequentially. We're also fully event-driven, allowing you to pause workflows and automatically resume when specific events are received (https://www.inngest.com/docs/reference/functions/step-wait-f...).

We also handle a lot of complexity for you that you'd have to build yourself:

- Rate limiting

- Batching, allowing one function to run from eg. 100 events, instead of 1:1 matching of event->function

- Streaming, for long-running responses

- Auto-cancellation, based off of matching events

- Branch deploys for all workflows, regardless of platform

- Debounce

- And a bunch more such as middleware, error handlers, multi-language support (including zero-downtime live-migrations of long running workflows), fully-offline local development, etc.

In general, I think we're tackling a similar problem with fundamentally different approaches in events and architecture. Trigger definitely have more in the space of integrations, while our approach is: send us anything, no matter the event or source, and we'll work with it.

While I don't know the Trigger folks I'm assuming they've seen similar problems as us and it's fun to tackle this area, so I'm looking forward to seeing how they build out their platform in the future and this area evolves :)

I did mean virtual threads, thanks! :facepalm:

It leaks in that the entire function becomes declarative, and state is colocated to a single function. You don't have multiple jobs for each attempt, passing indexes into each job queue. You don't have to worry about enqueueing each "step" as a separate job.

The code becomes "wait until this time", vs "enqueue this function with this state to run at this time via this message broker, which may enqueue other jobs in ways you can't see".

There's no real silver bullet that will let you run code in the future without specifying "when" that code should run — but workflow engines are by far the more productive of the two.

Hot take: I don't think that people should even use queues any more. It's like using raw HTTP2, or managing threads.

Use a workflow engine which abstracts queues for you so you can "just write code" without managing jobs/schedules/state/retries.

https://www.inngest.com/blog/how-durable-workflow-engines-wo...

(disclaimer: I'm biased as I'm an author of a workflow engine)

CRT TVs used a lot of energy. It’s getting better.

Java only just got green threads (goroutines), 11 years after Go 1.0 was introduced. CSP had been around for yonks.

Nothing is perfect to begin with, and I think you could probably be a bit kinder to Golang here.

To be fair this is the same as most regulatory bodies. You have specific things you must do for SOC2. There can also be known issues that you must document and, as long as they fall within acceptable risk tolerances and you have a plan to fix them within some defined time period, it can be acceptable (obvs, again, depending on the risks).

It's not perfect, but then again... nothing is. Personally I think that being honest about all risks, documenting them, and having a plan in place to fix things is a good place to be.

Of course, FAA, aviation, space, healthcare etc. should have much lower tolerance to risk, so you'd expect that the portion of these that can be ignored should be very low risk.

This is supported in Google Calendar. You set the TZ when you're creating an invite.

https://github.com/darwin67/ex_inngest. We'll officially support this soon — it's written by one of our engineering team.

The SDKs are lightweight, work in any language (we only support a few right now), and you can live-migrate long running functions across languages at any time. Long running state is automatically transferred across those SDKs.

One of our foundational aspects is to make long running state, queues, events really easy, and that means we need to make multi-cloud and multi-lang migrations easy, too.

Yeah, for sure. Individual companies run tens of millions of functions a month across anything as basic as braze-like customer lifecycle emails (https://www.inngest.com/blog/lifecycle-emails-with-resend) to running complex AI workflows at scale.

The basic premise is that you can send Inngest events (for free) then start writing functions hosted in your own API at any time, with local testing, branch deploys, etc.

RE. initial flows and diagnostics, I totally feel your pain. Part of the trouble with these systems is that they're too different from our average SDLC we have every day:

- Write regular code

- Test locally

- Then deploy

This is one of the key aspects we're fixing at https://www.inngest.com — you can write regular TS (or Go, or Elixir) code as a function, create simple workflows, test locally, and have your functions automatically triggered by events.

I still think there's room to go with improving automations here, and it's going to be an interesting few years as things develop.

Yeah. I don’t really love that companies exploited something hashicorp gave away for free, then play good when they finally have to commit to the ecosystem when hashicorp takes it away.

Where was this support when they built a business on this free work?

Not sure, but I'd guess they're using Vitesse (ie. Planetscale) which is honestly really fast and durable.