Temporal Python – A durable, distributed asyncio event loop (2023)
temporal.io
temporal.io
In short we were working on automating things like onboarding a new employee, which involves creating accounts for their saas apps, buying and shipping their device, email confirmations, satisfactions surveys etc. So a workflow could last up to 3 months with some fully automated systems, and some that required integrating with people (listening to jira event to trigger things, etc).
The error handling was the thing that sold me on Temporal, because things can break just about anywhere in unpredictable ways (not just code, can be process, employee quits during the onboarding, customer is out of licenses etc), so we need everything to be robust and be fixable by a person. With homegrown queue based systems or with BPEL it can be hard handling these situations (what if you need to roll back 3 steps?). With code you can use exceptions, write unit tests etc. We use the typescript sdk, promises made it very intuitive to code even some otherwise complicated scenarios (say event listeners etc).
After a year of high-scale Temporal work, I found it was only good for low-scale work.
The onboarding and learning curve were insanely difficult and complex. Ultimately it doesn't scale as well as you think. The temporal team invented their own database to get around this limitation.
I haven't yet used it in production, but I would've expected that a system which evolved out of Uber's Cadence [0] (and which I believe is used at Uber extensively) would've scaled very well.
The Temporal team has acknowledged that Cassandra-backed Temporal hits scaling limits pretty fast.
The limitations aren't a clean "X actions/sec", they're sneakier. Because you can run X/sec for days and then the memory on the history service will spike, or any tiny slowdown in the DB will cause looping degradation. There are nasty feedback loops hidden in Temporal that turn small problems into very very large problems.
I think the core problem with Temporal is the way its sharded. This affects history service and its caches. If anything tells them to reload or restart, or that any of the nodes are unreachable, you get a retry storm on the DB.
In addition to these issues, Temporal can create feedback loops within itself. I've seen cases where it would not return to health, even with 0 workers requesting work for 10s of minutes.
We could have kept using and scaling Temporal, but it required 10-30x the resources of building something else. And it was scary to administer. You really need an entire team. You can't have somebody who isn't a dedicated engineer take on-call for it.
> See the asyncio.sleep in there? That’s no normal local-process sleep; that’s a durable timer backed by Temporal.
That's the normal asyncio.sleep. What does backed by Temporal mean? Reading further, it appears that Temporal is replacing the default asyncio event loop. I don't understand why every third party async Python library/framework feels the need to take over the default event loop instead of just building on top of it.
A workflow is the code that handles only deterministic actions and calls activities.
Activities are functions that do anything you want, typically affecting other systems with network or file calls.
A worker is the running process connected to Temporal with registered workflows & activities for it to pass work to.
I'm doing a lot of work with alert handling and provisioning systems using Temporal. Temporal in two minutes is a great video explanation: https://www.youtube.com/watch?v=f-18XztyN6c
> you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it’s a zillion times faster which means it can be used to tackle huge problems in an instant
If you can replace the thing that people use with a distributed version, then that can make it easy to write distributed code.
[0] https://www.joelonsoftware.com/2006/08/01/can-your-programmi...
ev loops instead are a mid-layer concept that sits below other infrastructure such as threads and processes. And (perhaps somewhat frustratingly) it is not too uncommon to have multiple ev loops in parallel. See for example the proxy.py project, which offers to run one async loop per process for a speedup.
As a result, there are some incentives to swap out the loop itself, e.g. for faster implementations like uvloop, because they are somewhat pluggable anyways.
Does this also mean they have preemptive multitasking like in BEAM?
FWIW, as someone who has never previously encountered Temporal, and has only a vague sense of the specific problem set it's trying to tackle and architectural approach it has taken, I find the post to be fairly impenetrable.
I'd love to read a proper introduction to Temporal by way of Python (and probably also by comparison to Celery).
Sure, I 'could just use asyncio', or somehow work out how to crowbar serial i/o into tkinters' runloop. But in the end, writing my own just made more sense, and more importantly: it works great. I can have serial i/o and UI acting independently, but coordinating through a single queue .. this works so well.
>I don't understand why every third party async Python library/framework feels the need to take over the default event loop instead of just building on top of it.
Because you don't always have what you need to get the crowbar in place, nor big enough leverage to make space for what you have to do, asynchronously, in the app.
For example, in "Implementing a Workflow" the execute_activity refers to Purchaser.purchase, which is not declared anywhere.
If the execute_activity times-out after 1 minutes, the status does not seem to be updated anywhere.
In "Running a Worker", do_purchaser is passed as an activity, without explanation. (I guess I'd need to read the fundamental Temporal docs?)
This is more of a primer on the Python part of Temporal rather than an explanation of all Temporal concepts in depth. Definitely would recommend reading the fundamental docs at https://docs.temporal.io/encyclopedia/. For more exact samples, see https://github.com/temporalio/samples-python.
The nice things is that it abstract the conditions checks on whether something is done, has succeeded or should be retried.
The bad things is that it abstract the conditions checks on whether something is done, has succeeded or should be retried.
It's nice because that's something you do again and again, and that's a lot of code. A lot of ways it can go wrong.
But it's bad because that's a huge chunk of black box magic that may execute remotely. If you need a custom or more optimized behavior anywhere in this logic, you are done for. If there is a bug/problem in this logic, it's game over. I also have to imagine debugging and error reporting is likely not super fun.
One point in particular that strikes me, is that impotence is generically guaranteed with something like "has this task executed without error last time". But usually, what I want is something much more specific, like "has that entries been updated", "has that file been created" and so on. From a bird view, it looks the same, but from a system reliability point of you, they are not at all the same.
Hard to see how they avoid duplicate results, overlapping tasks, etc.
I don't think they really can at that level of abstraction, which means you need to implement it manually.
Eventually it seems it's a huge dep to bring in for the actual practical problem is really solves well.
But I'm willing to be proven wrong on this one, because the tech is really damn cool.
I have spent a lot of time in the adjacent space of event driven systems and there, like here, it seems like some of the biggest challenge is just education.
I wouldn't say that EDA or workflow based systems are preferable to traditional API services with DBs, just that the space they occupy in the industry is so large that I think it is really really hard if you to introducing any different paradigms, even when you focus on domains where API services aren't a great fit (like here with long running, complex operations).
My point with this comment is simply that I think if you are trying to build anything that does things differently, developer education is as important or even more important than design and architecture, but often not considered because those building these systems are already so deep into it that they can't approach the problem as an outsider.
That said, I'm not sure why this post from 2023 was posted here today. There've been multiple updates to the Python SDK since this post.
well its not entirely your fault :) what practices have you adopted now that you have some experience with it?
lower down OP mentions that they got the link from a HN discussion on asyncio 2 days ago https://news.ycombinator.com/item?id=40287354 . i guess the upvotes are today's lucky 10,000 learning about it for the first time.
Also, on handling HTTP errors in activities by raising an "ApplicationError" based on the status code, might simplifies error handling but might need to see how it accounts for more complex scenarios where errors are transient or where a retry could be successful even for some client errors like rate limiting or temporary unavailability etc.
As the asyncio library itself does have a steep learning curve, integration of asyncio with workflow systems like Temporal that also uses Pythons native asynchronous features, developers should be careful about indirect or subtle bugs, especially in error handling and task management.
This is why they are activities. Their results are stored in history, the workflow remains deterministic.
> might need to see how it accounts for more complex scenarios where errors are transient or where a retry could be successful even for some client errors like rate limiting or temporary unavailability etc.
Temporal allows you to specify whether an error is retryable or not.
(disclaimer, I'm the author of the post)
add.delay(1, 2)
It's relatively small compared to the other issues but it sticks out because it's one of only two listed as "you'll have to live with it".
Temporal has its own way of doing things; there's rules about what you can and cant do in workflows, what has to live in activities, etc. Its generally quite easy to adapt existing code work with it. We use typescript.
The worst part for us has been error/anomaly handling. Workflows can sometimes hit a state where the status reads in progress and errors aren't reported anywhere except buried in the event log; which surfaces great in the UI but we still haven't figured out how to programmatically respond to this condition.
A good example is: we use a home-grown version of this [1] to proxy large payloads to S3. However, if those payloads get REALLY large, they can take some time to upload and download; and if that "some time" is longer than 5 seconds, the control plane will believe that the worker has died, it won't reschedule, and the workflow just sits in In Progress. There's always a beautiful error on the temporal dashboard, and we can manually terminate/retry, but the world just seems to die when this happens and we can't do error-level cleanup stuff like alert the user that the thing they were doing didn't finish.
Temporal is also challenging to get support for. Its new, open source, we don't pay for temporal cloud, and there's not a ton of resources or people using it. The documentation is quite bad (if you like 500,000 word pages, codegen'd library sites with no comments, and one example for each feature, you'll like their documentation). Given we run our own temporal cluster, we've also had pretty large challenges in the self-hosting world. We work through them, usually after deep-diving into the temporal server code itself, but there's startlingly little documentation on self-hosting, and even less community support.
Overall, we don't regret adopting it, but if we had a time machine we wouldn't do it again. I feel it makes a series of sacrifices in order to create a system that has extremely high standards for processing, like financial/bank/healthcare level stuff. But, not only are we not building that, but the system has never behaved in a way which makes me think I'd even want to use it if I worked in those industries. Obviously I feel like I'm the one in the wrong here, and I'm sure its just a matter of "we screwed up something somewhere", but that leads back to: bad documentation, no way to get professional support without being on their cloud, and a lack of community support.
If I’m being honest if it is a big issue to self host but it’s value to developers is obvious and apparent why not pay?
Temporal Cloud is really, really new. Like, it was in some kind of closed beta for a while, with a "contact us" form, as recently as a couple months ago? So, the main reason we don't use it is because it simply wasn't available. It looks like its more widely available now though.
(disclaimer, I'm the author of the post)
Excited to get to use it at some point.
Avoid ADF (let alone Azure) until you've got some innovation tokens to spend, or go with another provider.
workflows and activities are called remotely, and you can have an autoscaled worker pool handling these calls.
you can retry any unit easily on failure and specify the non retryable errors. What it requires in exchange is full determinism - the same input should produce the same activities in the same order, as a good starting point.
src: I'm a user since over a year ago.