Hacker News Reader

Comment by slewis | Hacker News Reader

The response here makes me think most commenters don’t have experience with this particular footgun.

To clarify: Python can gc your task before it starts, or during its execution if you don’t hold a ref to it yourself.

I can’t think of any scenario in which you’d want that behavior, and it is very surprising as a user.

Python should hold a ref to tasks until they are complete to prevent this. This also “feels right”, in that if I submit a task to a loop, I’d think the loop would have a ref to the task!

It’d be interesting to dig up the internal dev discussions to see why the “fix this” camp hasn’t won.

View on HN

I can see this behaviour being useful if you are no longer interested in the result of a "pure" task. For example imagine fetching some data via HTTP. If you no longer need to response canceling the request could make sense.

But I agree that this is unexpected and most code probably isn't ready for being cancelled at random points. (Although I guess in Python your code should be exception safe which is often similar)

If you want to avoid an expensive operation in cases where the result is no longer needed then surely you'd want to cancel the task as soon as you know that and not at some indeterminate time when the GC runs so I don't think this behavior makes sense even for that scenario.

That would make tidying up rogue tasks impossible. Of course we all like to think we do cancellation perfectly, but it's nice to know that the task scheduler has your back.

Edit: I don't quite understand why a user would expect a task to remain live _after_ the last reference to it has been dropped...

Because they have expressed the intent to run it by scheduling it on the event loop?

I don't follow the argument wrt tidying up rogue tasks. What does it mean for the task to be "rogue"? If there was some state change that made the task redundant - because it clearly wasn't when it was submitted! - then the code that makes that change, or some other code that observes it, should cancel it. If it isn't cancelled, the fact that nobody is able to observe the value that the task will yield is not sufficient to auto-cancel, as there may still be a dependency on side effects.

And, speaking of tidying up, what if the scheduled task is the one that performs some kind of cleanup?

One might argue that dropping an object should result in it being deallocated, as is normally the case with RAII languages, which in the case of a task is to be stopped and tidied up. Speaking from experience with both threads and async Rust tasks, I find the async case in which tasks are dropped when the references are dropped is much easier to work with (once I overcame my prior expectation based on threads). If you want to keep a task alive, then explicitly keep it alive by holding on to its handle.

If you have a scheduled task to clean up, then you need to manage it at whatever level that occurs. You signal to notify completion of clean-up to the top (or whatever) level. It's no different to signalling the other way to notify of shutting down.

It's worth noting that, at least in rust tokio, you can spawn a background task. Not sure if there's anything similar in python.

A possible solution might be a built-in dunder flag to explicitly tell the interpreter not to get rid of a specific object. Something like __keep_alive__ or similar.

Reply on news.ycombinator.com