Hacker News Reader

Comment by Ralfp | Hacker News Reader

Just a heads up that currently Django is not cleaning up open PostgreSQL connections when ran in ASGI mode, leading to too many open connections error: https://code.djangoproject.com/ticket/33497

View on HN

I'll have zero trust running async Django until all sync_to_async and async_to_sync calls are removed from the code base.

Care to elaborate?

Look at the intended semantics [1], and then read the implementation [2]. Can you figure out if the implementation is correct? Can you infer the possible limitations of the approach at glance? Can your async library actually handle being called with multiple event loops installed?

I have zero trust in this code and I have been bitten by fixes to this library that introduced deadlocks in my own code.

[1] https://github.com/django/asgiref#synchronous-code--threads.

[2] https://github.com/django/asgiref/blob/main/asgiref/sync.py#...

Doesn't look like something I'd want to have to debug in a pinch, that's for sure.

Is there anything else in Python-land you would go with for the use case you have in mind? Or does pretty much all the async stuff turn on you like this, at some point or another?

If you need to buy into the async ecosystem for some special reason my recomendation is to:

- pick a mature ASGI server

- pick a framework and libraries that have been designed from the start with async in mind. Be suspicious of libraries that support sync and async APIs without clear guard rails between the too implementations

- avoid setups where you have multiple event loops in a single process.

Thanks - so is there anything in the Python space that ticks those boxes, in your view?

Same here, but without these weird utils it doesn't get any better.

I have 7 YoE with Django. Its great at so many things. You see some code, like middlewares, and immediately understand what's going on.

Now, we also have Starlette. The base of all new, fancy asgi libraries. Here's the base middleware class.

https://github.com/encode/starlette/blob/8d7a1cacfb3e1a30cbb...

In the last couple of years I heard 'we're running fastapi on production. Wanna join us?' so many times... but the reality is that it's still not suitable for prod. Who wants to work with a code like that if you have a readable, stable Django? I'm clueless.

Reading that bug report it looks like Django hasn't been isolated as the source of the problem? Looks like Carlton Gibson closed it until it is reproducible. Problem could be in uvicorn, psycopg2, etc

It was. Its an implementation change made in Django v4 that caused this.

Yeah, I've definitely seen odd things in Django ASGI. Had at least one instance where it looked like it leaked data between requests but couldn't get a clear enough reproduce. Switching back to WSGI worked fine though

Running Postgres without a connection bouncer is a huuuge no-no already, but if this breaks the default mode of pgbouncer it’s basically a showstopper.

Your average django app running on one or two servers with four workers each (which use connection polling) does not need pgbouncer.

that’s kind of a dealbreaker

could this be addressed by something like pgbouncer or another connection pooler?

Perhaps yes, would be worthwhile to benchmark an internal (to your runtime) conn pooler versus external only.

I'm using internal pools + pgbouncer but as I write this out I am beginning to wonder if that is even necessary.

Going off the pooling mode feature map[0], it might be possible to mitigate that issue using transaction pooling, but not session pooling — which means a restricted feature set and cooperation from the application (which would just not call restricted features up).

  [0] http://www.pgbouncer.org/features.html#fn:2

Reply on news.ycombinator.com