Django 3 | Hacker News Reader

Django 3

docs.djangoproject.com

657 pointscstuder187 comments

Zulip has been powered by Django since the very early days of its development with Django 1.4, back in 2012. As a reasonably mature web application with significant scale, we're at the stage in many companies' development where one starts to rip out more and more of the web framework to optimize things or just make them work the way we want. (E.g. while I was at Dropbox in early 2016, we discovered we only had about 600 lines of code left from the original Pylons framework that actually ran).

One of the things that has been really fantastic about Django is that we're still happily using it for the vast majority of code in the project, and every time Django comes out with a new release, I read the changelog and get excited about several improvements that actually make my life better.

Overall I think we've gotten a ton of value out of Python and Django and would recommend it to anyone starting a new full-featured web application project today.

My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial.

Thanks for your work on Zulip, I tend to be logged into at least 3 instances at any given time. Love the treaded model.

Zulip is quite amazing and I really like the product, however I somewhat dislike that the installer is a complete machine take over with many hard-coded paths.

Supporting only Ubuntu releases is one thing, but the code base actively fights installing it on anything else but the supported distributions (ubuntu). At which point, why even have a installer, just ship an image.

Since Zulip has so many moving components, I would love to see it be installed with something like Nix. Doing so would allow it to be installed in a consistent way across many different OSes, and decouple your reliance on software provided by the OS. It would also make upgrades seamless.

You can install nix on any single linux distribution, not just nixos, and since your installer already hard requires running as root, you can install nix and then install zulip in nix.

We've supported Debian in production since 2018, and we're close to supporting RHEL as well. And even if we only supported Ubuntu, there's a lot of benefit to providing an installer that works on every Ubuntu LTS release; most larger organizations have additional monitoring/security/backups/etc. software they want to run on every system they manage, and to make their own decisions about when to move their organization to newer major OS releases. You can also use our Docker image if you want that, but a lot of organizations don't :).

I wouldn't say the installer is actually a complete machine takeover; we're pretty cautious about using the system versions of dependencies shipped by the OS vendor. But we do __document__ that you should treat it as one, mainly for support reasons. The support experience that we currently provide is that where possible we respond to, investigate, and debug all reports of issues installing Zulip, with the goal of making it Just Work (this includes working to give nice error messages for common user errors).

Before we added those warnings in our documentation, we got tons of reports from users who were trying to do things like install Zulip in a shared hosting environment with other software that doesn't expect to share its database (etc.) with something else, and where they don't have root access. Also, investigation often determined the system in question had low RAM/disk resource limits from the shared hosting environment that meant running Zulip on that system wasn't going to work in any case.

At one time, that sort of shared hosting setup accounted for the majority of reports from folks having issues installing Zulip. Investigating those reports was a huge waste of time for our development community. We solved that problem by making our documentation recommend installing Zulip on a dedicated system. This also provides a lot of benefit in terms of our being able to do things like tune the configuration for postgres and memcached to allocate resources in an appropriate way for a Zulip installation with the system's RAM allocation.

There is occasionally discussion in the community of adjusting our approach here, but given how easy it is to get a VM or container these days, we're currently happy with guiding users to the dedicated VM/container path that we can guarantee works ~100% of the time and provide a great installation experience.

One of my favourite things with Django has been their mature approach to changes. Want to be guaranteed a stable upgrade? Step 1: Fix deprecation notices. Step 2: Upgrade. (If you're upgrading over a few releases, return to step 1). Coming from other frameworks it's a delight :) At least that's been my experience with smaller projects. I'm curious, does that line up with your experience with Zulip, too?

Yeah, that matches my experience; their documentation on upgrades is excellent, and there's generally only issues not covered by their deprecation notices in places where you were doing something not supported by the documented API (E.g. monkey-patching, subclassing to override something, etc.), and even then, they often comment if it's a common type of modification.

I've personally never had a painless Django upgrade (except for patch releases). I always seem to fall into that grey area where I'm relying on undocumented behaviour of documented features (e.g. not using framework internals per se). I remember one Django release which changed the behaviour of views that returned files to download (it was probably Django 1.4), but there was no mention of that in the release notes.

Sometimes I've knowingly relied on framework internals in ways I expected to eventually break, but those situations don't represent the bulk of my experience with painful upgrades.

> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial.

To be fair, I think the next LTS release of Django will be the last minor release before 4.0, 3.0 is not LTS.

If you don't mind: what are the key things that make Django useful for a project like Zulip?

How much does the framework help you, and where? What are the places where you have to go against the grain, if any?

I'd say the big thing is helping us avoid reinventing the wheel. There's a lot of problems that every webapp has, and Django provides a system for solving most of them. I particularly enjoy their migrations and automated testing systems. When we need to integrate something that a lot of other people have, whether it be 2FA or SAML or LDAP or webpack, there's generally a reasonable library for it (or often, a reasonable way to do it built in, and Django's documentation is great so it's easy to figure that out).

While there are some land mines with any ORM, the Django ORM does allow us to write the vast majority of our database operations with clean Pythonic code, rather than going to raw SQL. We need SQL in a few performance-intensive code paths, but it's the exception, rather than the rule, and the ORM is constantly getting better.

And in contrast with some other ecosystems I've worked with, the libraries generally are well-written, well-tested, and stable, so even if they're not well-maintained because the old maintainer changed jobs or got busy or whatever, it doesn't feel like a burden to fork or adopt the library.

I'm not particularly happy with Django forms or Django REST framework with how they organization validation code. For the former, I think the problem is just kinda messy; we mostly solve it by only using Django forms for the authentication code paths where we can reuse its libraries. For the latter, we have a framework we built that I'm very happy with and we plan to spin out for other projects to use when we have a bit of time. There's some documentation here: https://zulip.readthedocs.io/en/latest/tutorials/writing-vie... for those curious; though that doesn't cover cool details like the mypy integration we did more recently.

Ooooh, I'd be very interested in seeing that spun out. I also find DRF borderline infuriating from a design standpoint.

Hey, if you ever want to do a podcast episode that goes into your tech stack (why you made certain choices, how it's been going, deploying, etc.) let me know.

I'd love to have you on as a guest for a podcast I recently started called Running in Production.

Dan from https://realpython.com and I just did one recently. That site runs Django too. You can check that out here: https://runninginproduction.com/podcast/4-real-python-is-one...

Click the "Become a guest" button to get started if you want.

Note: If you're reading this and you're not from Zulip and you are running Django in production (or any stack) hit me up too.

One of the things I have learned early on is to not depend on the system-bundled Python. I either just use pyenv to install whatever Python version I like, or I use a Python Docker image (https://hub.docker.com/_/python/). That way, I'm decoupled from the system-bundled Python and it's easier to test out new Python versions.

> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021 ...

Django has an LTS release (2.2) that supports Python 3.5 just fine. Doesn't it make sense to use the LTS Django on an LTS operating system?

There are plenty of ways to use newer python, too. Docker, PPAs, etc.

Don't ever rely on python that is included with OS. It makes upgrades really difficult. This is especially true with RedHat and CentOS where they are way behind. There are other issues, like dependencies, you are bound then to only using the dependencies version they provide. If there is a bug in one, tough luck, you can try to upgrade but then you're risking that you will break system tools (yum etc). Just assume that the system python is for the system.

Use the right version you need (if you use redhat/centos I highly recommend ius.us repo, they make sure that it can be installed side by side and doesn't conflict with anything in the system) then create virtualenv and install your application there.

That way you have full control of the python version you use and all of your application dependencies. Also when ops team wants you to move from version 7 to 8 of the OS, your application should work the same without no or very small changes.

And you can assume that no security issues have entered python no matter which version? People use the OS version for security audit reasons.

The ius.io repo that I mentioned patches the version of Python. What it lets you do is you can install any major version you want (in fact you can install multiple at the same time and they won't conflict).

So you can chose whether you want python 3.5.x, 3.6.x, 3.7.x, 3.8.x or maybe still you want 2.7.x.

I have been using FastAPI for the last two months (which also is an ASGI server and makes full use of annotations and type hints with mypy) and the experience has been incredible. (https://fastapi.tiangolo.com/)

If Django can now also support annotations and async code I dream of an scenario where apis can be built using this two elements.

Does anybody know a good resource to learn/catch up with this release?

I recommend the release notes for each version, starting with the one right after the last version you’ve used.

There’s a couple reasons:

1. you get a breakdown of all the new features, which only takes a few minutes to kind of quickly go through each version and decide which bits you care about

2. you get a list of backwards-incompatible changes, which resolves any upgrade regressions, as long as you’re not using internal APIs that have changed

How hard was the learning curve? What are the main benefits over Django?

Learning curve is super low. Main benefits are:

1. Performance, it's a wrapper on top of starlette/uvicorn, which brings the performance closer to nodejs (https://www.techempower.com/benchmarks/#section=data-r17&hw=...). (I did run into some issues with it though due to the default response validation when serializing large response bodies)

2. Lightweight background tasks (from starlette)

3. Documentation generation from type annotations.

It's a nice tool for microservices but coming from django you'll have to roll your own database management, authentication, sessions, caching, admin and etc. I'm also not a fan of the magic request unpacking using type annotations and prefer getting a request object as is done in django and starlette. IMHO most people would probably be better off with plain starlette and a 3 line decorator to handle request validation and response serialization.

It looks pretty similar to flask as far as syntax goes.

Why do you think plain starlette is better? Are the type annotations annoying to debug?

I had a few endpoints that had to handle url parameters, query parameters, http headers and bodies which could either be json, form data or files, handling all of those options using type annotated parameters ended up being pretty messy and I ended up just using the starlette request object.

I was trying it out for a machine learning service that had to send back large dense vectors and I hit some other performance issues (like https://github.com/tiangolo/fastapi/issues/360#issuecomment-...), due to the coupling of validation with serialization.

I wrote a package that allows use of type annotations for validation/serialization in Django Rest Framework: https://github.com/rsinger86/drf-typed-views

(Partly inspired by FastAPI)

Pyotr [0] is a small library I've been developing for a while, based on Starlette. In a nutshell, it takes an OpenAPI specification and turns it into an API application, taking care of all the routing and validation according to the spec. It is conceptually similar to connexion [1], but it supports async and is Python 3 only. There is also a client component, in the spirit of bravado [2].

[0] https://pyotr.readthedocs.io

[1] https://connexion.readthedocs.io/

[2] https://github.com/Yelp/bravado

This is awesome! I'm one of the community maintainers of connexion (I added openapi 3 support), but I don't have write access to the repo, and it's been really tough to get any changes landed lately. nice work!

Thank you! ^_^

What's the difference from fastAPI? https://fastapi.tiangolo.com/

FastAPI generates specs from code, this generates code from specs

Just a bit of clarification: Pyotr does not generate any code, it uses the spec as configuration to construct the necessary routes at the app initialization, as well as to validate requests and responses.

Yeah didn't mean literary code generation, Python's dynamic nature makes such steps unnecessary.

In a way it feels kinda lispy (in a good way) to me, data is code, small DSL for working more efficiently with specific domain.

It's worth pointing out that the ASGI support in this release is very low level, and doesn't let you write async views or anything yet. We're still working on that.

Thanks for your work on this.

If I am willing to hack a bit, is it possible?

What is the main obstacle? Is it all middleware?

Andrew gave a recent talk about it here https://www.youtube.com/watch?v=d9BAUBEyFgM

middleware is a part of it, then there's the ORM, caching and anything else that does IO.

If you’re in the hacking mode - what do you think of taking the Django orm and grafting it onto fast api - sort of like a stand-alone sqlalchemy but with all the ease and power or django’s querysets...

Django ORM is not async so using it with FastAPI would block the event loop. I guess you could wrap the calls in sync_to_async from asgiref but it wouldn't be pretty.

Another option is using something like Tom Christie's orm project (https://github.com/encode/orm), which is a wrapper on top of sqlachemy with a django like interface.

Thanks for the suggestion - very interesting.

FastAPI runs blocking IO/sync functions in a separate thread pool to work around this issue.

Thanks for the info. I'll probably start fiddling with Django again, it has sure been a while for me.

Also, thank you and all the contributors for such amazing resource.

> Note that as a side-effect of this change, Django is now aware of asynchronous event loops and will block you calling code marked as “async unsafe” - such as ORM operations - from an asynchronous context.

Am I correct to understand this as meaning async views can’t even read from the database yet? I guess the only use cases for ASGI views currently would be interacting with outside-Django backends that implement async support and such?

See also the base starlette, which is what FastAPI is built upon and is ASGI from the ground up. https://github.com/encode/starlette

id like to chime in with positive experiences with FastApi and starlette

I've been messing around with FastAPI as well and it really is such a pleasure to use. ️ Its documentation is what really makes it a stellar project for me.

I love FastAPI. We're using it for a ton of different things right now, from machine learning model containers to API layers. It's a really easy project to get started with, and the developers have been iterating on it pretty quickly.

I liked this (and other!) episodes of Django Chat, https://djangochat.com/episodes/django-30-preview

Looked at fast api docs and it looks great. Curious how to interface it with an ORM or DB without duplicating model/field definitions?

I've been using FastAPI at work for the last 8 months or so, and it's been a delight.

Coupled with Pydantic for validation it really does make things easy :D

The community built around Django, coupled with the sheer amount of documentation and the stability of their APIs has done enough to forever ruin my expectations for any other coding project out there. Ostensibly everything else feels disorganized, poorly documented or rushly released.

But the most commendable value of the DSF – which I believe is a model to be replicated – is how effectively it has managed to both attract new developers and also mentor them so well that they have gone on to become core developers who maintain the framework later on.

I'm still subscribed to the django-developers mailing list, even if I have had no free time to contribute with more than a couple patches over many years, simply because I enjoy watching how incredibly smart and efficient developers collaborate.

Kudos to all of you guys, and congrats on achieving this big milestone!

I echo this sentiment exactly. I don't feel like OSS projects achieve this kind of success by accident. I'd love to see some write-up of some kind on how they do it. They're definitely the gold standard in my mind.

That’s a damn good idea!

If you can convince your company to keep Django up-to-date with every release, I have found this to be easy and pain-free (Django has been pretty API stable since ~1.8), compared with waiting for each LTS and then being forced to jump ahead three versions each time.

You also gain access to new features as they come out, this way.

But you have to be careful that all dependencies are up-to-date as well.

No to mention each new release have many incompatibility that you need to read the release note to find out

Django is very good at being backwards compatible. Deprecated functions emit warnings for a few releases before being removed. They are also documented.

Do you have some examples? Are you sure you are not confused with dependencies?

The release notes mention less than ten deprecations related to django. Other stuff which is not supported anymore is mostly non supported database versions.

https://docs.djangoproject.com/en/3.0/releases/3.0/#backward...

The Django incompatibilities is a small problem. If you do sign-up for the theory that it's better to use its ecosystem, the changes on the related projects are much larger.

Just want to say thank you to the Django core team. I was privileged to use Django full time for over 6 years. Both the project and the community are a joy.

There's a few times I had to dive into Django's internals, and every time it was logical and straight forward to understand and modify.

Thanks guys!

Was really excited to upgrade but the async safety check makes the ORM unusable in a Jupyter Notebook.

https://stackoverflow.com/questions/59119396/how-to-use-djan...

https://forum.djangoproject.com/t/is-there-a-way-to-disable-...

please tell me that python did not adopt the javascript async hell. async should be an optional keyword on the CALLER side, not an invisible trait of the function. go figure()

It didn't, async await is just syntactic sugar on top of coroutines/futures. You can define an sync function and get a coroutine back if you call it without await.

It's still kind of a mess though because you have two universes that are not really compatible, forcing you to rewrite everything that touches IO. For django that probably means a rewrite of the ORM, caching and middlewares.

JS async/await is also just syntactic sugar on top of Promises (which are basically just futures with the ability to write as well).

In Javascript, async is an optional parameter on the caller side. (Well, 'await' is, but I figure that's what you mean). It can be used with any function call, not just those that are marked async. Though it only make sense to use it with Promise-returning functions.

Alternatively, you can also use it with a plain Promise, which makes for a nice mutex. I've also used this for imitating async constructors, which is not currently possible.

I've replied to the post on the forum, but if this is the default way Jupyter runs then we're going to have to figure something out longer-term. Calling the Django ORM from an async thread just isn't safe...

Thanks for the reply and all of the great work.

Using django in a notebook is definitely not a typical use case so I wouldn't worry about it too much.

Ideally the ORM would be a standalone package that could be used outside of the web server context. (I know sqlalchemy is an option but then you lose all of the benefits of django)

Our developers love to work in notebooks running Django, it lets you iterate extremely fast.

Andrew already has a PR to fix this issue https://github.com/django/django/pull/12172

The ORM has been essentially unusable from my perspective since day 1

Why is that?

Try doing some simple joins, subqueries, unions, etc., and you’ll quickly see why. If you’re able to find the correct incantation, you’ll quickly see that the Django ORM builds awful queries that perform horribly. I’ve found the Django ORM to make simple things hard and hard things harder.

I've been building some fun side projects with Django lately and it's really a breath of fresh air. Normal, old, boring web applications with server side rendered html and basic forms. Sure, I can't add some super fast interactivity on my forms as easily without some json endpoints, but for most of my projects it's totally fine.

https://www.aeracode.org/2018/06/04/django-async-roadmap/

https://github.com/django/deps/blob/master/accepted/0009-asy...

Groovy. I am excited to try out the new ASGI goodness. Last time I used Django Channels it was a constant struggle to tune it, prevent it from becoming deadlocked, etc. And the transition from Channels 1.x to 2.x was also not pleasant. I hope that Django proper did things better/differently.

Not Groovy, Python! :)

Do you know if this update changes how Django Channels works?

I really wanted to use Django channels for some soft real-time stuff like chat and notifications and it worked but I couldn’t get testing to work reliably (tests would just hang sometimes), so I had to use a different setup.

Well there's no actual async support for views in 3.0 - we missed the cutoff due to performance concerns! Hopefully it'll be in 3.1.

Thanks for the info & for your work!

Django and Python have made me a much better developer.

Using it for more than 10 years professionally, full time with no stop.

It helped me to pay my bills, bring food to the table and build my ideas and get one step closer to my dreams.

Thank you Django and everyone around or behind or even remotely related to it.

Can you explain lil bit more like what exactly you do with django (freelancing or job??) and your journey with django. I am sensing their is a great story which can inspire newbies like me! Thank you very much sir!

Prior to working with Django, I worked with ASP.NET/C# and later PHP with various web framework CakePHP, Zend, Symfony) and then worked Ruby on Rails for 2 years until I found Django (version 1.0.x I think) and from there in less than week, without even having a knowledge in Python, just by following tutorials and reading the source code I become productive and made production-ready apps.

Almost all of my work with Django is through full time and once a while contracting gig here and there (very short ones.)

Not just my day time job, I also implement all my side/personal/commercial projects with Django (gonevis.com is my latest project).

Django never gave me a tough time to understand anything, for almost everything there's a well-maintained library and properly tested with enough community around it, it's ORM is crazy easy to work with, optimize, tweak and change when needed.

Every new release is easy to upgrade, you may have some problem with third party libraries to catch up if something is not compatible with them but in my experience with medium-size code base (~300k LOC) the whole upgrade would take 2-3 hours and that was just changing code and running the tests, all in all, in 2-3 week most of the other third parties would have upgraded as well.

It's mature framework, doesn't get crazy with new shiny things (NoSQL, MongoDB, WebSocket, etc), it might be late to some technology but it's because it stays in the corner until all these shiny things have worked out their problem and issues, then those things will become part of the official code base, otherwise if you're in hurry, there's always another third party that bring those for you, either NoSQL, WebSocket, push notifications, OTP, etc.

For me, the most fascinating thing in Django is the ORM, Authentication, Admin, Views and template engine and the crazy support of PostgreSQL (almost the same applies to other database backends in Django), sessions, caching and every other single thing in that carefully have been layered upon each other and work in harmony. The database migration (formerly South migration then part of official) is something you can't easily find in other frameworks (Rails migration is great as well).

Now, DRF is something else, since starting to work with DRF, I've become an API monster :D

The framework that lets you implement your idea without getting in your way and allows you to bring ideas from zero to production very quickly, is the one I look forward to using (I use Java Spring once a while as well).

For me, Django and it's surrounding makes sense, it's logical and it's not magical, maybe it's just me and it may not be something tasteful for others, if you find a framework you're comfortable with it, then use it, if you get into Django, I hope you feel the same.

Congrats to Django's contributors and community! It's been a while since I've actively worked in Django but I found the people in the Django ecosystem to be generally very helpful.

Interested in reading about agencies that used to (mostly) build web applications in Rails but then switched to (mostly) Django.

I find the productivity somewhat higher from a business point of view, to turn around projects for customers when using Rails instead - largely due to the gem ecosystem and the way community libraries tend to play well with one another.

I think it really comes down to the type of project that you're working on. For data science / machine learning projects python is a much better choice and that spills over to django.

REST APIs are also incredibly easy to throw together using DRF.

I feel like you could just unload the data science + ML stuff to a separate process though.

This looks neat. Does anyone have experience with using type hints and mypy with Django? Any gotchas?

No specific gotchas with type hinting, but be aware that Django has not implemented any notion of type hints. So, you lack quite the benefit of type hinting

I'm currently experimenting with it, so I don't have a lot to say yet. But you'll want to use this: https://pypi.org/project/django-stubs/

I gave ‘django-stubs’ a spin on a Django 2.2 project. The ORM typings and mypy plug-in worked great. ‘.filter().first()’ and similar all returned the correct types.

Ran into a couple issues with the typing of ‘client.force_login()’ where the ‘user’ parameter was typed as the ‘get_django_user()’ which wasn’t the same as the custom user model I defined in the ‘models.py’. I also had issues with ‘from django.conf import settings’ not being the correct type so I’d suggest changing that to ‘Any’.

You might be able to override the few types that don’t work for your given setup or you can just fork the repo.

Overall, I think the pain is worth the benefits.

Hi, I'm a primary maintainer of 'django-stubs'. We're going to release new version with mypy==0.750 and Django 3.0 support soon. Could you create an issue for your 'client.force_login' problem?

> I also had issues with ‘from django.conf import settings’ not being the correct type so I’d suggest changing that to ‘Any’.

This one also has plugin support, and should be working mostly correct. Fill the issue if you can reproduce, I'd love to help with that.

What's your experience with that whole "plugin has to actually run django.setup() before typecheck" thing?

Painful at first, but well worth it. You can ignore all existing code and only use it going forward.

django-stubs is black magic and helps a lot.

Here's an example setup that I use for my library:

https://github.com/silviogutierrez/reactivated/

the "sample" site is 100% type checked. Look at scripts/test.sh for running mypy in ci (and .github for the CI config)

Django (and DRF if you use it), uses too much reflection, which doesn't play nice with type annotations. The strategy that works for me is to annotate my application code, and keep using dynamic typing when interacting with the framework.

Do the stubs for each not work for you?

When I tried them a year ago, they used "Any" a lot, especially around the ORM code (Django) and the serializers (DRF). Since those are also the parts where type checking would be the most interesting, I didn't find a lot of value in using them.

We (Zulip) added type hints to most of our Python Django codebase a while back. The blog post[1] below has a lot of details on this. Also included a link to the hacker news discussion of the blog post[2]. Feel free to come and ask in https://chat.zulip.org community server if you have any further queries :)

1. https://blog.zulip.org/2016/10/13/static-types-in-python-oh-...

2. https://news.ycombinator.com/item?id=12703008

There's a DEP discussion going on at https://github.com/django/deps/pull/65. Though I kind of gave up on it for now, I don't believe it'll be accepted in the current state of mypy and typing ecosystem.

Try 'django-stubs' mentioned above too https://github.com/typeddjango/django-stubs, it's mostly complete. Mypy plugin inside takes care of a lot of stuff.

Disclaimer: I'm a creator of the 'django-stubs'

Having worked with Asynchronous python since the days of twisted, tornado - I haven't really found any real reason to switch to asyncio. Generally speaking the old forms of async syntax (@coroutine, @inline_callback) make it incredibly clear that you are executing code well outside of a sequential paradigm - they force you to think a little more about what is going on.

The idea of Django throwing errors when you try and run async unsafe code within an event loop gives me cause for concern. I can just imagine folks who don't understand adding 'yield', 'await', 'yield from' until the errors go away - while being confused with the resulting execution. I think there is always a place for synchronous code in Python - I think Django should lean in to the simplicity afforded by it.

Last note - I also am very weary to mix asynchronous code with other primitives like threading, multiprocessing. If I am going the route of async programming - I usually just intend to optimize a single thread - and then I leave it up to the OS to optimize across multiple processes.

> Having worked with Asynchronous python since the days of twisted, tornado - I haven't really found any real reason to switch to asyncio.

Agree. On the other hand, Trio (https://trio.readthedocs.io/) gives me plenty of reasons to switch.

I'm a really big fan of using aiohttp + aiopg + some light wrapping for CRUD APIs

> I also am very weary to mix asynchronous code with other primitives like threading, multiprocessing. If I am going the route of async programming - I usually just intend to optimize a single thread - and then I leave it up to the OS to optimize across multiple processes.

A lot of the time you simply can't not have another thread or process, for example, when you have a blocking library call or CPU intensive task.

I learned, in this order:

bottle

flask

falcon

django

Django is incredible. It was useful to start with the minimalist frameworks to understand the true value of Django when I got there.

Django is amongst my software top 10. Maybe top 5.

Curious as to what your top 10 is.

Right as I was just about to start a new Django project...

The thought going through my mind right now is "I wonder if there is some way of creating a Django project and make it resilient to future Django updates and releases with minimum fuss?"

I've had to deal with ongoing and inherited legacy projects which run on Python 2.7, use the long-deprecated Pylons, for which there is no easy upgrade path other than a total re-write in less deprecated technologies. No practical way to convert that Pylons project to Pyramid, for example.

I guess the answer to my thought above is to "Not do/use anything /too exciting/ when creating your Django project" - i.e. only use the most basic and generic Django features, but even then what's to say those won't get deprecated in favour of some new exciting replacement?

I guess that's one of the dilemmas developers have day to day?

Well, there were a few things in the 2.x series (e.g. naming of default auth views) which ended up breaking existing code and needed updates.

But in all honesty: django is the project I would worry about the least here. Even when stuff is deprecated, an improved alternative tends to exist and is pretty well documented. The level of care that goes into each release is very high; the number of brown bag bugs that get out very low. If you want a quiet life, django is a very good fit.

> i.e. only use the most basic and generic Django features

I don't think you need to do this with Django. Features are generally very mature, stable, and are not deprecated frequently. When they are, they are deprecated years in advance with long term support still provided.

In all honesty, I've seen very few projects run feature lifecycles as well as Django. I'd use everything you need to in it without concern about this.

Seconded, I haven't really had many problems upgrading Django projects, and I've upgraded a lot of them.

Yep. I work on a codebase started with 1.4 and we're on 2.2 now, and almost all of that was straightforward. The biggest difficulty was migrating from South to Django migrations, and that was only because we did it between real releases on Django's master branch (we don't do that anymore!).

Some languages are more prone to prioritize of your previous investment. The most popular examples are Java, C# and Go. Different tools for different needs.

>Right as I was just about to start a new Django project...

It's rarely a good idea to use the latest version of Django simply because most projects will be reliant upon 3rd party libraries which will often be slow to catch up to supporting the latest version.

The difference between 2.2 and 3.0 is quite small, as you can see in the release notes. Most projects won't need changes.

> "I wonder if there is some way of creating a Django project and make it resilient to future Django updates and releases with minimum fuss?

Have lots of automated tests. The more surface area you cover, the easier it'll be to upgrade.

> [...] use the most basic and generic Django features, but even then what's to say those won't get deprecated in favour of some new exciting replacement?

In my experience, Django maintainers are slow to rush to new exciting replacements, and even slower to deprecate and remove things. From a compatibility perspective and ongoing maintenance, it's a great approach!

I feel your pain. If maintainability is your concern, I would advice to skip this release as it won't have long term support. The current LTS release is still 2.2 and will only be replaced with 3.2 sometime in 2021.

You probably already know, but the long term release timeline is shown at https://www.djangoproject.com/download/

Sticking with LTS releases really helps. You'll still have nearly a year to migrate from one LTS release to the next.

In my experience it also helps to avoid small (e.g. a single developer) and young Django apps and stick to the big ones. The amount and quality of the documentation can also be an indicator of longterm viability of the project.

I've done a few more significant migrations (including Python 2 to 3), and about the only bad things are apps whose development stopped and those with developers implementing hacks using Djangos internal API. Everything else is usually done in less than a day for the whole Django project. I only did small projects with 5-10k loc, though.

I work as a contractor and have codebases in python, f#, delphi, swift, obj-c and now rust.

> "I wonder if there is some way of creating a ???? project and make it resilient to future??? updates and releases with minimum fuss?"

- Use control version (mercurial, git, ...) in everything, including utils and side projects.

- Record all dependencies somewhere in the project.

- Automate the builds, so you also have how reconstruct the project

- Use VM or containers or virtual envs or similar

- Rely in a small "core" of solid dependencies (in my case: PostgreSQL, sqlite3, nginx, ubuntu LTS) that could cross across projects and versions.

- Reduce dependencies (but each day is HARDER and HARDER) or better, NOT USE VERY FANCY STUFF (rely on PAAS or kool-aid kind of tech.) For example, If I rely in a http client library it can be dead years after I need to upgrade. That sucks, but, a "http client library" is so "normal" to have that I can rely in found a alternative easily. However, if I base everything on top of Firebase or other cloud stuff, if it die... good luck with that!

Very interesting. I myself was dealing with a Java project I need to maintain for legacy reasons on MySql 5.1. I Just spoon up a Django project aside, and used Peewee[0] to access without changes the DB. Works like a charm and easily integrated in Django midleware.

[0]: https://github.com/coleifer/peewee/issues/1995

>NOT USE VERY FANCY STUFF

They said "with minimum fuss". Write every non-trivial functionality yourself from scratch is the opposite of minimum fuss.

When you use something fancy, and the devs stop supporting it, worst case is you have to write it from scratch yourself at that point, or hope that an alternative exists.

Your suggestion is acting like they don't exist in the first place and write stuff yourself like fancy stuff disappeared - it is a huge hassle and investment all the same.

I mean VERY fancy stuff. Things nearly "experimental" or that rely in a huge black box. Like "not chance in hell I could replicate it" instead of "very hard, but doable".

I know everything have "but..." somewhere (for example: We rely in operating system and the browser, 2 things too hard to recreate) but the idea is always about minimization.

Sometimes you NEED to rely in VERY fancy stuff... but then you are walking on luck...

Perhaps it's time to think about ditching the "batteries included" monolithic frameworks like Django (Rails etc.) and start your serverless/microservices/docker/k8s/etc journey.

The beauty of this type of architecture is that you don't need to worry about upgrading your entire stack just because your underlying framework bumped a version number. You can create a service w/ Pylons here, a service w/ Whatever there, and lock them down without needing to refactor them all just because you want to add another feature.

… and then you have to track versions in many places and test all of the various combinations yourself rather than having a third-party do it. Microservices have their advantages but they are not a magic wand for dependency management unless you’re using very little of a framework.

Edit to add: I would class an advantage of microservices if it made it easier for you to have automated tests or deployment/rollbacks, since that’s much more effective for this problem in my experience. I’ve seen people do that with monoliths and fail to do that with microservices so again, no magic wands.

No.

You push all your complexity to your infra, where you end up having an order of magnitude less visibility, way more problems debugging trivial issues, and making your life more difficult for dumb, simple problems that have been solved two decades ago.

Monolithic frameworks are amazing. They have well-designed APIs by people who don't want to waste time upgrading to the latest framework of whatever, and splitting them into actual services is equally complicated provided you've architected them well. And you haven't, it's as much of a PITA as any "serverless" thingy of the day.

I just finished rewrite of big (400+K LOC) 10 year old pylons codebase to pyramid. We were able to share our models, services and 100% template code, and about 80% of view code was untouched.

What you want to do is follow this recipe:

https://docs.pylonsproject.org/projects/pyramid-cookbook/en/...

You put a 404 handler in pyramid to delegate not found views to pylons, and move views one by one.

In a pyramid tween you can push pyramid session, settings, request,response to pylons app globals. This approach works really well - you don't have to do total rewrite. You have same request/response objects, same sqlalchemy ORM, so you can certainly take advantage of that during migration. I don't think anyone at our company regreted the path we took and it was a massive codebase that was making money - no one can afford rewriting that from scratch in $WHATEVERISCOOL today solution.

It certainly is possible - if you need any hints contact me.

Awesome reply :)

This project has reached its end of life, as far as the owners of the site are concerned. I've just completed a 4 month revamp on it, basically giving it a facelift by moving it from hand-crafted CSS, javascript and crufty old templates, to Bootstrap 4 and a more modern look, with some new features; project "Lipstick on a pig".

The company in question has plans for new areas of business for which this site won't be able to handle. It's simply not worth spending the time, bother (and money) trying to hammer that round peg into new square holes, and hence not worth rewriting to Pyramid. The revamp mentioned above will tide them over until something more suitable (and bespoke) is created, which is where Django comes in.

Release announcement: https://www.djangoproject.com/weblog/2019/dec/02/django-3-re...

(The release note erroneously still show the "Under Development" banner.)

OT, but what do you use to validate JSON posts without DRF? We mangled django form fields to validate a list of ids, but can I use forms to validate nested JSON? I now use Marshmallow for that use case but would love to stick with Django forms.

For those on the 1.11 LTS release, and considering getting off the LTS train, I wonder whether it's a feasible upgrade path to go straight to 3.0, or if it would be smarter to go to 2.2 first.

I suspect it would be far kinder to yourself to first upgrade from 1.11 to 2.2 and get all your tests working and then consider upgrading to 3.0.

My experience has been that the easiest method is to to upgrade one minor release at once (e.g. in your case I'd do 1.11 -> 2.0 -> 2.1 -> 2.2 -> 3.0). This has almost always gone quite smoothly, but it is somewhat tedious.

That said, jumping a bunch of releases (e.g. 1.11 -> 3.0) actually often works out just fine, especially if I've got good test coverage. It's just that when it doesn't work, it can be tricky to find out what's going wrong. That's because the release notes are typically written with just a single step in mind, so with a bunch of releases I find myself scrambling to figure out which set of notes I need to read.

I recently went from 1.11 -> 2.0 -> 2.1 -> 2.2 over the course of about a week, deploying to production each time we had all test passing to ensure there wasn't anything amiss. There were actually relatively few changes to get there, the upgrade to Python 3 was a bigger (albeit still manageable) change. I'm expecting very few changes to get onto Django 3.0, will probably wait till the first bug fix release though as it tends to catch a few things.

The best thing to go is run the dev server with `python -Wd` to get the depreciation warnings, and fix them before each upgrade. Worked perfectly!

I recently upgraded from 1.11 to 2.2 in one go. I didn't have any issue at all, besides a problem with mysql drivers. I had to switch from using PyMySQL, which is not the one recommended by Django anyway, to mysqlclient, and that caused one minor problem that was trivial to fix. Looking at the release notes for 3.0, I'd expect that if I migrated from 1.11 directly to 3.0 it would've been a smooth transition.

Of course it depends on your codebase. In the past I worked on a project which interacted a lot with Django internals, with a lot of code reflection involved. Upgrading that was a pain even between minor releases.

The recommended method is to go from one minor version to the next. 1.11 > 2.0 > ... > 3.0.

It's not much work, and you might just take the opportunity to stop and read the release notes of each release and find something useful there.

> Django is now aware of asynchronous event loops and will block you calling code marked as “async unsafe” - such as ORM operations - from an asynchronous context.

So DB calls aren't async?

Correct - the Django ORM is not (yet) async.

The asyncpg library[1][2] could be used for direct, asynchronous database access, from within a Django 3.x app.

The asyncpg library is a low-level async Postgres adaptor, similar to psycopg2, except that it provides much better abstractions and automatic encoding / decoding (vs just piping bytes back and forth like psycopg2), not to mention it is much faster than anything else out there. IOW, asyncpg would serve as a much better foundation for a future ORM. My best guess is that, in the future, Django's ORM will be refactored to optionally support async and will supply an adaptor for asyncpg.

1. https://github.com/MagicStack/asyncpg

2. https://magicstack.github.io/asyncpg/current/

aiopg is also great and has full SQLAlchemy support

Asyncpg is amazing, I’m writing a thin dataclass-style semi-orm-layer ontop is asyncpg. Still very early but I’m considering cleaning it up and extracting it to a stand-alone library if I see any demand for it.

OT: Are there any backend JSON/REST open source applications* that allow declarative configuration for your storage (eg. set SQL statements in config files) instead of needing compilation?

I have been learning flask for development. Can someone describe advantages of using Django over flask?

Please check out the following posts: 1) https://devel.tech/features/django-vs-flask; 2) https://www.airpair.com/python/posts/django-flask-pyramid. Hope this helps.

Is substituting the Django ORM for SQLAlchemy doable in V3?

i <3 django. thank you to all the people involved!

The last time I used django was for one piece of a capstone related project back in college, and for what I was doing, it was dead simple (to get up and running, that is).

I’ve seen python used extensively for data science work, but I’m not sure how well dynamic typing works (in terms of support ability/maintenance) in a larger code base, especially as some trivial knowledge gets lost and newer folks are introduced to an older code base. Do you just set break points to trace? I digress, but open to hearing folks thoughts.

Does Python still have performance issues? That's always been my concern when considering its adoption.

It's always prioritized delivery speed and quality over raw performance. If you want an ultra optimized web framework it will always be the wrong choice - it was never designed for that.

nothing significant changed.

The are some new-ish features like async and there some new parallelization options in 3.8 that help in some very specific cases, but generally you choose Python because you value development speed more than execution speed.

You can use Python (or Ruby, etc) to build fast websites that scale well if you are smart about how you use it and how you design your application, but it still isn't a particularly fast language compared to others.

Yes. If you can afford the CPU power, by all means go with Python. Otherwise just don't.

Instagram can afford it so it works fine for them. Reddit can't, so their site is always slow or down.

I think reddits issues are more related to database IO than cpu time.

And you know it is becouse of python reddit is often slow or down? Or are you just blaming pythin without any actual data on the underlying reasons?

You don't know what you're talking about.

I mean, what kind of performance issues were you running into?

You're not going to get maximum performance out of a python program, generally. But unless your python program is using 100% cpu that's not really an issue, and there are very few cases where I'm using python and it's CPU that's slowing down the program. Mostly it ends up being some kind of IO.

If you're looking to do complicated mathematical operations, you can call out to an external library like `numpy` of tensorflow to take better advantage of cpu/gpu compute.

You can also use a faster python interpreter, like `pypy`, although then you start getting slower startup times and more memory usage. The performance is closer to something like java but drawbacks are generally enough that they don't get used, since it's very rare that you need that extra performance.

There's also micropython, which lets you run python on a microcontroller and makes it pretty easy to optimize things for your hardware, although you lose some useful debugging and introspection features.

But for the most part, I haven't run into anything I'd call "performance issues", just the knowledge that python is a scripting language and it's never going to be the fastest thing around. Python's performance just isn't an issue very often.

In my experience most people creating CRUD websites with Django will hit a CPU bottleneck.

Would Django going async ease that a bit or not really?

async doesn't decrease the amount of CPU cycles needed to do work, in any case it only increases it.

I've got more than a decade of on-and-off web-dev experience, a lot of integrating data-science projects with a web-interface, and I've never seen anything like that.

You have to work pretty hard to hit a CPU bottleneck, and I can't imagine how you'd do that building a simple CRUD website.

Can you explain a bit more about how the people you know are hitting that bottleneck? I mean I've literally built CRUD apps on an esp32 microcontroller using python, and they work performantly. If real python with a real web-framework can't do the same then something is going horrible wrong.

If you do a bunch of filtering/joining etc that should be done in SQL by grabbing whole tables you can hit a CPU bottleneck...

But then maybe your problems are deeper that choice of language.

That's trend that I see often. The other day I saw someone providing a response to argument that when using NoSQL database you need to plan in advance how the data is accessed (so you can pick the right key). The response was "don't you need to do that in every database?".

So many people immediately dismiss relational databases, then re-implement all that functionality in their apps with various bugs and performance bottlenecks.

ORM is another bad technique. It supposedly promises you that you don't need to know SQL to use, and that is true for the simplest examples, but you absolutely have to know it for anything less trivial, then you have to figure out how to write query in ORM to get a desired SQL statement (it makes it very difficult to use advanced SQL functionality), and that's not the end. ORM constantly will make unnecessary SQL queries and by default request all fields, even if you don't use them, adding additional performance bottleneck.

In my current workplace thanks to ORM we make on average 10 queries per request and at peak generate 1Gbps throughput from/to database because of those inefficiencies.

I think the way to go is SQL support in IDE, I recently saw PyCharm with DataGrip where if you configure it to connect to a database it will start recognizing SQL in string statements and start treating it like code (so you now have autocomplete, refactoring etc). I think this is probably the proper way to do it and I wish that other IDEs would have similar features.

> It supposedly promises you that you don't need to know SQL to use

please support this assertion with an ORM whose documentation promises this.

> ORM constantly will make unnecessary SQL queries and by default request all fields,

as does "SELECT * FROM table" if you don't write out the fields and use a buffering database adapter (which is the case for nearly all Python database adapters), so, when using an ORM, you need to give it instructions over what columns you need to fetch. This is not unusual nor even anything a library could possibly guess for you if you do not give it this intent.

The problem is that once you start fetching specific columns in your queries, the resulting objects aren't entities, just records - i.e. it's not really object-oriented, which is the main allure of ORM.

ORMs that are more honest about this, such as SQLAlchemy, are generally better than those that try to pretend that you really are dealing with entities.

I use .values() and .values_list() to fetch raw records instead of objects. It still allows me to construct the query using the ORM which is nice.

I think that "ORM" is a bit of a misnomer for that. I'd call it "language-integrated query", if not for the potential confusion with LINQ (which stands for exactly that, not coincidentally).

There's a clear distinction between query builder and ORM though you can especially see this with SQLAlchemy.

But anyway even query builder is not that great. Recently used PyCharm's integration with DataGrip. Basically the way it works is that if you configured a database in your project and let PyCharm fetch its schema, suddenly the IDE started recognizing the SQL statements providing autocomplete not only the statements but also table names etc, it also offered refactoring which created migration scripts.

After using that I think that's the proper way of solving the impedance mismatch and at that point you no longer need ORM or query builders. I hope other IDEs will start doing the same thing.

> I think that "ORM" is a bit of a misnomer for that.

Good point: I'm no longer doing object mapping in that case. :) Though I think the most complicated part of the process that the Django "ORM" does for you is build the query. Maybe it should be called Django query generator.

> please support this assertion with an ORM whose documentation promises this.

I don't think any ORM would officially claim that, but many people decide to go that way to avoid SQL. For many simple examples it looks simpler than SQL.

I also experimented myself and wrote my code using SQLAlchemy's ORM. Then edited my code to use psycopg2 natively. I realized that SQLAlchemy didn't save any code for me, in fact it was more verbose. It also heavily encouraged me to get the raw data from database and do processing of it in the application. It also did unnecessary extra queries.

So with it now I need to:

1. still understand SQL

2. trying to figure how to write the SQL statement using ORM

3. figure out how to make ORM not to do extra queries

> as does "SELECT * FROM table" if you don't write out the fields and use a buffering database adapter (which is the case for nearly all Python database adapters), so, when using an ORM, you need to give it instructions over what columns you need to fetch. This is not unusual nor even anything a library could possibly guess for you if you do not give it this intent.

My problem is that in addition to knowing SQL, I also need to know how to do in that specific ORM that's for the specific language I'm using.

As someone that knows SQL I find that the ORM enables me to have decent queries rather then getting in the way.

I don't know what kind of operations you're doing, but I started having issues for anything that's not trivial. Perhaps those are achievable, but then in addition to knowing SQL I also need to know ORM (which is also different per language)

Some things:

- to make my application scalable I want two endpoints one that is read only, that goes to replicas and one that goes to master for modification. That way in my application I have full control which queries go to replica and which need master

- use datatypes that are recommended by the database

- do upsert

- do select and only return fields that I need

- perform insert and return only fields that I want (my ORM was performing another select)

- ORM also does bunch of queries that I don't need, I don't want it to add extra load to database if I never use the results of it

Those are very simple things, but I'm also curious how you can do some advanced operations like group by having, window functions, CTEs, aggregate functions that return JSONB structure (so you don't have to send duplicate data and avoid N+1 queries). How do you use PostGIS?

> Those are very simple things, but I'm also curious how you can do some advanced operations like group by having, window functions, CTEs, aggregate functions that return JSONB structure (so you don't have to send duplicate data and avoid N+1 queries). How do you use PostGIS?

You don't! That is not what an ORM is for. Use SQL. Any decent ORM with allow you to integrate SQL.

Check out the Fluent (query builder) and Eloquent (ORM) from Laravel. They allow all the features you want.

IME database always bottlenecks first.

Your experience is very much unlike my own. Every project I’ve seen hit issues with the database (e.g. unindexed queries, changing data access patterns) first, followed by I/O (especially other services like S3, search, etc.), and RAM usage before CPU became a major factor.

It’s possible, of course, but I’ve usually seen it as a symptom of not having a good culture around monitoring and troubleshooting — e.g. I remember someone porting an entire site to Jinja2 alleging performance wins, and it’s true that template rendering got (IIRC) 10-15% faster but they’d missed that 99.999% of the total runtime was being taken up by unoptimized database access generating many thousands of queries.

Of course, if you are incompetent enough to write queries that use no indexes and you don't detect that before you push to prod, you will have problems there.

What I mean is: when you start optimising, no matter how well you do, you will eventually hit a CPU performance wall. Then you will realise that only a rewrite will get you out of that hole, and by then it will be late.

In my experience, many sites never hit that point of having CPU-bound Python code which they can’t afford; the ones which do hit that wall are usually large enough that they can carve out room for a C/Rust library, microservice, etc. rather than rewriting the entire thing.

I concur, I’ve worked on at least a dozen “web scale” Django sites and I can’t think of a single one where we were CPU bound.

In my experience it doesn't sound like you have a lot of experience anywhere near CRUD websites with Django or otherwise.

just going to post my usual answer to these types of comments: if it's good enough for Instagram, it's good enough for us.

Yes and no. The Python interpreter/bytecode-VM is not as fast as it could be, because only a few people work on it. PyPy is great, but not drop in. Lack of funding is the proximate cause, of course, but that's just masking the ultimate cause of lack of interest. Because people use Python as a glue, just like PHP, or Node. To do the heavy lifting you have many options. (Even in Python, you can just use Cython, use FFI to call out to anything with A C interface, like Rust, or C++, or C. Many Python packages do exactly that, wrap a battle tested old native lib.)

That said the Python VM is a well optimized beast, it's very dynamic, you can monkey patch anything and everything at runtime on-the-fly. This is of course what makes it hard to speed up, too many things to "check for overloadedness" when doing anything. A JIT could help, but that's a lot of work. (Hence why PyPy took a decade, and the whole Python C API compatibility problem is still there - PyPy can only JIT the Py parts.)

Anyway. With that much dynamic stuff and with the VM and the language already well tuned for concurrent memory safety its 'async' story is great and performant. (Like NodeJS'.)

So Python is great for orchestrating whatever you need to handle requests. And if you want to scale up, just spin up more instances. (Just like with Java, but you don't have to think much about the overhead garbage collection, because CPython uses simple reference counting.)

Finally, my concern with Python was the lack of static typing, which made any large project hard to manage. (And resulted in a lot of boilerplate tests and excessive defensive coding.) But mypy is great, and with that it's a joy to work in Python. (Just like TypeScript made the JS world exponentially much more saner.)

> PyPy is great, but not drop in.

Nowadays it works correctly with almost all C modules. Mostly you just need to use `pypy3 -m pip install foo` instead of `python3 -m pip install foo`.

Last year I had to fight with it quite a bit to get it to work with uwsgi.

It seems that any new/greenfield Python project should target PyPy though, just in case, and the performance never hurts.

What about the type hints they introduced? I enjoyed how you can go back and add them after the fact in separate files.

Yes it does, and almost everything else will have performance issues except assembly. With web services, chances are that you'll hit other bottlenecks before you hit it. In python you have multiple ways to optimize performance, for example use PyPy, or various libraries like Cython that compile your performance sensitive code.

That's how asyncpg can outperform[1] some libraries that you normally would think was not possible.

[1] https://github.com/MagicStack/asyncpg

> ASGI support

> Django 3.0 begins our journey to making Django fully async-capable by providing support for running as an ASGI application.

Bleh. ASGI is probably the worst implementation I can think of for a common async interface. Stringly-typed callbacks? async generators exist for a good reason. It's also very asyncio-centric, and asyncio is Not Good.

Can you explain why asyncio is not good?

It's convuluted, it's not structured concurrency (https://vorpus.org/blog/notes-on-structured-concurrency-or-g...), fundamentally despite being async/await a lot of networking code is based on callbacks anyway (see: asyncio.Protocol), and I've had a lot of trouble debugging code that spawn tasks because asyncio code has zero obligation to care about any of the tasks it spawns.

Yo. Two points here.

* You can absolutely stay within structured concurrency constraints using ASGI.

* The ASGI spec doesn’t have anything to do with asyncio. It’s purely an async/await interface - You can build against it just fine using other concurrency backends.

I'd love to hear your suggestions for changes we could make while keeping it somewhat WSGI-compatible. It took a few years to refine it to where it is now, so it's not like we just threw something at the wall.

> while keeping it somewhat WSGI-compatible

There's the problem. WSGI is fundamentally flawed too - it could also be using generators for a two-way communication channel instead of stringly typed callbacks.

In a world where Python has optional static type hints, it would be nice to have concrete objects passed too.

Can you expand on the stringly typed callbacks? Where are they required? Are you referring to string keys in a callback dictionary, in which case stringly-typed seems a bit of an odd choice of words.

Calling the receive async function gets you a dictionary with a string type key, which defines the layout of the dictionary. It's not nice.

It's much easier to say stuff sucks than to actually get a huge project like this shipped. Thanks so much for your hard work Andrew + team!

I don't often comment on HN, but I feel like this time I have to step up.

Having worked and developed with it extensively for over a year now, ASGI is one of the most elegant and well-designed interfaces I've seen.

Just wanted to address this:

> async generators exist for a good reason.

While the idea of using async generators might sound attractive at first, there's so much more to ASGI than exchanging messages in both directions.

Even on that aspect, I'd be interested in knowing how you'd achieve bidirectional communication async generators. How would the ASGI server call into the application and handle messages coming both ways? Surely, it can't just `async for message in agen: ...`, as that would only replicate the `send()` ASGI callback — what about `receive()`?

Besides, how would middleware look? Would it be as simple as wrapping `receive()` and `send()` into whichever other callable adds the functionality your middleware needs?

And you probably wouldn't be able to support all the possible syntactical implementations allowed by the notion of a Python callable: functions, classes, callable class instances.

So, yeah, I think ASGI is a beast.

Also, I wholeheartedly congratulate the Django team for pushing the standard out there and driving the Python async web ecosystem forward. You rock!

> Even on that aspect, I'd be interested in knowing how you'd achieve bidirectional communication async generators. How would the ASGI server call into the application and handle messages coming both ways? Surely, it can't just `async for message in agen: ...`, as that would only replicate the `send()` ASGI callback — what about `receive()`?

Yield is two-way. ``result = await agen.asend(next_message)` ` et al.

> Having worked and developed with it extensively for over a year now, ASGI is one of the most elegant and well-designed interfaces I've seen.

Having worked with Python async for 3 years, anything that asyncio touches is completely poisoned. ASGI is another in the line of terribleness of the modern async ecosystem.

> Besides, how would middleware look?

A second async generator.

> And you probably wouldn't be able to support all the possible syntactical implementations allowed by the notion of a Python callable

Require an async generator. Done!

> Also, I wholeheartedly congratulate the Django team for pushing the standard out there and driving the Python async web ecosystem forward. You rock!

Much like asyncio, something isn't good just because it's being pushed.

> Yield is two-way. `result = await agen.asend(next_message)` et al.

It's still limited. To get a result you need to send something. Using it for something that's not strictly request-response can get complex.

> Having worked with Python async for 3 years, anything that asyncio touches is completely poisoned. ASGI is another in the line of terribleness of the modern async ecosystem.

That's your own bias. I also used asyncio from the beginning and got a different opinion. The asyncio package that comes with python is low level and is meant for building frameworks. If you used aiohttp, asyncpg and other packages packages build on top of asyncio it's actually quite enjoyable.

Reply on news.ycombinator.com