https://www.djangoproject.com/weblog/2015/jun/25/roadmap/
I've grow to highly respect the Django project for its good documentation, its healthy consideration for backwards compatibility, security, steady improvements and all round goodness.
I hate their API and overall architecture, which I find to be the result of glueing features on top of features for many years. The internal code also is just like that: looks like every single method is riddled with out-of-band conditionals, which is the result of a community that prefers to hack things to work, instead of rethinking/refactoring.
I'm curious, what's an example of a framework that you think has a better architecture, and in what context are you evaluating this?
However, once you're out of the unrealistic scenarios (single query benchmarks, etc.), it doesn't do that well[1]. It's not prohibitively slow, but to make the jump from something as well documented and with as large a community as Django, most developers would need to see major gains in one or more areas; performance, docs, community, coding efficiency, etc.
Pyramid doesn't offer these gains. It only makes promises about maintainability, which are hard to verify, unless you personally know somebody that you trust as a skilled developer, and who has worked on a sufficiently large enough project in Pyramid to make such claims... it's easy to see how this creates a hole that Pyramid has to dig itself out of.
Also, Pyramid still talks about "supporting your decisions", like Jinja2, etc., as if that's a problem people are still dealing with. Django supports Jinja2 even in the admin now, let alone being able to use whatever you want elsewhere. As a side note, I don't see many Python developers wanting to use anything other than Django templates (for simplicity and separation of concerns) or Jinja2 (for speed and flexibility) these days.
If the argument, in response to the aforementioned difficulty in verifying claims about maintainability, is "well, it's up to you; Pyramid stays out of the way", then the immediate response is that you're better off using Flask or Falcon; the former if you need third-party tools, and the latter if you need maximum raw speed, but still want Python. Both of those frameworks will drastically outperform Pyramid and stay out of your way.
IOW, I don't think Pyramid fills any reasonably sized and easily understood market gap.
1. https://www.techempower.com/benchmarks/#section=data-r13&hw=...
Django has grown up a lot in the last several years. It used to be really hard to use a different templating language, and I personally really dislike Django's templating language. That was the biggest turn off for me every time I looked into Django. If it had been pluggable earlier, this probably would've changed the calculation for me.
I've always thought of Pyramid as the ideal framework, conceptually. It feels like you're writing Python, not Pyramid, which is GREAT. Pyramid only shows up when you say "OK, I have my Python stuff; I need to make this show up in the web browser now." It is entirely unobtrusive. All frameworks should strive for that.
Unfortunately, the vast majority of frameworks take the opposite approach. Django at least used to be an example of that by forcing DTL (is this the abbreviation for "Django Templating Language"?) down everyone's throat, among other things.
I've also been very impressed by Chris McDonough and the rest of the Pyramid team. They're extremely helpful in IRC, they keep up the great work even after spending years as a small/niche framework, and their code is very clean and well-tested, not obnoxious or overcomplicated. It's a tight, consistent framework that stays out of the way and is just there to serve the developer, not to force him/her to comply. It is so everything a framework should be and so everything most frameworks aren't that I can't help but love it.
I haven't done a major Python-based web project for many years (have one in progress, porting a Rails-based site to a Mezzanine-based site, but work on it only rarely), so I'm sure some of this is outdated.
What makes Pyramid better for you than e.g., Falcon?
From your comment, it sounds like a lot of the value you've gleaned from Pyramid is based on the framework staying out of your way, so why not use something 5x faster that does the same thing? Does Pyramid provide some unmatched abstractions? (I'm coming from a position of no true experience in Pyramid)
You have event system, security abstraction, all kinds of overloading helpers for views and internal machinery that falcon doesn't seem to provide at least from my cursory look. Compare the size of documentation and configuration options between both projects.
Also looking at this http://klen.github.io/py-frameworks-bench/ the 5x better speed of falcon seems to be completly made up or occur in some very specific situation. More realisticly it looks like falcon can be 30% faster in some scenarios not involving storage - but for the price of giving you less options.
Which probably means that if you use any kind of storage the difference between those two can be ignored.
I'm curious about the result since it didn't matched my experience. So I proceed to take a look at the source code for both Pyramid's[1] and Django's[2] benchmarks. From the look of it, I feel like there are too much difference in implementation to call this a realistic comparison.
From a quick glance, Django's benchmark seems to contain a little bit of micro-optimization and is rendering JSON response directly with uJSON[3] (which looks to be a lot faster than native JSON module)[4], while Pyramid's benchmark did not seems to go through any optimizations and is returning a list of SQLAlchemy objects that get passed into a custom renderer[5]¹ and render using native JSON module.
So I don't think it's entirely fair to say Pyramid is slow in a real world scenario based on this benchmark alone.
[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/c8a5...
[2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/c8a5...
[3]: https://pypi.python.org/pypi/ujson
[4]: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-j...
[5]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/c8a5...
¹ Using pyramid.renderers.JSON here probably make more sense, but to match Django's implementation, one can wrap a serialized JSON into `Response` object.
And the developers seem to think this is totally ok.
In my experience, anything that doesn't have a commitment to at least fixing security issues for ~5 years, isn't a good choice for business. Businesses don't want to tie their internal development and release cadence to an external development team with less than 3 year cycles, ideally 5+.
I've seen it time and again over dozens of companies. There is no time to qualify apps for new major/minor releases. Anything that doesn't have a LTS release story is just not something I can feel good about deploying in my business.
To be clear the core maintainer-ship of pyramid is about 2-3 people. We would love more contributors to the core codebase, but with that level of commitment we're obviously not able to provide a comparable level of support to a project with more contributors. This is the nature of open source and it is the rare exception to the rule to find packages / projects that have grown to be able to offer such a level of support.
(Aside: mmerickel is the developer I was talking to in Freenode)
Honestly, I didn't know that Pyramid had such a small amount of manpower behind it. It presents itself and has a reputation in the Python community, at least in my experience, of a bigger project.
But, as far as not being able to commit to security releases for old versions: it's obviously a choice in where to spend the available resources, not one of the size of those resources. Choosing new development over providing critical fixes of older releases is a choice, it's just one I need to understand before I commit to using a component. I understand why one would make either of those choices, I just need to know which one has been selected. It is great for people who want to choose forward development focus, I'm just not one of them.
I believe it's better for them to focus on what they know they can do day in and day out regardless of the circumstances and let companies make the decisions that make sense to them.
In a world where the hardware is written off by the business in 3 years though and websites age out at 3 years and car manufacturers are increasingly trying to hit a 3 year model revision mark it seems to me it's not too much to ask that companies investing hundreds of thousands in an application using Django or Pyramid also invest in maintaining it and keeping it modern or perhaps their existing business model needs additional revision too.
Just a thought as the least informed member of HN.
- request/response (webob) gets its own updates.
- templates get their own updates.
- sqlalchemy gets its own updates Etc.
You WILL get security fixes for most of your application even without upgrading the framework itself (the attack surface for pylons/pyramid itself is smaller than monolith).
By being on a lower version you don't really miss much - the API's are very stable. And 99% of time you can update pyramid version itself without risking application breakage (the chance of it is probably a lot lower than with monolithic framework).
I know about very heavy websites that are still using pylons(precursor to pyramid) in production without issues (8 years now).
[0] http://keitheis.github.io/use-pyramid-like-a-pro/?full#Cover
Use Pyrapid Like a Pro
This makes it sound like an advanced toy rather than a bullet-proof work of engineering you can rely on. Which is really a pity, because Pyramid _is_ a framework you can rely on.
It's a shame, because I've wanted to try Pyramid since I watched their talk at PyCon 2011, but the branding (including the new 99 Designs looking logo / theme) has always made me think, "This will never really take off."
Still my go-to framework though. The websauna framework linked above might be something I have to try.
As I build more and more side projects I find myself wishing pyramid were just a bit more opinionated so I could get the basics up and running a tad faster. At this point I mostly cut up old projects and port them to each next one, heh
The theme is one of the included ones, Ribbon.
They say "Start small, finish big", and push the fact that Pyramid scales from a single file codebase, up to many files. This is something that I just haven't seen in practice.
A Django project requires quite a few files, so it's never a great answer when the total amount of business logic is <100, or maybe even <1000 lines of code. However, a 10k line Django site and a 100k line Django site basically look the same in my experience, and that is hugely valuable.
The biggest red flag from Pyramid for me is their point: "Use Pyramid as a "framework framework" to craft your own special-purpose, domain-specific web system".
In my experience, this means that every project is a special snowflake of a project that works in a very different way to other projects. With Django you have a tried and tested architecture, with Flask and Pyramid, it often seems that you have to create your own architecture in some ways, and that results in each project being really quite different.
Offers Django feature parity with best of breed components.
If you have any questions please pop into the chat http://gitter.im/websauna/websauna
And breaking backwards compatibility
Django is not RoR. Their users rely on being able to upgrade seamlessly.
The overall Api makes sense, there are some rough corners (yes Sites, I'm talking about you) the docs are ok once you get the hang of them
If so, I have to very strongly disagree. It's almost mind boggling to me how much they break backwards compatibility as a framework. They usually warn users with a deprecation warning in one version and then they make the backwards incompatible change in the next version, but the sheer amount of these kind of changes makes upgrading an infuriating process. Especially when upgrading a large codebase.
That's exactly how _not_ to break backwards compatibility.
No, no, no. Backwards compatibility means that there are no breaking changes at all - announced by deprecation warnings or not. Think Linux's public API.
IIRC the most impacting changes I remember were in Config, replacement of South with a native solution and some timezone issues
The documentation was telling something like: Ensure you applied all previous migrations and then start from scratch. Thats not an upgrade path. It may work for standalone web applications, but I have a python project (packaged as deb/rpm package and using the packaged django version of different distributions) that should work with multiple django versions and the user may upgrade the django version at any point.
I had to write raw sql statements to get the mirgration status of south (after the upgrade, without a working version of south anymore) and "fake" apply the migrations of the new native solution.
That said, having started on updating that same codebase to Python 3, the Django team tries hard to provide the KEY thing you need to seamlessly update a codebase: a version which can support both incompatible features. Without this, you need to have one giant branch with a bunch of renames and other changes. They actually introduce deprecation warnings, and have something non-deprecated you can use in the same version. I think if Python core had introduced a version of Python including both versions of all the module renames with deprecation warnings, the community would be much further along on the upgrade path.
A platform that works hard to be backwards compatible does only additions and extensions, and keeps maintaining the old API as well so that old apps run without changes; possibly keeping around multiple depreciated ways to do the same thing (e.g. as win32 does).
It is debatable whether backwards compatibility is worth that cost (and it often isn't), but it's certainly a choice.
Then view HN threads of people saying they have thirty-million-line codebases which utterly rely on deprecated functionality and they never ever plan to upgrade or do even basic maintenance ever for any reason ever, and will abandon your platform and completely rewrite in something that treats them "better".
Not at all. That would have the added benefit of core developers thinking twice before introducing a new API - exactly like it happens with Linux syscalls.
content_types, generic foreign keys, and basically any other uncommon use pattern for RDBMS are very rough in django. Tradeoff of their ORM being heavily streamlined for the 90% use cases (typical SQL selects and upserts).
Might be easier to just write SQL in those cases
We are users of Django, and it helps us to deliver projects, quickly. Interested to know how you think things could be improved.
But there are components that fit that. The entire form subsystem is awful to work with. Working with Javascript, webpack apps etc is a huge pain.
The template syntax is also a failed design experiment, based on the premise that backend coders and template authors are not the same people and should not have the same level of power -- some of this is based on a good idea at its core, but Django ended up backtracking on that half-way and we're now with a very inconsistent template system and having to implement kludgy template tags and filters or new object methods that do not take parameters whenever we want to do anything remotely complex.. At this point I wish it'd just move towards supporting Jinja syntax (that is, alongside the regular syntax, not as a separate engine like it's currently possible).
on a really fundamental level working with complicated forms is just a complicated thing to do, no matter what kind of technology you're doing it with.
Could you explain what you mean? Django doesn't need to have any connection at all to your JavaScript stack.
We use Django Webpack Loader for our sites: https://github.com/owais/django-webpack-loader - This allows us to do `{% render_bundle ... %} which pulls in the appropriate script tag.
But even then you can really feel how painful it is to work with, especially if you're adding typescript/scss to the mix. Solutions like Django Compressor are not really the right model anymore.
And there's other issues as well. If you want to share settings between Django and the JS stack for example, you'll need to build your own pipeline for that. Building a form in react? Say goodbye to DRY on the form fields. Just, in general, Django predates JS apps being anything more than in-place enhancements and it shows.
For large apps it makes sense to do it this way. For smaller apps it really sucks.
This also has the side-effect of making the transition to client-side templating easier.
There was a patch to change this to use templates, but it was rejected at the time as it slowed things down too much.
FormWizard (which I hear doesn't exist now), was a nightmare to use for any sort of complex form - I tried to do just this and had to call many private methods.
I understand metaclasses, but found the implementation of the ORM a little overcomplex.
Trying to use the ORM API to work out the structure of the DB was a bit of a pain last time I tried (had to call private APIs).
The html forms (used in the admin and everywhere else) have until now been generated from strings, however the next realise 1.11 (went to alpha yesterday) this has been changed to using the template tools.
I myself and also other people I know have written quite some domain-specific applications almost entirely within the Django admin, which are as a result: secure, fast and easy to use, which in some cases are not just a bit better than the existing commercial tooling in these domains.
The overall architecture is sound though. Tight coupling of core components, loose coupling of non-core components. Keep in mind that it's an "opinionated" framework though, so they do safely assume that if you're using django you want to be using its core components as a bundle, and will work with them as designed.
A project I was really impressed with the internals of is Celery (was expecting the worst having seen Django).
But is it really? Speaking from personal experience it is easy to compare project with large featureset (and one with heritage) to one with scope on doing single thing and come with conclusion that smaller, focused codebase is more consistent and better implemented. At the end of day what matters is if those terriblenesses actually bite back:
- is this code changed frequently? Does it need to be changed frequently? - is it written in a way that that makes fixes and improvement unbearably costful? - is it written in way that allows it to be put apart? How costful are those individual parts to improve?
Django is large codebase that is worked on by different people and when time permits this, which means different parts differ in their age, practices, and ultimately, quiality. This eventually results in codebase that may give appearance of being messy.
Joel Spolsky explains this nicely in his article about old and large codebases appearing as hairy and messy to developers:
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Especially the part that follows below quote is valuable wisdom to keep in mind:
> When programmers say that their code is a holy mess (as they always do), there are three kinds of things that are wrong with it.
Also, up until recently it was a one man job by Ask Solem. He's done incredible work making this project live and I am grateful for that. However I fear others have found it hard to maintain as well, slowing down progress on a very popular backend python project.
Is this a joke? They break backwards compatibility with every minor version, wasting tens of thousands of man-hours all over the world - time that, if we're honest, is not exactly billable.
Most people don't upgrade because of this and keep on using vulnerable versions. It's good that they are trying to kill the project. It saves newbies from stepping into the tar pit.
For the most part, they think they things through a lot and most importantly they document breaking changes. Their approach is an absolute dream compared to, say, updating xcode/iOS apps. It's a total shit show at Apple.
I think the project is venerable and I have respect for it in general. It's one of the most accessible major open-source projects for newcomers; like I said, I'm not a heavy user and I've already submitted a couple of patches and had them merged. But issues like this one are worth pointing out.
Is there a deficiency in the test suite? Is the open and accepting nature of the community that I just praised the cause of this issue, because it leads to a bias against thorough vetting? I really don't know enough about the project to know, but it'd be nice if they figured it out.
I manage a modest project and there are plenty of issues with corner cases that I knew were possible but never could reproduce. As people stumble upon them and report issues I can update the test suite to avoid stepping on the landmine again, but that doesn't change the fact that we've barely scratched the surface.
Here is a strawman low level test: Randomly generate a sequence of nonsensical (but legal) API calls by randomly generating some data layout, then feed it into a state machine of legal API calls (eg. CRUD), and check the return values. Once that runs for 10 minutes without crashing, extend the test to be multithreaded and run with 1000 thread for a few hours. Dial back the runtime to 60 seconds, and stick it in the regression suite.
As long as there is a well defined API, this finds most bugs (and I also write targeted tests to exercise tricky / error prone paths).
Any thoughts on why it is so much harder to implement reliable high level frameworks? Is it dynamically typed languages / lack of encapsulation, or something more fundamental?
I had to search for Tox. It seems like analogous tools would be nice for statically typed "systems" languages too, though there is less need for them there (more bugs are caught at build time, instead of after deploy).
Debian sort of does the same thing when it builds packages, but only checks compatibility with current versions of dependencies. It would be nice if they also checked / tracked compatibility breakage (the "not all libraries are good" observation is language independent, and a "n days since we broke users of this library" label would be great).
Java 7 had less than four years of support.
Ubuntu Desktop LTS was three years, and starting with 12.04 it increased to five years.
That's a language runtime and an operation system known for their stability, forming the bottom of very wide and deep ecosystems.
Show me a web framework with a 3 year LTS, and I'm overjoyed.
Like screwing the huge majority of users with Python 2.x Django projects and having them update or be left behind?
I still have one customer with a Django/python2 code base. When migrating to Django 1.11 in 2017, that project gets security updates until at least 2020. Plenty of time to gradually transition to py3. I honestly don't see the problem.
That's illogical and inconsistent. they're providing an LTS release for you. Get over it
I think this is smart and will help them focus on the road forward. Django 1.11 is an LTS release, so legacy stuff can stick with it and be just fine.
That's the version of Python that most of their users actually use -- with no major plans of mass updating.
I know that there are domains that are mostly on Python 2, and of course you'll always have legacy / unmaintained things lying around. But "zeee majoritie is Python 2.7!!1111" does not become true by some people chanting it over and over again. The simple fact that frameworks and libraries are moving away from Python 2 already proves that the majority does, in fact, not use Python 2. Otherwise maintainers would also be in an approximate majority to block/veto such changes.
Maybe check your language a little? "zee majoritie", "chanting it over and over" etc, gets tired and offensive soon.
That aside, there are actual numbers for Pypi supporting that. What do you have to counter these?
>The simple fact that frameworks and libraries are moving away from Python 2 already proves that the majority does, in fact, not use Python 2.
It just proves that after 7+ years, some frameworks and libs managed to justify porting over to 3 too. It doesn't say much about which is used more.
99% of Pypi traffic is composed of mirrors and bots. Python 3 toolchains are more likely to use tools like devpi, wheel, and Docker to cache their packages, while Python 2 toolchains are often going to hit Pypi directly.
We're concerned about which version has the majority of users, not about which has more downloads on Pypi.
Every OpenStack build, for example, pulls in hundreds of packages from PyPi.
In terms of real-world use, all of the Python devs I personally know moved to Python 3.
As seen with Django, they were able to support both. I've been able to support both with the code base to some of my projects as well.
Currently I have one project I'd like to be Python3, but it will involve forking two dependencies (owfs and phidgets) and having them support Py3 (phidgets actually builds and entire python3 tree and installs it and yet you can't import anything from it because it's all python2 syntax -_-)
Python 3 has been out for quite some time. I don't see why everyone is still holding out.
> Ansible 2.2 features a tech preview of Python 3 support.
So they are working on it.
What are the alternatives? Forking the language or switching to another language look to have a higher cost.
(auto-correct)
The JS community deals with breaking changes every 3 months or so.
From Python 3.0 the end of support, you have __15 freaking years__, warnings, tutorials and excellent tooling at your disposal. Oh, on a free software. Half made by some charity workers.
Now you made a choice, and there are very good reasons to have made it. We won't criticize it.
But you lost the right to complain.
asyncio is going to be a slow build, as the ecosystem starts unifying around it. But as that happens, we'll be much better off for it.
Plus, 25% of performance, while nice, is not really something that would matter for most Ruby projects. They are web projects, and their bottleneck is not Ruby. Same for Python. While I'm telling you, good unicode support in most european countries is a HUGE deal.
But that's not all. Often People thinking about Python 3 think unicode, but for me there are 2 other things that made my life much nicer:
- better debugging. Error handling is a hell lot better, with better and messages, greater granularity, more safety nets... E.g: you can't compare some objects anymore, you have several exceptions to handle file opening, imports are absolute by default, division is what you expects, a lot more operations are lazy, the stdlib has been cleaned manu redundancies, encoding parameters everywhere, etc. - less verbosity. Writting is consistently reduced. You get finner file boilerplate, shortcuts for OO, unpacking generalization, yield from, f-strings, etc.
Now those are things that are not easy to sell. You don't see them as a reason to migrate when you hear about it. But once you used to them, going back to Python 2.7 feels so bad.
Python 2.7 will EOL in 2020. But I guess even then some luddites will complain how they will have been left hanging
1 week is not enough
1 month is not enough
1 year is not enough
5 years is not enough
10 years is not enough
20 years is not enough...
Do you really have Python projects that can't be migrated in 3 years?
At some point the line has to be drawn to move everyone forward. Extending the EOL for the last long-term of 2.7.x by 5 additional years is pretty generous.
Unless you see some big corporate support contracts coming over the horizon for PSF, not to mention they'd have to be worth the money vs the technical debt, I don't think Py2 support will get extended.
There's less muscle behind keeping Python 2, but there's a lot less muscle needed to keep it going than to extend copyright.
Start your migration now.