“Python's batteries are leaking”
pyfound.blogspot.com
pyfound.blogspot.com
Amber's suggestion seems to be in the same direction (though perhaps not as extreme.)
[0] https://www.slideshare.net/mobile/hsbt/gemification-for-ruby...
The problem is that I cannot count on being able to install new software in many environments.
If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.
Consequently, the standard libraries need to be very complete and very useful.
And, while people seem to love the Rust approach to libraries, I'm not necessarily a fan. Far too many times I have pulled a library that is "obviously" something that a language should consider to be "standard library" and gotten bitten because it was broken. Only VERY core libraries in Rust are guaranteed to work across multiple architectures and OS's.
I think Rust is probably doing the right thing for Rust as "batteries included" is NOT one of its tenets. However, that doesn't make it right for everybody else.
Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."
At which point you have to scale the IT wall all over again if you work at a Fortune 500 company.
I work at a Fortune 500 company, and the only wall I have to scale when I want to use a library that nobody at our company has ever used before, is to get someone to check and approve the license (typically takes 1-2 days), and import it into our code repos.
I mean I see your point, but not everywhere is as bad as you make it seem.
As in, it is very difficult to get software installed in general images or on multiuser servers.
There are obviously good reasons for it to be conservative about this.
Not OP but same. I'm currently debating with myself whether I should attempt to install PUTTY. Given that port 22 is blocked and it's not needed for my core role it'll be dicey if I get challenged.
Pulling executable code off some repo...no way that is ever officially passing muster. People might do it anyway, but on a personal risk basis.
>Can you explain this more?
Place that are heavy on confidential financial info basically. Practically everything I touch is confidential client data. So employer is naturally jumpy about what's on my laptop software wise.
Ironically the above comes full circle...need putty to get onto a VM in cloud where there are no restrictions and crucially no client data. Nobody cares what I do there - hell they'll even pay for it thanks for MSDN enterprise
Found a work-around though - Google cloud shell being *nix works fine for SSHing about the place. Gets me around the port fw too
It might me more helpful to think of these types of external factors as fixed points that cannot be moved and just engineer around them.
You'll burn out if you try to boil the ocean on every business process that doesn't seem "logical" from your cursory examination.
The corporate world is full of stupid things that will never not change, or take years to change.
In a large company this gives the compliance folks a central place to blacklist packages - along with a trail of what systems have downloaded the package to target for upgrades.
And setup.py is a trainwreck, e.g. some packages compile download and compile huge dependencies (e.g. a full Apache httpd...), the default compiler flags may lack all the mandatory security flags (e.g. for using ASLR on python 2.x), or ship their own copy of openssl statically and break your FIPS-140 certification that way...
After about 9 weeks of emails, meetings, and pitches, I finally got Anaconda up and running. A week later, I tried to upgrade the 3rd party packages.. No dice. Blocked by the corporate VPN. I'd need the sign off every time I wanted to `pip upgrade` anything
Needless to say, I do not bother anymore.
We have our own development team, our own servers, our own freedom to deliver to clients fast without the hassle of the main corporation. How? We talked to the right persons.
Theoretically there's a process for requesting new software and getting it approved, but actually pushing it through requires getting one of my program architects to care enough to file the request (As a mere level 2 engineer all I can do is write it, can't submit), then potentially weeks of followup, for 1 specific version of 1 specific package.
In the case of python packages, perl and the perl packages we need are already approved because a few senior devs got together and pushed them through 10 years ago (was before my time, but I understand it was with quite a bit of arm twisting). It's more time-efficient to just code perl than to fight for python.
It's one of the many reasons I intend to get myself another job for Christmas. :)
As for why the system exists: Cost cutting, in the sense of "the less we invest in infrastructure the more we can divert to sexy hardware for the cameras and shareholder dividends. So long as it's theoretically possible for you to do your work, we don't care how many hoops you have to jump through to do it. And our competition is even worse than us, so we don't have to worry about anyone undercutting."
As a result all our infrastructure is centralized. Programs have to jockey with each other for everything from virtual servers to physical workstations and monitors. Hell the only reason my program has our primary test server is because one of our architects literally overheard a hallway conversation about a program that was spinning down and getting rid of some servers, so he jumped on it.
I worked for a much smaller government contractor, but before I left they were moving to a system where you needed approval from the customer in order to get new packages. (For those who don't work in this field, that means you are actually making a request to the contracting representative from the particular government agency for each package you want.) So it wasn't just in-house bureaucracy in the way of progress, and I generally just went without or wrote my own instead of trying to deal with it.
[0] https://training.kalzumeus.com/newsletters/archive/enterpris...
I'm afraid the standard library has to be aligned with the needs of more normal users who, as already discussed, want to allow libraries to have their own release cycles and to be more "opinionated" and specialized than the standard library would permit.
I'm afraid users dealing with that sort of bureaucracy are much more normal than you think, if not the norm. They're just usually not the types of folks that are hanging around HN, or they're at least less vocal.
You could install packages with `pip install --user` to have them install under your home directory.
> You could install packages with `pip install --user` to have them install under your home directory.
You might find that the issue isn't necessarily always technical (ie permissions), but policy. A lot of places don't let you arbitrarily download software off the internet onto a system. Pp's point is that having a full feature stdlib let's you only need to crank the policy/approval process once, rather than once for every dependent lib.
In this case, even If you can't upgrade, you still get the same libraries but perhaps older versions.
The approach Ruby is taking with gemification and default and bundled gems for standard libraries is equivalent to the traditional standard library if you can't install updates, but superior in other cases.
I don't know how many hours I've spent battling Psych errors because of this very thing, but it's way too many. Calling the gem something, anything else, would solve the issue.
It's great that they're unbundling a lot of things, but there's still some serious friction between external and internalized versions of these gems.
For Ruby, EventMachine sub-universe is really in bad shape. EventMachine is creaky and old. Event-aware packages are in short supply and are usually woefully out of date, unmaintained.
In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library.
For example, the libstd doesn't even have support for random number generation. There's a rand crate, which is now on 6th major breaking version. That's perfectly fine, because multiple versions can coexist in one program, and every user can upgrade (or not) at their own pace. And the crate was able to refine its interface six times, instead of being stuck with the first try forever.
As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys).
I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.
- What would you define as a while? I think Go has done well with it, given that it is almost 10 years old now at this point, and in reality, was in development internally at Google years before that.
- Do you feel like an interface being frozen forever is particularly a bad thing in of itself? What if the interface does a really good job describing the thing, whatever it may be? For example, Go's io.Reader/io.Writer interfaces.
I agree sometimes an interface being frozen is bad, but for example when Go standardized the context package, it simply added "Context" to the existing functions that now take a context (e.g. in the database/sql package, you have the old, Exec, Query, QueryRow functions, and after 1.8 you have ExecContext, QueryContext, QueryRowContext, which some people may view as a reason to have method overloading, but I view as adding better clarity.
I read some discussion the other day about some ways in which the Any type in Rust isn’t as flexible as it could be, since it is frozen the only way to improve it would be to introduce a new name, such as Unknown. Similarly in C# along with adding async support, the standard library added async versions of many methods, eg Read now also has a ReadAsync partner.
It does seem that having multiple names for basically the same thing adds a small but tolerable level of overhead to a language. At least if as much as possible is moved out, then projects can choose to only use the latest versions, and live in a world as if past versions never existed.
Rust libraries should be expected to take a few tries to get right, especially earlier in its lifecycle. There's more possibilities and less experience in the language.
You can see a similar effect in Haskell, which has iterated many basic bits of functionality many times over.
CompressedSize uint32 // Deprecated: Use CompressedSize64 instead.
CompressedSize64 uint64 // Go 1.1
// Deprecated: HeaderMap exists for historical compatibility
// and should not be used.
Sometimes improvements are not worth the cost of deprecation and replacement, so things are just left as they are. For example, an HTTP interface designed for request-response HTTP/1 works for stream-oriented HTTP/2, but support for push, prioritization and custom frames is bolted on. Packet-oriented HTTP/3 will add even more stuff that will have to be retrofitted somehow to the old model. Libraries can come and go, but std can't just throw away an old interface and start over.
The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.
• crates that expect to be used for interoperability are often split into smaller crates (like API and back-end, or low-level API and high-level API), so that they can evolve some parts without breaking others.
• sometimes breaking changes are technically breaking, but easy to upgrade (e.g. methods renamed, args reordered). In that case most users catch up quickly.
• in desperate cases, a new version can import its own old version and re-export old structs and interfaces that haven't changed, so they're compatible across major versions.
• proper sharing and composition should be done via traits, so that you can implement a trait for any number generator, not just one version of one implementation.
Maybe in the future we'll see more hacking of libraries (people managing to deliberately sneak exploits in) and in response stronger lockdowns on important library code.
But I think a lot of the value of a large standard library is that it makes it possible to write more programs without needing that first third-party dependency.
This is particularly good if you're using Python as a piece of glue inside something that isn't principally a Python project. It's easy to imagine a Python script doing a little bit of code generation in the build system of some larger project that wants to parse an XML file.
So all you need is one dep that needs Guava version X with method M that is removed in version X+2 (say) and another dep that needs something new introduced in version X+2, and you have a Guava version conflict. That's, Guava releases are not backwards compatible due to removal of classes and methods.
You can sometimes fix this with a technology like shade or OSGi or whatever to allow private copies but it does not always work.
It's also a nice API for dealing with XML.
I had enough trouble using it efficiently that I went and wrapped Boost property tree[0] and can happily churn out all sorts of data queries (including calling into python for the sorting function from the C++ lib) in almost no time.
I was taking daily(ish) updates of an rss feed and appending it to a master rss file but sorting was pretty slow using list comprehensions so now I convert it automagically to json and append it as is. No more list comprehensions either, just hand it a lambda and it outputs a sorted C++ iterator.
Though I probably should've just thrown the data into a database and learned SQL like a normal person...
[1] https://github.com/jazzband/pip-tools [2] https://docs.pipenv.org/en/latest/ [3] https://tox.readthedocs.io/en/latest/
I do think there's a lot of low-hanging fruit where Python could bake something in to auto-setup a virtualenv for a script entrypoint & have the developer just list the top-level dependencies & have the frozen dependency list also version controlled (+ if the virtualenv & frozen version-controlled dependency list disgaree rebuild virtualenv).
I used it to distribute dependencies to Yarn workers for PySpark applications and it worked flawlessly, even with crazy dependencies like tensorflow. I'm a really big fan of the project, it's well done.
When you have no third-party dependencies, then adding the first one requires picking amongst trade offs and lots of work. A subset of choices include using virtualenv, using pip, using higher layer tools, copying the code to the project, using a Python distribution that includes them, writing code to avoid needing the first dependency ...
* You have to document to humans and to the computer which of the approaches is being used
* Compiled extensions are a pain
* You have to consider multiple platforms and operating systems
* You have to consider Python version compatibility (eg third party could support fewer Python versions than the current code base)
* And the version compatibility of the tools used to reference the dependency
* And a way of checking license compatibility
* The dependency may use different test, doc, type checking etc tools so they may have to be added to the project workflow too
* Its makes it harder for collaborators since there is more complexity than "install Python and you are done"
I stand by my claim that the first paragraph (adding another dependency) is way less work, than the rest which is adding the very first one.
But, I do wonder whether it needs to grow. Python is in a stage where adoption of new std library features is inherently slow, not only in third-party libraries like twisted, but also in applications, even if they use newer Python versions.
What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac). This reduces cycle times drastically. Compare this to newr language platforms that people like to install in newer versions on older platforms quite regularly.
Some recent additions would be fine additions at an early stage of language development, but surely not for Python.
It doesn't kill it, quite the contrary, rather makes it ubiquitous. If you want another version you just install virtualenv. It's the same with Perl. We use the Perl version shipped with the distribution (openSuSE) and deploy to that. It's older but it's stable and it works. On our dev environment (mac) we have the same version with all of the modules installed in plenv. We also chose a framework with as little deoendencies as possible (Mojolicious). It looks like it was a great choice.
If only that were the norm amongst long-tail python users. Heck I don't do it; I have the anaconda distribution installed on Windows and when I need to do a bit of data analysis hope I have the correct version of packages installed.
Making this core to the python workflow (bundling virtualenv? updating all docs to say "Set up a virtualenv first"?) is the first required change, before thinking about unbundling stdlib
And it is a totally miserable experience on Windows, every single time.
The reasons are similar, it's a constant drag on core compiler development to need to support various batteries included that most core contributors aren't going to care about, so it's easier to tell people "use CPAN".
There was even talk of "distros" for the interpreter. Where the core bits would be similar to what Linux is, and all the batteries would be provide as collections of add-on packages.
Strangely enough these efforts seem to stop at OS distributors. They really seem to like to install just the one "compiler", and wouldn't stand for a project like Perl or Python telling them "we mean for you to distribute the core compiler plus these 100 packages, because that's what forms our 'language'". "Strangely" because you'd think they'd be the best positioned to make easy work of packaging up such a thing, and it shouldn't in principle make a difference if you need to install 100 RPMs / APTs by default.
I've been using built-in environment isolation tools such as virtualenv for ages but have recently switched over to using miniconda for all things python. Among other things it has amazing support across all three major OS's, and I happen to be dealing with all three at any given time. Whether one uses miniconda, pipenv, virtualenv, or anything else like it, as far as I am concerned the days of ever using the system python are over. I will always create my own personal "distro" on the fly with full control over the python version and every add-on package.
You don’t have any Python scripts in your bin folder?
Not a million miles from the modularisation that Java has been going through.
When did you have to use cpan in a modern system? Compare that to how many times you had to use pip.
Now, if you use a crappy OS or distro (or god forbid, some container built by you have no idea who on top of nobody knows what) then yeah, you are bound to do the leg work yourself, but you will be doing that regardless of the language/subsystem you are trying to use in that case.
Not to mention that it is the only way to do things professionally. For example, if you must have a system that parses XML but for company policy is not allowed to have even the means of performing a network request. With python you either have both xml and an http library and whatever else included and you will either have to do a special package with a striped down python+xml only or get a corporate exception. While on other languages you can install only the xml parser component package and your code will run happily and be compliant with company policy.
Well yeah. A sizable amount of new software is still being written in Python. But when I use Perl software (besides my custom scripts), it's always stuff that's old enough that the distribution is carrying packages for it.
If you disagree, please name a significant new software written in Perl that was released in the last, say, 5 years.
I see your comment as the goal, not the problem.
...wouldn't the company policy involve removing the means of performing a network request from the computer, making the notional capabilities of the software irrelevant?
Python will let you drive network requests through the OS. It's just that you wouldn't normally want to.
That alone have probably paid of handsomely over the years considering all the XSS we patched, which could very well have been full network compromises.
I mean that a significant use people get out of Python and Perl is that they aren't bare-bones like say Scheme or Lua where the standard library is really spartan.
It allows you to write useful code that works on the lowest common denominator of "just OS Perl or Python". Whether that's some random version on whatever Linux distro, or *BSD or Solaris or whatever without needing to write your own getopt library or whatever.
Which is why the "let's ship a bare-bones compiler and have people use CPAN or PyPi" is contentious. In theory it shouldn't matter, and for a lot of shops who install hundreds of packages it doesn't, but it does for people who target stdlib-only, which is a big use-case. Particularly since the people who have that use-case are drawn to these languages.
But you don't have to ship just a bare-bones interpreter to deal with the problem of stdlib staleness, you just need the stdlib libraries to be updatable via package manager, you don't need to not ship a baseline version of them with the interpreter.
That doesn't deal with the bloat issue raised with relatively unused libraries, but if they are relatively unused because they aren't good rather than because the use case is uncommon, upgradability could solve that.
Of course, you don't solve compatibility for versions before the move to upgradable packages, but at the same time if you solve problems going forward you increase the incentive to upgrade.
Now not only do they need to ship a stable compiler+large-stdlib, but they can't even rely on there being a 1=1 version relationship between the two, instead it'll be many=many as users might use multiple library versions with multiple compiler versions.
Well, yeah, but you're not going to have much of a language at all if you optimize for quality of life of the language maintainer.
As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests? Tried it and it didn’t work? Too much bloat? Already got too much on the to-do list?
> As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests?
It is possible (ie. asyncio was separate package). It is slow process though.
To the best of my knowledge, minor version updates in Python 3 have been entirely backwards so far and shouldn't have broken any library code. And as for incompatible changes sick as the transition to 3, then these of course had interface changes in the standard library.
That would be my guess. I would also add that getting through the process of adding a third-party set of modules to the standard library can take quite a while.
This may somewhat explain why lots of successful packages are not in the stdlib: putting them in effectively killed future development until a few years ago. But today that argument is inconsistently applied and I'm not sure if it's a rule worth keeping.
Pypi have thrown out the downloads counter—a huge misservice to coders. Like I got all day to figure out the best libs for ten different features which I only need in passing, so my primary concern is to not pick complete garbage.
So, my solution to that now is to look up Github pages for the libs and choose the one with most stars. As much as I dislike Github for its occasional typical proprietary behavior, Gitlab doesn't help in this case.
A better approach might be to add a core of basic requests-like features built from the stdlib’s existing resources. That would be beneficial to many users and if they need more then there’s always Requests.
I'm not entirely sure what you're looking for, but have you tried https://www.enthought.com/product/enthought-deployment-manag... ?
Edit: example
$ edm envs create tester36 --version 3.6
$ edm shell -e tester36
(tester36) $ edm install ipython matplotlib pyqthttps://python.libhunt.com/ Not exactly curation but it does have rated libs for lots of categories.
Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It's the end of the great async war. Is it perfect ? No. And I don't care. It's one thing less to worry about. For those who know what they are doing, you can still choose and pip install twisted, but most people don't, and that's solved. Before that, just choosing the lib was a nighmare, as basically it's a definitive call. Out it on pypi, even with a "stdlib" tag, we go back to the 200X era. And it was not fun.
And the goal for having things like xml/sqlite/ssl without installing anything makes python very useful in a load of situations where you can't install stuff. Sometime you are offline. Sometime you are in a restricted env. Sometime you are not on your machine. Sometime your security protocol is hell. Don't assume people use Python as we do, from our comfortable dev laptop driven by the knowledge of our craft. Python is used in banks, by scientists, in schools, by kids, by poor people in the third world, by geographers and pentesters. The python user base is incredibly diverse, it's why it's so popular: it fits a lot of use cases.
So I see the benefit of having a side version of official modules we can pip install that can move faster. I see the benefit of cleaning the stdlib of old stuff, like the wave module, Template or @static.
But I'm glad I don't have anything to install to generate a uuid or unzip stuff. I'm glad I don't have to worry about twisted anymore (depiste that I did write a book on the topic !).
Also, pip install is NOT simple when you learn the language. I have to spend some time in the classroom, even with adult professionals, to explain the various subtleties of site-packages, import path, py -x on windows, python-pip on linux, -m, virtualenv, header files, etc. before my students become autonomous with it. Without a teachers, this turn into months of bad practices and frustrations.
You'd have to fix that first, way, way before moving stuff to pypi. I do think it should be high priority actually: it affects way more than pip.
If you're in the flow and trying to hack together something, the last thing you need is to lose all momentum to pick a date time library. I've had this issue tons of times with Node and Rust, where I'm not up to date with the current meta and my 30 minute hack job is interrupted 5 minutes in by having to google which library should I use to do an HTTP request. (I've actually lost interest in whatever I was doing a few times because of this.)
Python's stdlib is nobody's favourite, but when you start to get to its limits, you're probably past your flow state, you've written most of the logic and you can spend some time to replace http.client with requests because the latter is much better.
On a tangent note, I've been trying to find another scripting language to replace Python because I'm not a fan of it anymore (I won't get into it right now), and considering what I just wrote, there's not much that can replace it, as most languages have a bare-bones standard library and if you're not up to date with the current best library to do X, you'll never achieve great productivity.
Have you considered ruby?
It was pitched first as a common low level async loop for other applications like Twisted and Tornado.
Then people started using it directly and the keywords were added.
It's great for the people who think the way asyncio does, others are now forced to use it. I find all of Twisted, Go and Jane Street's Async easier to use.
Perhaps Python is just the wrong language for me.
Wait... what ? No way ! Some of us do you use Python to process wav files. If anything, I'd like this module to be updated, not removed.
Explain? I’ve hated the slow adoption of 3.x from 2.x, and generally how terrible it is to have apps that are 2.x on your 3.x system, and would like to know more about how that happened.
Perl has a much better set of modules that extend standard functionality, which considering how much flack Perl gets for being hard to read, is rather funny. Rather than every new feature being its own independent project, most of the useful modules inherit a parent and follow the same convention, leading to very simple and easy to use extensions. And Perl Core isn't all that great, but it does have some batteries included, and everything else is extended easily and in a more standard manner by CPAN.
Sure, I've found some gems on CPAN, but, having worked on both Perl, Python, and Java at reasonable scale for awhile, I cannot understand all the praise CPAN gets. It's the worst-quality scripting language package ecosystem out there. Even NPM does a better job, and some things about NPM are awful. CPAN might have been the first/only/best package manager for a get-shit-done scripting language at some point, but not any more.
Separately, I agree about modules which extend language functionality (e.g. class systems, async programming, runtime typing) specifically. Perl does pretty well in that area. While many of those language-extension modules really don't play well with any other metaprogramming tools being installed in the project, I don't imagine that any alternatives in other languages do, either. My main beef above is with "simple" (read: not pervasive semantics changes) modules like IPC utilities, HTTP clients, or loggers that don't know how to stay in their lanes.
Search engines are a "cool" technology that have become the de facto way to find what you're looking for. But if there's a lot of content related to what you're looking for, they can suck.
Go to PyPI and search for "semantic version". 10,000+ projects for "semantic version" found. As you go through page after page of different modules related to versioning, the one module you won't find immediately is Versio (https://pypi.org/project/Versio/), a well-documented and useful module which I ended up using. I have no idea how I found this module, but it certainly wasn't from PyPI's search engine.
Now go to CPAN (really metacpan) and search for "semantic version". Yes, you're still looking at thousands of results - but wait! There are only two modules here that look useful: Version::Dotted::Semantic, and SemVer. And the description comes straight from the docs' README, rather than being a short uninformative blurb. The first module, Version::Dotted::Semantic, is inheriting a separate module, Version::Dotted, and adding some extra functionality. Not only does the search page give more information about the module, but the hierarchy makes it easier to find (and later extend) useful modules in an intuitive way. Since the base module's functionality is boring, generic, and simple, it's less likely that people will make 20 different versions of it, so it'll be reused more often and thus remain stable for a long time.
A lot of CPAN's module names have sprawled over time and gotten less useful, but there's still a general convention that you name your module as a hierarchy of what it does (even if it's kind of verbose) and make small, reusable modules, rather than giant modules that are hard to extend. Not all modules measure up to this standard, and there's definitely room to improve, but I think Python modules could benefit greatly from a system like this.
As far as curation goes, PyPI is often filled with cruft. While searching for Jenkins packages, you will come across lots of entries like this: https://pypi.org/project/jenkins2api/. The homepage leads to a GitHub 404, it's only ever had one release, and it has no documentation. This project should probably not have been listed on the main search page, or at least sorted well down the list by default with intelligent filters and marked accordingly. (The "date last updated" and "trending" sorting just results in having virtually no Jenkins-related modules in the results at all)
I’m not sure if Python’s ideal solution is to reduce stdlib and have endorsed packages in PyPI, but it would be an improvement over the current process.
> Van Rossum argued instead that if the Twisted team wants the ecosystem to evolve, they should stop supporting older Python versions and force users to upgrade. Brown acknowledged this point, but said half of Twisted users are still on Python 2 and it is difficult to abandon them. The debate at this point became personal for Van Rossum, and he left angrily.
To ensure things move along: pip has been printing highly-visible "python 2.7 will deprecate soon" warnings for a couple months or so now.
The end of life date is already set: January 1, 2020.
It's open source so I don't know what you're looking for in terms of a formal handover. Yes, I bet Red Hat and others will continue to maintain their own versions past that date.
Edit: based off of all the replies below, everyone understands the validity of what I'm trying to say, but also have fortunately pointed out my admittedly grevious error of not knowing you can just import hashmap from stdlib. The extra step is pretty minimal and not a problem. I'm hoping this is covered in the Rust book.
I didn't know it was included in stdlib (I really thought it wasn't) and feel idiotic now. It is still odd (having a primarily scripting background and not coming from the systems side) that I have to include what seems to be essentially an import statement at the top. I guess it is a lot more efficient that way though. Thanks for pointing out my error!
https://doc.rust-lang.org/std/collections/struct.HashMap.htm... https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...
use std::iter::FromIterator;
BTreeMap::from_iter(hash_map.into_iter()) for (k,v) in &src
{ dst.insert(k,v); }
?https://doc.rust-lang.org/std/collections/struct.HashMap.htm...
That being said the point still stands, I remember specifically feeling it at the lack of included regex. I overall prefer the slim std lib of rust over the massive python though. Especially with the community being pretty good about nominating de facto standard packages like for regex.
Had I been using Java or Visual Basic or even C++ proirly maybe I wouldn't be so impressed as I were.
I think the mistake Python is doing is messing with it's simplicity with decorations, halfass lambdas and stuff. As a newbie you could understand Python code, while eg. C++ templates were magic. You need to know more Python to understand Python code nowadays.
There are other new stuff that might be confusing to beginners (like our mighty walrus operator), but those two examples have been there for effectively forever.
I just had the feeling there's more "stuff" that you need to know.
In Python's defense, this was not obvious at the time; my understanding is Rust came to this approach by looking at the experience of Python and other languages. When Python's standard library was first being written, there were no easy package managers for any language, and the normal thing to do for installing dependencies in e.g. C was to grab random tarballs and figure out how to build and deploy them yourself. So avoiding that process made perfect sense.
Obviously it doesn't apply to everyone, and it certainly doesn't apply to most startups or open source developers, but I spent most of the last 20 years working in environments where you have to get permission for every third party library you bring on to the network. Many networks were essentially "airgapped", so it's not like you could just ignore the rules. The bureaucratic process alone meant that we preferred large bundles like Anaconda or Qt. Trying to use Cargo as it is typically used and documented would be a complete non-starter.
We do have an internal PyPI mirror (with devpi) and we point `pip` at that, and it works pretty well.
(Pointing at an internal mirror is now stable. Setting up that mirror is the hard part.)
Situations like this will really make you appreciate "batteries included". I think this particular issue is fairly revealing of the attitudes common among programmers of different languages. I think it's a good thing to be skeptical of a program pulling in a bunch of standard libraries over the Internet. It worries me when I find something on Github I want to try and I can't download and compile it without it pulling in 30 or 100 other libraries that I haven't looked at or decided to trust come along for the ride. I don't like that way of doing software, and unfortunately it's the norm in node and starting to become a norm in Rust as well. Real security fails have been caused this way in node's case at least.
Even in cases when Python programs depend on external libraries, I usually don't need to use pip for anything because Python programs will pull in dependencies provided by your distribution just fine. (My distribution doesn't even have any Rust libraries, so even if dynamic linking is possible in Rust not many people are shipping software that way.)
"Download by default" is a worse way of doing things, and it makes me sad to see newer languages like Go and Rust embracing it.
So Rust can get away more easily without having support for things like command line parsing in the standard library, because it isn't really trying to support situations where it would be inconvenient to give your program its own project directory and Cargo.toml and all.
If it really matters, then do it right (in whatever language makes the most sense).
Wasn't perl's CPAN developed around that time period (mid '90s) ?
Yes it does. Perhaps you're thinking of browser JS, which does not.
This is actually an interesting paradox to be in, and one that Linus Torvalds recently commented on. His focus, like Guido’s, is the user and even fixing a bug can break the user.
That's how you get an inconsistent mess that never evolves. There's something called semver, increase the version number and do the fix / refactors / radical redesign / whatever. People will see that you've went from version 1.0 to 87.3 in one year and they may choose not to use your thing because you're moving too fast for them, but that's life...
Amber is nothing short of an open source hero, having brought Twisted, one of the best open source projects in the world, to new heights. Her insights are as important as anyone in the python community, and after six consecutive PyCons sprinting at the Twisted table (including literally in a chair with Amber to my left and Glyph to my right earlier this month), I consider Amber's voice to be one of the truest and clearest among the leadership of the language into the future.
Amber and Guido are both beautiful human beings.
In the dispute that is the topic of this blog post, Amber is basically totally right. Moreover, the distinction has less to do with any kind of nagging python 2 holdover than this article suggests. The standard lib's role as a place where code goes to die is a view that is widely held and accurate for many cases.
The following question went unanswered during the Steering Council Q&A:
"Every feature request has a constituency of people who want it. Is there a constituency for conservatism and minimalism?"
...and that's really what this whole thing is about.
When it comes to constructive criticism, I think Amber did a good job with her criticism but can do better at the constructive front. Her problem statement was spot on and I agree that the direction she proposed is a good one.
However, to separate the standard library from the core is probably even more dramatic than the Python 2 to 3 migration. Is that what the community can afford at the moment? What's needed to make the transition? What's the opportunity costs? i.e. what other developments we can do for a bigger impact? What are the pros and cons?
My problem is on the like part.
Where shall we draw the line and how do we decide? To me this is a far more interesting discussion. (Maybe it has happened. I don’t go to many conferences these days so I might be missing something here. )
She mentioned http.client vs requests, datetime vs. moments etc, which are also quite correct to me. How about the cgilibs? Or pickle? Or the collections? Or unittest? Stay or go?
Lastly, the title of the talk can be tempered a bit. No? We all know what a leaking battery mean right? Toxic.
Why does there need to be a line? As long as the package manager is part of the core distribution (even if it is itself an upgradable package) why not moving everything into packages, even if some are maintained by the core team and have the stable version at time of distribution release included with the core distribution—but perhaps installed only on demand?
pip install unitest? Oops I spelled it wrong wonder what I just installed?
Kidding aside, most of the problem with the fake libraries could be solved if pypi namespace them per author, like GitHub does for repos.
“have the stable version at time of distribution release included with the core distribution”
Ruby, for instance has both “default” and “bundled” gems with the core distribution.
Come on. Python has had the 'batteries' metaphor for decades and this is a straightforward and obvious play on it. You're bringing 'toxic' into this which I suppose is technically and biochemically correct (both, surely, the best kinds of correct) but has far, far more negative connotations than the title warrants. You're having to work really hard to make a generic thing sound dreadful.
To be honest, I think she might be biased here given that she maintains a competing package.
I hope that you take time to reconsider this view.
We're not talking about competition in the same sense as in a capitalist system, between two companies.
Prior to asyncio, Twisted maintained the only viable flow control for serious asynchrony in python. Put another way, python had no standard flow control for serious asynchronous abstractions in the standard language (and standard library) before asyncio (and `await/async def`, etc) landed.
Nobody is saying that the syntactical changes to python belong in a separate package on PyPI (cue Gary Bernhardt's Pretzel Colon). These things are fine.
But `asyncio.Future`? Yeah, I see a very reasonable argument for that stuff (ie, the asyncio namespace) being in a separate package.
But OK - looking again at the "competing package" narrative: now that these things have landed, Twisted has done an amazing job of using them alongside all the other tooling that Twisted also provides, most notably its test infrastructure.
Amber doesn't stand to personally benefit from asyncio failing. To the contrary, having the flow control taken care of so that Twisted doesn't have to be its sole brainparent gives her much less free work to feel obligated to do.
Could you provide a reference/explanation for this? Thanks in advance!
[0] https://www.destroyallsoftware.com/talks/wat
[1] https://www.destroyallsoftware.com/talks/the-birth-and-death...
[2] https://technology.customink.com/blog/2015/06/08/ruby-pretze...
And it wasn't the only thing at the language summit that proposed expanding that approach: there was a discussion of carrying time zone updates in the same way, by shipping something with the interpreter that works but allowing updates from PyPI. http://pyfound.blogspot.com/2019/05/paul-ganssle-time-zones-...
So I think the development community / target audience at the language summit already understands the pros and cons of the suggested approach and the technical route to get there. (For an end user, my guess is the experience will be that anything in the Python 3.x standard library today will still be in the Python 3.x standard library, but you'll have to `pip install` a newer version if you want more features, and you get the benefit of being able to `pip install` something from the standard library where you previously couldn't.)
What do we know? Software configuration management is a heinous problem.
Python gets it more correct than most.
Let us rejoice, be patient, and respect everyone's good-faith efforts.
Amber wants to remove asyncio from the standard library. That's minimalism but not conservatism.
Also note that one way of interpreting her proposal is "generalizing the 'ensurepip' model" to keep things already in the standard library in the standard library, but move feature development externally and make it easy to upgrade packages in the standard library. (Then Twisted, which is also installed externally, can simply depend on a fixed version of a standard-library package.)
I agree with a lot of her arguments. I never understood why tkinter was included (I could see why it might have been added years ago, but before Python3 came along it felt obsolete and unused). I've had to support old versions of Python along with their bugs, and it sucks.
But languages are about choices. Python's syntax and decision to use whitespace is a strong choice--so is their stdlib. I've been on projects that saw performance problems with etree, moved to lxml, then had to go back to etree because of missing features. I've spent a lot of time looking over arrow, dateutil, and moment because datetime seemed inadequate. But I like that there is a thoughtful default that serves many needs. A lot of these examples, like requests, are built right on top of stdlibs--so both would be needed even if it was shipped along with stdlib. I'm ok if code goes there to die because I would hope that due diligence was taken when it was included.
I kind of agree with Guido for a lot of this. I can't wait to move from Python 2 and have looked a lot of the new stdlib and looked for backports. I've looked at twisted and alternatives and I'm so happy something is built in even if its far from perfect.
Perhaps, but Guido's response was shitty here (and purposefully refusing to get the point), whether one think Amber was right or not.
I deal constantly with the shitshot that is Python packaging (and lack-luster default packages) and I happen to think she is 100% on the spot.
Did asyncio module feel bloated? It certainly did. It seems like every module from subprocess to networking to io is crammed into it.
On the other hand, did it get the job done without resorting to any packages or threading? Yep, and that is pretty powerful and rare.
I think bloatedness of the stdlib isn't actually a practical problem. It's just an inelegance that you kind of have to learn to tolerate.
I maintain a Python 2 & 3 compatible project that has no external dependencies.
fwiw, six can easily be vendored into a project to avoid the technical external dependency. that is how we manage it for kafka-python.
there are a few modules that are simply at different locations but have the same API
if PY3:
from http import client as httplib
else:
import httplib
PY3 = sys.version_info >= (3, 0)
PY2 = sys.version_info < (3, 0)
PY26 = sys.version_info >= (2, 6) and sys.version_info < (2, 7)
isinstance(<maybe_string>, basestring if PY2 else str)
# Python 2.6 doesn't properly UTF-8 encode syslog messages, so it needs
# to be performed in a custom formatter.
formatter_class = UnicodeLoggingFormatter if PY26 else logging.FormatterAs the maintainer of WebOb/Pyramid/Waitress we have a compat module that contains all of the changes/renames/functions to help with the Python2/3 compatibility and all tests run across both platforms.
Are the functions borrowed heavily from six? Yes, but we don't need to vendor all of six.
After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss.
Does writing python 2 code stop being 'worth it' at some point? If so, where is that point? It sort of sounds like you're already there. There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no?
I'm not a professional python dev, so it doesn't impact me as much as some of my colleagues, but 'backwards compatible' issues crop up in other platforms/langauges as well. Wordpress might be the biggest example in PHP. They've kept a minimum target language which is far behing 'current' or even 'currently supported', and it's been a catch 22. Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4. WP 5 was, alas, a missed opportunity to target PHP7 as a minimum. :/
How do you know I'm there? If nothing else I still have my own previous Python 2 code that I've spent time on and that's useful to me! Why would I just throw them all away or waste massive amounts of time rewriting them into Python 3? Is my goal supposed to be to please the masses here?
Honestly that's sufficient reason already. But if you want a more "standard"/politically-correct response: Python 2 is still being supported, Ubuntu 14.04 LTS literally only just reached EOL last month and can be found in the wild, Python 3 support is lagging in a lot of places (e.g. PyPy is still on 3.5, although I'm just using it as an example; I don't use it much), and I still come across Python 2-based tools and packages once in a while.
So I guess the answer to your question is it "stops being worth it" when I stop coming across situations where I'd regret abandoning Python 2.
> There's still people using P2 at least in part because of all the hoops people are jumping through to keep supporting it, no?
"Because" is an odd way to put it... it's true that if they didn't support P2 then people would use P2 less, but people use it because it benefits them, not "because it's supported". I mean, you're also alive "at least in part because" of all the hoops people jump through to grow and bring food within your reach, but I doubt your conclusion is that this should stop being the case...
> They've kept a minimum target language which is far behind 'current' or even 'currently supported', and it's been a catch 22.
What I don't understand is why are people supposed to keep abandoning good software just because someone made something shiny and declared it "unsupported"? I hate this "you have to like my updates... or I will force you" attitude that every organization seems to have nowadays. People are trying to solve their own problems, not please the leader they're following.
> Hosts keep supporting PHP 5.4, for example, far later than they 'should' have, because people kept writing new stuff targeting PHP 5.4.
I mean, this isn't even the same situation? I'm not targeting Python 2 or introducing a dependency on it generally. I try hard to keep my code both Python 3 and Python 2 compatible. So the decision as to whether to move on or not is left to the client (which is often myself) and there's no obstacle either way.
Part of the argument for using much of the stuff out there is because it's "supported" (commercial, community, etc). Without some degree of support, things dwindle/die. The 'support' for language stuff may simply be security patches. If/when those stop being provided, you're starting to do a disservice to people to continue to push that language, even if the core functionality you need is still working and perhaps even unaffected.
> I'm not targeting Python 2 ... I try hard to keep my code both Python 3 and Python 2 compatible.
Given that there are breaking changes between the two, I can't read this any other way that that you're 'targeting' Python 2. If there's a way to do something in 2, and it's not working that way any more in 3, and you're making code so that it runs in both (weird syntax, translation layers, etc) then... you're targeting 3. The same way that if you use 3 syntax, you're 'targeting' 3. That's just how it is.
> but people use it because it benefits them, not "because it's supported".
The benefit is that they don't have to go through whatever hoops it takes to upgrade to have the latest versions. In many cases the later version may not add much directly, but 'community support' and 'security' are two ... emergent properties of a critical mass adopting new versions and dropping support for old versions.
> How do you know I'm there?
Wild-ass guess from the gist of all your earlier posts in this thread?
> But if you want a more "standard"/politically-correct response: Python 2 is still being supported,
I'd say that's partially 'under duress'. My reading of the situation years ago was that the Python community - at least the leadership - wanted to EOL P2 on Jan 1, 2015. There was a gigantic stink/pushback, and support was extended for another 5 years. I will grant that the v3 switchover was pretty bad - I'm not a pro user, but had some colleagues that dealt with a lot, and certainly, 8+ years ago, it was pretty hard to just 'switch' running systems. Even developing new stuff from scratch - there were a lot of 'only works in 2' libraries that were entrenched standards without clear update paths. However, rather than doubling down and making 3 a more attractive proposition (work on upgrade path, faster language, better docs, whatever), the agreement was to support P2 for another 5 years. That's an eternity in the tech world. You could argue it shouldn't be, but it is. Yet, from python colleagues - those who've stayed - the upgrade story isn't significantly better. I'm taking their word for it, but based on your posts, it doesn't seem to be either.
In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.)
To your specific pain points:
> Unicode stdio?
Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.
> Unicode file paths
This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier.
Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions)
> Unicode sys.argv
This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does".
1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else?
> I feel like Python 3 completely wrecked strings instead of making them better.*
A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.
> Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.
Does io handle stdio? I was referring to standard input and standard error/output. How do you read/write Unicode in a cross-2/3 way from/to standard input/output/error without adding your own translation layer?
>> Unicode file paths
> This is going to be a mess in any language, because file paths really aren't text.
Sorry I need to be more clear. That general mess is not the aspect of it I was referring to. I'm specifically referring to a 2/3 compatibility mess.
I meant that, for example, to have any semblance of Unicode handling, in Python 2 you do os.listdir(u"."), whereas in Python 3 you do os.listdir(b"."). I know how to handle it in both Python 2 and Python 3 in a way that's Good Enough (TM), but how do I even get that with cross-compatible code? I'd need to write a translation layer of sorts for every single I/O function I might use.
>> Unicode sys.argv
> I think I'd say "do whatever 3 does".
Hmm okay thanks, I'll need to try to see what the implications are again. I think the problems I recalled from this may have been just been a result of the other issues, not sure.
I can't "forget about 2" though, it's still on Ubuntu LTS systems and there are still packages in 2 that haven't been ported to 3.
>> I feel like Python 3 completely wrecked strings instead of making them better.
> A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.
Again, I think this is "cleaner" in theory, not in practice. What happened to the string_escape/unicode_escape nonsense I pointed out with the new system? Any rebuttal to that one? ;-)
No. No on just so many levels.
It finally fixed it by introducing two (or three if you count bytearray) types for completely different semantics of data.
Python 2 was just a mess whenever you left ascii.
But looking solely at python3 , everything byte and unicode related just got so much easier and better testable and with better error messages at better points in your code than in python 2.
So claiming python 3 broke stuff just not what the case is.
Python 2 had a deeply flawed, boundryless view of unicode vs bytes and python 3 fixed that.
What is your reason for using this obscure encoding anyway?
Yes and my entire point is that this makes zero sense when we're talking about escaping.
> What is your reason for using this obscure encoding anyway?
Seriously?
The encoding `unicode_escape` is not about escaping unicode characters. It's about python source code. It's defined as:
> Encoding suitable as the contents of a Unicode literal in ASCII-encoded Python source code, except that quotes are not escaped.
It makes absolutely no sense to have escaped unicode characters as actual unicode string. If you really need that, a version that also works in python 2 would be:
u'hellö'.encode('unicode_escape').decode('ascii')Absolutely no sense? So is a basic Python expression evaluator like this complete nonsense to you? #!/usr/bin/env python3
try: # Python 2
from Tkinter import Tk, Entry, END
import tkMessageBox as messagebox
except ImportError: # Python 3
from tkinter import Tk, Entry, END, messagebox
import ast
def my_eval(s):
# assume I've implemented this functionality manually...
return ast.literal_eval(s)
root = Tk()
e = Entry(root)
e.insert(END, '"Hell\\xc3\\xb6"') # Assume the user typed has typed in a Python expression, not me
e.pack()
root.bind('<Return>', lambda evt: messagebox.showinfo("Result", repr(my_eval(e.get()))))
root.mainloop()
Note that that wasn't even my choice! That was the choice of the built-in Python GUI toolkit... distributed by the same folks who decided this string/bytes overhaul was a brilliant idea...
To be fair, one of our three dependencies does in fact depend on six, but we don't use six anywhere in our code. This dependency of ours is used only in one specific area of the codebase, and in fact I (the person who added the dependency) wish that we didn't have to use it at all.
Tkinter needs to go...There is very little reason except for the legacy ones, why it needs to be there still...
Taking tkinter out of the standard library won't stop IDLE being built with it, if they chose to do so.
Just move tkinter like what they do with typing package is good enough to address everyone's need.
Coming from Visual Basic, and similar things (HyperCard, Turbo Pascal), being able to run a program with GUI on both Windows and Linux with no code changes was nothing short of amazing to me. So maybe that's one way to draw the line with batteries, namely to make it so a beginner can get started with Python, with no regrets about which OS they're using. This requires some out-of-box hardware abstraction, where they keyboard, mouse, and display, all effectively count as hardware.
Like if you were in a resource constrained situation, you would be customizing your python install anyways (if nothing else, getting rid of the docs, the tests, maybe stripping out all but the compiled code...).
Being able to add items to a canvas, attach callbacks to them - without tracking them yourself, setting up the infrastructure for indexes, bounding boxes, all that stuff - I haven't found anything else as easy as Tkinter's Canvas.
If there is something better, I'd love to hear about it.
https://cairographics.org/pycairo/
You do have a good point about Tk's canvas easy input handling though, which is something Cairo doesn't do. It's just an immediate mode drawing api like PostScript's rendering model or html canvas 2d context.
You can pass a pycairo context into C++ Python extensions, and they can go to town with it quite efficiently.
https://github.com/SimHacker/micropolis/tree/master/Micropol...
https://github.com/SimHacker/micropolis/blob/master/Micropol...
If you're worried about people not RTFM when using your projects, you could always start your scripts with the standard try/import/except wrapper around the required package, and tell them to run pip install when it's not found--or (and this is a terrible idea) run it in a subprocess for them.
There's always going to be a good debate about which batteries should be included. I'd almost put numpy ahead of Tkinter, but maybe that reflects my bias for scientific programming.
However brown has solid good points.
The brilliant selling point of python is the massive standard lib. If the quality of the libraries fall, then python's use as a tool drops dramatically.
One of Node's massive failures is that is has no standard lib.
I've been astounded how few non-trivial packages are actually in a consumable state, and how many seemingly-simple packages have N dependencies that pull in M more. By and large it's a zombie wasteland of cruft.
I wouldn't really trust only-crowdsourced ratings, but I think that might be a nice component for npms.io to include, perhaps. Stackoverflow answers, for example, seem to be directionally correct if you sort by upvote count.
Guido should stop acting like a child. Listening to people, hearing them out - even when it’s uncomfortable - is the mark of a good leader.
I tell new PMs “this is the best job in the world 90% of the time but the other 10% is eating shit with a smile”
There is always a new person coming with a new idea. It's exhausting, because it's required to make the language evolve, but it's also a new opportunity to screw things up evertime.
And if he had let most people got their way during the last 2 decades, python would have ended just meh.
Of course, everytime a new debate starts, everybody thinks that this time, just this time, he is wrong and they are right. I did too. We are all part of it.
I get the reaction. There is a limit to what a person can take, and it's why he stepped down as a bdfl.
But seing your baby and your reputation at stake is hard.
The language itself is inferior in expressiveness and performance to almost any other modern language. The only reason anyone uses it anymore is the network effects of the library are very strong, especially is fields relating to ML and data science.
Python didn't have any specialty like PHP, or an accidental monopoly like JS. It didn't come with a killer app like Ruby. It hasn't been made by a giant company like Go.
It's pretty much a self-made language.
The inherent qualities that draw people to Python are that it’s so inexpressive and crippled that you can learn it in 30 minutes.
You have a lot of contempt to offer, I'll let you with it and get work done.
They don't have the desire to learn what a monad is, they don't want to type variables when exploring a badly formated heterogenous data dump, and they do want to be able to read the code of their intern freshly out of school once he leave without writing down any doc.
I remember one of Go's core devs voicing some of the same concerns regarding the SMTP-library [0] a while back.
If a package is really niche, it may not make sense to put in the integration work. But for a package that is used by a significant majority in a general application - why would you want to keep it separate if it is so much better?
I am ignoring human interaction here - there are probably of dozens of answers to that question if you count personal motivations.
Seems like a false dilemma to me. The core team could still maintain "blessed" packages that don't ship with the default installation.
> why would you want to keep it separate if it is so much better?
This is addressed in the article, most of the 5th paragraph is dedicated to it.
And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.
There's definitely tradeoffs with package/dependency multiplication, but I don't think standard libraries are as clear cut as people make them out to be.
That depends on what you are writing.
For an application that is deployed stand-alone, you'll likely fat-package it with python and all the libraries. In case of a security issue, you create a new version of your application that bundles the fixed python.
For an application that is deployed on the system python (more typical on Linux), it's the system admin's task to update the system python.
As the user of the stdlib, it really depends on what the security issue is. For most users, if they can verify they're not using module X, then there's no problem and no rush for them to upgrade. In a language like Python that's very dynamic, you also might be able to download a hotfix .py file provided by the language maintainers (or any other party) as an alternative to upgrading, or as something for the interim while you wait for the upgrade to get released, backported and released for older major versions, and available for the deploy environment -- this might even take the form of an iptables rule, depending on the issue. In short my point is that there are alternatives besides bumping one release's release version and hoping people upgrade.
> Except, users don't upgrade stuff.
Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.
For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.
There's also another party of users worth mentioning, those that don't upgrade specific things because past upgrades have been a terrible experience. This is a reputation issue.
> And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.
Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.
What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.
You might be interested in this talk that makes the case to stop breaking your API especially if you're requiring more and providing less. https://www.youtube.com/watch?v=oyLBGkS5ICk
Sure, some people definitely upgrade. But there's a big difference between requiring someone to actively hear and heed your warning and having them automatically upgrade whenever they create a newer project. Or even just prompting them to upgrade like NPM does for package vulnerabilities.
> For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.
That's not the point. If your standard library is a package, any new project will automatically download the latest version of the standard library. If I have a vulnerability in an npm package, I simply push a new version and people will automatically download that new version when they create a new project. If there is a vulnerability in the Python standard library, users will continue to have that vulnerability until they actively patch it.
> Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.
Not a canonical one. They're starting to add a new promise based API, but now we have two different APIs doing the same thing. What if we come up with a newer way to do async? Do we add yet another API?
> What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.
That's a bit of a straw man. With packages, you can still support multiple versions. It's just that fs would no longer be tied to Node. You could make a newer fs package, fs 2.0 or whatever, that uses promises. The older one would still be supported, of course, but it wouldn't be stuck in Node.
Unless, and maybe this is a source of disagreement, it's still the standard in Node land to specify dependencies as "whatever the latest version is, I don't care"? That approach has teeth. After you've been bitten a few times you learn to say "no more" and version pin. There are other nice benefits to version pinning too (like reproducible builds). It's pretty common in Java land, even though some people still specify "version x or higher". It's a tradeoff.
One downside is that if you don't occasionally check maven central or wherever for new versions you might miss a security issue, and if you're copying dependency specifying files over to new projects the issue will persist. This is solvable with tooling -- http://www.mojohaus.org/versions-maven-plugin/ comes to mind and I was pleasantly surprised to get an email from Github that even a toy project of mine pointed to a version of a JSON parser with a known issue.
> What if we come up with a newer way to do async? Do we add yet another API?
Sure. Especially if people were just fine using the older API. For the fs situation, putting it in its own package would be great, up until the point I have to modify years-old code that's been working just fine through upgrades until someone decided to break things... for JS that's even more intolerable since nothing in the type system forbids the function updating to accept a promise or a callback. That (as would be a new API for async) would be a non-breaking change, it only provides more and the requirements for old code are the same.
If it's possible to break up the core in a way that preserves legacy code, I'm all for it. Java's Project Jigsaw did this to an extent with its module system. Another way to help that is to separate what's standard (such as with an official ANSI or ISO standard) for the language and what a particular implementation of the standard bundles with its releases on top of the standard. I recall at one point the old IO fork of Node promised to never break "core javascript APIs", but the industry consortia that defines what that means makes me think it was more a hollow promise.
To take an example from a language that has official standards, C, across decades of versions of C compilers and standard libraries from various vendors, if my code is C89 then it compiles with compilers that support C89. People might be offended that a function like strcpy exists, but it's there, I can use it if I want, or a separate library like bstring, or maybe the compiler provides their own __builtin__strcpy_chk that I can use directly or switch between with a flag (_FORTIFY_SOURCE). This might end up being less work for the compilers, too, since they don't need a security bulletin saying "some uses of strcpy can lead to a buffer overflow, removing it in v2.1.2, update immediately to strcpy2". They can (and did) just provide a new strcpy2, made available a seamless upgrade (I believe even Node has done this with some shims for deprecated APIs) if you don't want to change the source text, and warn people about the old one.
Perhaps I'm misreading this, but isn't that the purpose of package managers? I can run an `npm install foo` and update it at will.
I have to develop enterprise software that runs across a wide range of platforms, and being able to take advantage of the fact that Python is pre-installed on all of these systems with its standard library is a godsend.
https://www.google.com/search?q=site:golang.org/pkg/+"frozen...
Third party package problems will be an issue at some point but that’s more due to them be so focused on minimalism they don’t have clear guidance on setting up and maintaining packages.
The primary reason Go can get away with this strategy is because the Go community actively promotes fewer dependencies = better. So if you write Go you have to often accept the fact that the second you add a 3rd party dependency that you're now officially on your own if that dependency breaks or becomes unsupported.
This is not necessarily a bad thing. But in order to move software forward I still think we can do better than to push this responsibility to all individual end users.
This is one area where I feel like most popular languages today still fail compared to CPAN. CPAN's value was not just packaging and distribution, it was an integrated test report pipeline and infrastructure, actively managing and gatekeeping of library maintainers, CPAN mirroring functionality, and easy acceptance of bug reports and user feedback against a library.
My experience with Go has been equally pleasant (though it does require more boilerplate for commonly accepted reasons in the community), but in no way unique.
Why? Not only Go. Clojure as well. Clojure's standard library is incredibly consistent and stable. Documentation pretty dry though. That's the only complain I can think of.
I've never liked asyncio, while Twisted felt natural to me. So I would agree that an inferior solution has been pushed heavily in the stdlib and also to the syntax level.
Moving the entire stdlib to PyPI is of course entirely foolish and would destroy Python.
In certain situations, I've been working with a python interpreter on a RHEL machine where pip was not installed (and I was not allowed to install it as well as make other modification: the machine was owned by the client and I had to work with what I had available).
- having some basic functionality in the core libraries was a godsend because I could work with that, even though it was not "ergonomic"
- not being ergonomic, it was a "poor experience" (and certainly not optimized or anything nice to see).
This heading is the essential problem in innovation writ large: some giant can ignore you and squash you without any effort at all, without even considering your existence.
Right now the equilibrium in this tug-of-war is that a certain set of functionality comes by default in the python standard library and everything else is just a package that you can install.
Obviously from a dev point of view it's a hassle to have to decide which of two or more packages for X is best, the pythonic way would be that there should be one and only one package for X. Of course at the cutting edge there have to be competing packages because there needs to be room for innovation.
But obviously not everything can be in the python standard library.
Not really sure what the solution is, but maybe there should be tiers of packages, with "Tier 1" being standard library, "Tier 2" having some kind of official stamp that it has been security audited to a certain standard, that Python has some control over who gets to modify it and why etc. Then maybe "Tier 3" could cover everything else, i.e. any bob random can go make a package on PyPi and it's Tier 3.
In addition, the process of going from Tier 3 to Tier 2 would give people a chance to winnow libraries down to one way of doing each thing at the Tier 2 level.
This might not be realistic but it's what my gut is telling me. C&C welcome.
People leave all the time during talks, and I've been to talks where most people left as soon as Q&A started.
PHP has a similar issue to this. The people writing C were not using the language. The best example is PDO. A lot of C was written, but it was essentially abandonware because the PHP users could not make any changes without getting the C maintainers to both agree and have the time.
Put another way, the whole Python universe from my point of view has become a cautionary tale about breaking changes. Given Python's popularity, this might be an unpopular opinion, but I have yet to find someone who loves Python who still loves it as much when they discover other newer languages (I'm sure you exist, I just haven't met you!).
Python is having its time in the sun really because it is a default install for most unix distributions, so even people stuck in government labs can use it because 2.7 is already installed. Even apt depends on it via the debian software-properties package, so it isn't going anywhere any time soon.
The real question is how many people would use Python if it was as little known as, say, Elixir.
I think this goes to her point of Twisted wanting to support really old versions of Python and they would be a lot more comfortable not supporting really old versions of packages.
Force the upgrade already. Code still on 2.7 if still useful can be upgraded.
My "favourite" library quirk. socket.fromfd is only available on "Unix" on Python 2.X, that was fixed in Python 3.X
The worse offender being the logging library. It's the least pythonic thing in the whole std library (ok maybe ABC is worse, but oh well)
https://pythonclock.org/ has Py2 reaching EOL in 7 months. Realistically I'd say the time passed years ago.
py3 is over ten years old now. We're not talking about some new unstable piece of kit, I'd imagine that a large percentage, perhaps even 50%, of the HN audience started their career after the transition had already started.
Years ago I was in contact with the author of Nuitka, who was very excited to share his work thus far. During his presentation, Guido kept huffing and making snide comments under his breath. All because he disagrees with the premise behind Nuitka.
I like Python, and can appreciate his work and contributions. That said, I can't help think the community can become less toxic without him as BDFL.
[0] https://mail.python.org/pipermail/python-committers/2018-Jul...
I know he gave up the title, but as he personifies Python, I imagine the influx of requests for his attention may not have subsided much.
This is not meant to speculate on van Rossum’s health, just trying to point out that there are situations where apparent rudeness may be out of the person’s immediate control.
(Also, personally, if I were behaving poorly I would much prefer people to say "hm, 'geofft is being rude but I know he's a better person than that and can improve" and not "hm, 'geofft is being rude, that's just the way he is, I wonder if he's got mental health problems.")
Basically agreed.
> If you're sufficiently in control of your actions to keep developing a major programming language, you're sufficiently in control of your actions to [...] not be rude while doing so [...]
That's a lovely thought, but no. When I've had bad times, my social skills and emotional cope would sometimes go to shit and leave my technical skills intact.
People are screwed up. You are, too. I hope your business partners and allies don't hold you to the standard you're bearing when your time to behave badly comes.
let's hope everything settles smoothly :)
(This was years ago, and i don't have the email exchange anymore, so i'm doing my best to describe it from memory)
I reported a bug I had debugged pretty heavily and believed was likely a bug in the appengine datastore (this was before it was publicly available, IIRC), with a fairly detailed repro recipe, etc.
(I was not on the appengine team, just building an app)
I had debugged all the client side code all the way down to the rpc to the datastore server and was positive there was nothing weird going on there at all.
Within a minute or two of me sending the email to the email alias , he replies with "This must be a bug in your code, that can't happen". He didn't even look at it (i looked at the logs)
I replied with "I agree it should be impossible, but if you could look at it for a second, i think you'll see that it's not and is actually happening. The code is very simple, etc".
He says "I don't have time to fix your code".
So i spend the time and reduce it to a simple, 20 line piece of completely obvious code (IIRC i believe it had no real code except to instantiate the class and store it in the datastore) with no dependencies.
and say "here, i took the time to try to make it as clear and obvious that this is really not a bug in the code, because there is no real code here"
He replied with something else abrasive and dismissive.
Then, about 20 minutes later, one of his teammates replies with basically, "oh shit, this is bad".
(Because what i had discovered turned out to have caused data loss that they couldn't automatically fix. The could get the data back from restores, but it wasn't clear what to do with it, you needed intervention from a user)
Honestly, it would have been better for him to not respond at all. (he was not the on-call team member at the time anyway)
We all have our bad days/times of course (i definitely did!), so i do hope he's found more inner peace than he had back then. But yeah.
I did observe that you first provided a "fairly detailed repro recipe", and then after he rebuffed you, you provided a "simple, 20 line piece of obvious code". What Guido should have done was ask you to make the extra effort to provide the latter case if the "recipe" you provided was in fact too much effort for him to look at, for something that didn't seem to be a bug to him. That is, he wanted you to "spend the time" as you said, and if he had more experience with this kind of thing perhaps he would have known to just ask for it rather than dismissing the whole thing.
What happens to me, at least, a lot, is I get bug reports that are like "here just unzip this 10M attachment, install theses libraries and datafiles and then watch the log output for the thing I spent five paragraphs not really describing". These are not coworkers or customers of mine, for whom I might be obligated to go through all those steps for, they are regular users who have downloaded my software for free (as they should), and likely saving their company thousands or even millions of person-hours of work by doing so. The thing I ask these people in return is that they A. report issues to me and B. do as much work as they can to help me fix the problem - I'm not a concierge, the help process is more of part of how the give and take of open source software is supposed to work between parties.
I encourage these people to please pass along an MCVE, e.g. the most succinct demonstration script possible, and quite often when I get the sense that they don't really have the experience to know what I'm looking for (despite my sending them the link to what an MCVE is), I will often read their verbal description, then write my own MCVE in about one minute that shows what they are asserting is not true, and then I paste that into the issue; I have a Python fragment script that I use as a starting point for writing 90% of these test cases. I ask them to please modify the MCVE to show the thing they are actually trying to do. That's how I get them to send me a succinct problem description that isn't a huge waste of my time.
I understand this is likely not at all what happened in your case as you were both at Google and I'm sure folks there are more sophisticated than this. I just had the thought based on how you had sent two versions of the issue.
FWIW: the detailed repro recipe was hermetic and single-binary (and guaranteed to work on his machine for various reasons), it just had more code than strictly necessary (It was still <500 lines, it was a very simple webapp).
The actual recipe was closer to "run this binary, click on new record button, click save, observe results in datastore".
I actually reduced it not because the recipe was too detailed, but to remove the argument that it was my code.
But everyone has their limits.
A core skill of any open-source project maintainer is recognizing when you're burnt out enough that your continued presence isn't helping the community and is just getting in the way of others, and stepping aside. If you can't do this, you're not a good maintainer. I'm happy to have sympathy for your personal problems, but you're still failing at your job. Here, for instance, is a short but effective way to do it: https://mail.mozilla.org/pipermail/rust-dev/2013-August/0054... (That was almost two years before Rust reached 1.0, and the language hasn't suffered one bit for it.)
And part of Amber's point is that core Python development is taking on too many burdens for no actual benefit. By bundling so much in the standard library and expecting the standard library to be usable for real work without installing additional packages, the core team (and Guido in particular) increases their own workload, impedes the ability of others to contribute, and produces worse results for end users than they would otherwise.
Sure, if your definition of success is being used by as many people as possible, but there are other (more) important criteria for asserting the quality of a language.
When you have a vision for your project, you might be afraid that other people are going to ruin it, because they don't understand, or they don't have good taste, etc.
Clojure sort of had this moment recently, where various people in the community were unhappy with the language direction and Rich Hickey (who is a person who cares about vision and taste in a language) made it quite clear that addressing their problems directly wasn't his definition of success. He was building a language for himself / his company to use, and if it worked for other people, great, and if they wanted to contribute to his vision, great, but if it didn't work for them, they should not expect Clojure to change. https://gist.github.com/richhickey/1563cddea1002958f96e7ba95... That, at least, is clear, and it means that authors of significant third-party libraries that are clashing with the vision of the language (and its community) can make an informed decision to spend time elsewhere, avoiding frustration on all sides. https://twitter.com/cemerick/status/1067111260611850240
I don't think Guido is/was actually trying to do this, and I think it's unfair to say that was his goal. If it was, then he was deliberately tricking people by having a core team, BDFL-Delegates, a language summit, etc. If it was, then he was being rude by asking her to come to the language summit instead of saying "Amber, Twisted is very good but your vision for Python is not my vision for Python." Guido, as far as I can tell, built Python to be a widely-used language, not a language following any sort of vision he started with. Guido does want Twisted and other Twisted-scale projects around, and does want Python to be a useful language for them. That's why I say that if he's burned out, the right way to execute his vision (which is exactly what he's doing, in fact) is to step aside graciously.
Also, if you're going for max language usage through design, how do you do that? By trying to perfect your design, so it's kind of an irrelevant motivator. Rich got a ton of clojure users (relative to his resources), he did that by having strong (reasoned) beliefs, not by opening the floor to a vote. Rich did /not/ say that he was only building Clojure only for his company, that's a very misleading statement that you've made.
https://www.zdnet.com/article/linus-torvalds-takes-a-break-f...
[1]: https://mappingthejourney.com/single-post/2017/08/31/episode...
Once he chose to step away, he truly stepped away, which is a good model for any open source maintainer.
Any chance you could give a short little blurb for how the community is organized and works? I'm curious how different it is to Python. I also sometimes worry about the future of R where you have Python-Pandas coupled with Spyder IDE and Julia-DataFrame library with Atom-Juno as IDE as well as JuliaDB. It seems like a lot of communities are moving in on what R does best.
- A language that is simply less active or whose users feel less ability to have a say in the language's development is going to be both less social and less dramatic, but also carries a high chance that existing users are only there for legacy reasons and are always considering moving to another language better suited for their purposes.
- A language run by a corporation is less "social," and has the drama play out in business negotiations, closed-door committee meetings, and lawsuits instead of on blogs and public mailing lists and (in this case) closed-door meetings with an expectation of public reporting.
I'm not aware of any drama in the ASP community, but that probably also had something to do with many users having picked Python for their next project instead of ASP. I'm aware of some drama in the Clojure community, the conclusion of which seems to have been that Clojure is their corporate sponsor's language and if it doesn't work for you you should find something else.
(Though in nothing on the level of some in open source).
Sure you are going to need to redo code anyways from scratch eventually. But, programmers like I mentioned, which is surprisingly a huge chunk, make problems worse for themselves throughout the lifetime of the code by being short sighted and stamping their approval on code their too lazy to rewrite because their boss doesn't know any better.
Well, yeah, of course. It's expensive to reinvent the wheel.
Like I can see where I'd make that tradeoff, but it'd have to be a small function in a really niche use case, which isn't really that common. (I work on the JVM though and don't know how good/bad Python libraries would be in general.)
So yeah, I'd say the dependencies are pretty bad. It's a process outside of your control and you kind of just have to deal with whatever is under the hood. More importantly though, what's under the hood was likely meant to be general purpose and there is more than likely a better way to do it for your particular situation. And as she mentions a lot of the bugs are indefinitely there so you just have to have permanent workarounds, which is never good.
For speed, pypy is the way to go.
Recently started using Go a little, and it does feel better but it just takes longer for me to write (probably because it's newer to me too). At work, people want things quickly. I'd love to be able to spend my time writing things properly in Go, but it's just not as quick and easy as Python is.
On top of that, I believe that I am able to produce production ready code in other languages even faster than Python! Particularly when it comes to doing refactors for exploratory architecture in early POC/MVP; love the tooling assists so I'm not drowning in "is not a functions" and other dumb stuff while I'm working on hammering out interfaces, abstractions, and other structure.
Granted, I have almost certainly not worked with the best Python programmers, or the best programmers that happen to only use Python to put it another way. But looking at Python itself, the languishing ecosystem and stdlib, and every large OSS Python project, what does that even look like?!
So, I would say that yes it has a lot to do with familiarity and comfort... I'm very suspicious of people who insist on using Python and have little familiarity with anything else.
I’d rather have the first and risk the latter than the alternative.
I may have had to clean up in the aftermath a couple of times.
Me personally, for my company I would not hire any contractor making code for me in python. Not because python is inherently bad or unoptimized in all cases, but because I know the general tendency is to use lots of dependencies. Pythons motto over all is easy over hard that has really spread in the community in a not so good way. Like I said though, I use it, I like it, it is awesome for banging out prototypes, but I avoid it for production. There is just to many potholes to the point I would rather just code something from scratch closer to the hardware, which I will admit I do not enjoy xD, but if its what needs to be done, then so be it. There are just a lot of things that I work on where even 5% efficiency could save you thousands.
If an explanation boosts your ego, there is a decent chance it is flawed or incomplete.
https://groups.google.com/forum/?hl=en#!topic/nodejs/erDWyS4...
I think any discussion about NPM or JS packaging compared to other package managers needs to discuss those things as orthogonal, largely unrelated concepts. Otherwise everyone just picks a favorite punching bag (e.g. left-pad) and talks past each other.
NPM installs (and downloads?) a copy of each package for each project. This is the wrong thing to do.
I'm cautiously hopeful on the direction that .Net Core is taking, with the major pieces of the old framework modularized as nuget packages.
I think there is in general nothing wrong with a language ecosystem where key parts of the whole platform are in well-maintained third-party libraries rather than the standard library. Which is also something Amber Brown is saying here.
> cryptography/src/cryptography/hazmat/__init__.py
Hazardous Materials
This is a "Hazardous Materials" module. You should ONLY use it if you're
100% absolutely sure that you know what you're doing because this module
is full of land mines, dragons, and dinosaurs with laser guns.I would be surprised if either 'kentm or their security team wanted the rest of the Python standard library developers (or the Go ones, for that matter) to mess with cryptography modules when it's not their area of expertise.
I recently hacked together some Python to process MIDI files. Should I have used this:
https://pypi.org/project/MIDIUtil/
or this:
https://pypi.org/project/mxm.midifile/
or this:
https://mido.readthedocs.io/en/latest/
or this?
https://github.com/vishnubob/python-midi
A well-maintained and thoughtfully curated stdlib makes these choices for you - which is one less thing to worry about, and can be a significant time saver.
For some kinds of libraries, you can sacrifice a whole bunch of those constraints and still have it make sense to host it in the stdlib. I'm fine if the JSON library is slow or inflexible if it covers a bunch of use cases and doesn't impede people from writing better ones. You can see this to some extent with the state of "router" HTTP libraries in Go.
But for some things, most notably cryptography, it's worse to have a suboptimal version in the stdlib than to have none at all.
For all I know some Python devs are even more sophisticated about security than the C developers who are maintaining libraries for Linux and *BSD distros.