Python3 is removing crypt.crypt and not replacing it with anything | Hacker News Reader

Python3 is removing crypt.crypt and not replacing it with anything

eighty-twenty.org

41 pointstonyg76 comments

In this thread: People don't read the PEP and make knee-jerk assumptions.

In the PEP (https://peps.python.org/pep-0594/#crypt) it points out that this module didn't work on Windows at all and didn't provide any useful real-world functionality on Linux, BSD or macOS.

The use case mentioned in this blog (SHA512 password hashing) is considered a bad practice because unlike bcrypt, SHA512 is fast for an attacker to execute to check if a credential guess matches a hash.

> People don't read the PEP and make knee-jerk assumptions.

They just came over from the Java string interpolation thread, because after not reading the JEP they weren‘t sure how to spend the rest of the evening and thought „I know! Now I will not read the PEP.“ /s

The PEP is not accurate, probably outdated.

Linux distros are converging towards using libxcrypt with yescrypt, which is a modern password hashing function. If you have a crypt function that wires back to libxcrypt's crypt, you can use that.

It's a bit unfortunate that people seem to be unable to agree on a modern password hash (other places tend to use scrypt or argon2), but well, I think yescrypt is fine, and having a way to generate and verify crypt-compatible hashes is certainly useful functionality.

The rationale mentioned here https://peps.python.org/pep-0594/#crypt

Highlight:

> Only DES encryption is guaranteed to be available. DES has an extremely limited key space of 2*56.

It's a bad reaction to a real problem.

A good solution would:

* Puke warnings on the screen when using insecure algorithms

* Guarantee secure algorithms everywhere

These algorithms aren't hard to implement, and anything relying on DES will be faster in interpreted Python than in optimized code on the 80486 it was intended to run on.

I'd rather have this removed than broken, but it makes sense to just fix it.

As a footnote, a lot of code doesn't rely on this for anything high-stakes. If this is hashing passwords on my local machine, it's a backup to a backup for security. It should move to something modern, but it breaking in the meantime is a bigger failure than an obsolete crypto algorithm.

> These algorithms aren't hard to implement

As I understand it, they are difficult to implement securely. By that I mean avoid side channel (often timing) attacks.

PyPi has robust cryptography libraries from likes of Google, others. You do not need to implement them yourself. You can just install a well-known library used by the best in the software development industry.

Cryptography should fail to operate rather than operate insecurely.

There's also a lot of precedent for removing these security footguns from the stdlib that no one actually uses. From the top of my head, the `imgop` library and the `cgi.escape` function. (My personal bugbear is `csv.Sniffer` which, as far as I can see, has no correct use case, it's just broken.)

I'm somewhat out of touch with Python, but as far as I know, it's been best practice to use `cryptography`[1] for a long, long time.

ETA: Just to note - DES is super, super broken. It shouldn't even be an option. DES was published in 1975. 3DES was published in 1981 - DES has been considered too small a keyspace since 1981. That's older than Python. Exporting DES in your crypto API is a bug. (Apparently, according to Wikipedia, NIST has deprecated all use of DES [including legacy applications] as of 2023. But they only deprecated new applications in 2017, so perhaps I overstate.)

[1] https://cryptography.io/en/latest/

Better late than never

"Guaranteed to be available" isn't the same as "available in practice". Linux's current crypt function provides yescrypt, gost-yescrypt, scrypt, bcrypt, sha512crypt, sha256crypt, sha1crypt, sunmd5, md5crypt, bsdicrypt, bigcrypt, descrypt and NT (ordered by decreasing strength).

https://manpages.debian.org/unstable/libcrypt-dev/crypt.5.en...

So, if you're on an ancient unix box (or a Mac, apparently), then you're stuck with the obsolete standard library your OS vendor packaged, but that's true of other things.

Presumably, they'll remove SSL support soon too, since, when provided with a sufficiently old version of openssl support, it won't provide modern algorithms either. That means "only obsolete SSL modes are guaranteed to be available".

It's a remarkably gentle deprecation schedule. PEP 594 has been worked on for four and a half years. Python versions since 3.11 (released over a year ago) printed warnings. Python 3.12, with crypt, will continue to be supported for nearly five years.

This package says it's a drop-in replacement for the library. https://pypi.org/project/py-purecrypt/

In the Year of our Lord 2024 we still refer to Python3 as Python3. /meme

Because Python 2 only died a couple of years ago. There are people still using it.

Python 2 EOL'd over four years ago.

4 years is only a couple years ago.

Though in fairness it had been deprecated for 12 years at that point.

> We did not want to hurt the people using Python 2. So, in 2008, we announced that we would sunset Python 2 in 2015, and asked people to upgrade before then. Some did, but many did not. So, in 2014, we extended that sunset till 2020.

https://www.python.org/doc/sunset-python-2/

It blows my mind some people are still on Python 2, but I worked at a place that shifted an app to Python 3 in 2021, so I guess I'm one to talk. (Incidentally, I have written much more Python 3 than I ever wrote Python 2, but I still write print without parenthesis the first time, every time.)

A couple is 2, but yeah it took some time after EOL to properly die anyway.

"A couple" can mean "a few". Like "give me a couple minutes" doesn't mean exactly 2. Even "two seconds" doesn't actually mean 2 seconds.

A couple does mean 2, not "a few". It's just that people are often very inexact about times and numbers. So yeah it's silly to argue that "a couple of years ago" is wrong because it was actually 4 because the context clearly implied a degree of approximateness.

Inflammable means flammable. Couple means few. You can disagree with descriptivism, but you've lost the moment the issue was noticed if enough people keep doing it.

Yes, but the big thing which will make it truly EOL is when the long-term support Linux distributions drop it. Right now you can have a fully supported system from Red Hat, Amazon, Ubuntu, etc. with Python 2 on it. All of the slackers have been delaying upgrades because that extended support means they don’t immediately fail their compliance audits.

I'm not sure about the others, but Python 2 is not officially supported on Ubuntu. It was moved into Universe and is considered a community-supported package rather than supported by Canonical.

Ubuntu offers paid support. 14 is only falling out of that this coming April:

https://ubuntu.com/about/release-cycle

Yeah, but it's incredibly expensive. We got quoted a $$$$$$+ _monthly_ number several years ago, which helped us justify to the business to upgrade to Python 3.

Interesting, they must scale up with age. I had to deal with that recently and it was cheaper than trying to rush a late development project for a replacement.

But since it died, the only one Python getting any changes now is Python 3, so the "3" part may be viewed as redundant now.

It avoids the next renaming issue when Python 4 appears. It may be redundant, but it's not a pain to use the one extra character.

Unfortunately, Python minor version changes are neither forward nor backward compatible.

As part of the Internet archive, are there people who make VMs of older systems and libraries that can still open these old formats?

Thinking of Vernor Vinge's information archaeologists from A Fire Upon The Deep.

Debian is quite useful here. They keep prebuilt packages in the archive, one just needs to disable PGP key expiration checking to install them. But otherwise old Debians still work.

That's modern python. There's really no single stable python to target anymore. Only pythons. Every project requires you to set up a container in which you just create a custom python specifically for that project.

Breaking feature removal is more of the same spirit. These are the consequences of popularity and rapid progress. Perl would never be forced into a corner like this and chose to break things.

Even nodejs doesn’t do wild shit like this.

Sure it does. I'm still dealing with the fallout of old projects built with old webpack which uses md4 (as a file name cache buster only!) which is gone in recent nodejs+openssl binaries.

That sounds like a horrible design decision. I was using md5s in 2006 which was years before nodejs even existed.

You’re wildly overestimating the impact of this, but if you actually use Node you’ll learn two things: deprecations do happen (just look at the release notes!) but to the extent that you could say it doesn’t that would only be because Node doesn’t have a large stdlib and so the code you need to replace wasn’t deprecated by the core developers but rather the thousands of third-party modules’ maintainers. Either way you can’t ignore maintenance needs but I’ve found it to require more work in the latter model since the schedules aren’t synchronized.

What alternatives are there to Python for general, versatile backend scripting?

Any language that becomes a popular replacement will eventually suffer the same fate. It's the popularity and rapid "improvements" that cause this.

Bash and Perl?

Unironically yes - with a good shellcheck setup and a good local wiki of documented utility functions I’ve found bash to be an absolute powerhouse for writing portable code.

I’ve not used Perl as much in the last decade as I used to, but I’m always pleasantly surprised when I dig up some older stuff and it just works.

That's my main jam lately - shellcheck + shfmt + bash-lsp is a fantastic toolbox.

Ruby has managed to be incredibly self-compatible while adding new features for many many years now. Its main weakness compared to Python is its performance but that’s been steadily improving every year with each new (backwards compatible) Ruby release. And for 99% of apps it’s fast enough.

I always found Ruby super picky about specific versions, hence having shitloads of configs for rvm to use a specific version for each project.

Personal PowerShell

2.x era python also had rolling deprecations and removals, so I thought it was a bit similar. For example in 2.5:

> The old regex and regsub modules, which have been deprecated ever since Python 2.0, have finally been deleted. Other deleted modules: statcache, tzparse, whrandom.

https://docs.python.org/3/whatsnew/2.5.html#new-improved-and...

After over a decade of Python as my language of choice, I have stopped developing new projects with it, directly as a result of its instability through releases. Bit rot is a thing, and Python routinely causes more than seems reasonable.

What alternative(s) are you using instead?

I begrudgingly have moved to Go. Mostly because my current day job uses it a lot, so I have been paid to pick up more knowledge about its ecosystem.

Don’t get me wrong: there are plenty of things to dislike about it (or any language, once you start pulling that thread). However, they have made a much stronger commitment to avoid breaking backward compatibility.

Really? I even ported 2.7 code back when 3.6 got good, and it was all far simpler than I had dreaded.

I might be not as eager as some to learn new features though, but I am also not so conservative that I won't do things some new way, especially if the old way is being deprecated (but so far warnings have always given me plenty of heads up).

Quite the opposite, I like Python as much as I did when we first met. Nothing is without warts, but they are rarely really problematic for me in Python.

If your energy levels drop, or you lose interest, ports will become harder.

When you write code that has a short life span none of this matters. But when your software has a several year or decade life expectancy it will matter.

Also economics plays a role. If we have a real economic downturn in IT bit rot will become a huge problem for many small businesses.

If any effort is too much, you're basically stuck on a very select set of dead-ish languages or dialects thereof. But I'm not sure that's a more viable approach then using a language that is still pretty great about backwards compatibility.

I’m going to have to start specifically version pinning and vendoring pythons at some point.

What’s really funny is my older Python2 code doesn’t shit the bed half as much as the Python3 stuff due to it effectively being frozen in time.

IMHO, there should be a way to make de facto standard libraries included in python batteries. My (personal) short list: - requests - cryptography - numpy - pandas - tornado - plotly at least half of the top10 https://pypistats.org/top should be candidates for being included in python

That won’t be good. Their release cycles need to be different for patches and upgrades. Being included in Python stdlib means they’ll only get updated when Python does, which won’t be good for neither maintainers nor users.

That's hundreds of megabytes of stuff isn't it?

You could use a Python distribution that includes a lot of that stuff. I think Anaconda ships with a bunch of libraries by default.

They're removing a lot more than crypt. Also, telnet, smtp, nntp, various file format support, xdr, and so on.

Crypt is particularly painful, since removing it will effectively cryptoshred user password databases.

Anyway, I'm not particularly surprised. I don't think I've ever written or encountered a python script that didn't bit-rot after six months.

> Crypt is particularly painful, since removing it will effectively cryptoshred user password databases.

What do you mean by that? Anyone interacting with passwords on Linux, macOS, etc. has needed to use PAM anyway for decades. If you are building some kind of network app and just tossing passwords through crypt() you are going to have a much worse time on a security audit than you will installing a single python module.

The loss of telnet and crypt is going to mean I have to version pin a lot of older code.

I should look into which file formats are going away, I had some stuff that relied heavily on uuencode/decode (along with telnetlib) for bootstrapping file transfers into older systems…

Or just use an external module from pypi. Python stdlib doesn't have a monopoly on the bits you're using.

That adds extra steps, I prefer (where possible) sticking with just the standard library for portability when working with other people’s horrendously broken systems that may not have outbound internet access.

You can vendor it in your app yourself. That skips the internet requirements.

Between this and the telnetlib removal, a significant amount of code I have written is going to need to be pinned to a specific version, or vendor in those features.

A lot of that is code to interact with weird old shit, so I guess I can’t expect upstream to maintain old shit forever.

I do wish Python came with a solid cryptography library “out of the box” instead of having to use PyCrypto/Cryptodome/Cryptography/Whatever - and I also wish the way it’s ssl sockets module worked didn’t have weird unexpected things to do with how it handles file descriptors - you can’t just dup2 a ssl wrapped socket for example, unlike a normal socket

`cryptography` is the de facto standard crypto library for Python https://cryptography.io/en/latest/

But it does not have a function to use libc-style password hashes, so that doesn't help.

Out of curiosity, why are people storing SHA512 hashes in that format? Is there a protocol or tool that requires it? I'm going out on a limb and assuming they aren't using it to check passwords.

In my case (old style DES hashes) it’s to interact with some old crusty garbage.

Unfortunately a lot of that old crusty shit is still in production in many places, despite being EoL for years, with no plans to replace it this side of 2038.

Can you switch to ctypes?

  >>> from ctypes.util import find_library
  >>> find_library("c")
  '/usr/lib/libc.dylib'
  >>> from ctypes import cdll
  >>> libc = cdll.LoadLibrary(find_library("c"))
  >>> import ctypes
  >>> libc.crypt.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
  >>> libc.crypt.restype = ctypes.c_char_p
  >>> libc.crypt(b"toomanysecrets", b"az")
  b'azi4LBG1VJohQ'

Here is what Python's own _crypt does:

  >>> import _crypt
  >>> _crypt.crypt("toomanysecrets", "az")
  'azi4LBG1VJohQ'

Python core devs are succumbing to wikipedia editor disease it seems. Not the first bit of crypto functionality they've removed without a good alternative :(

Read the documentation, not the angry rant. There are good reasons for removing it and there’s a direct replacement:

https://docs.python.org/3/library/crypt.html

I still remember when Python 2.4 dropped the "rotor" module, which supported Enigma-like encryption and decryption. From https://web.archive.org/web/20030620052059/http://python.org... ;

"The original Enigma cipher was broken in 1944. The version implemented here is probably a good deal more difficult to crack (especially if you use many rotors), but it won't be impossible for a truly skillful and determined attacker to break the cipher. So if you want to keep the NSA out of your files, this rotor cipher may well be unsafe, but for discouraging casual snooping through your files, it will probably be just fine, and may be somewhat safer than using the Unix crypt command."

See https://docs.python.org/3/whatsnew/2.3.html for the deprecation notice: "The rotor module has been deprecated because the algorithm it uses for encryption is not believed to be secure. If you need encryption, use one of the several AES Python modules that are available separately."

Rage is all the rage these days, and following the PEP link he provided, the explanation is outrageous:

"The algorithms are mostly old, of poor quality and insecure. Users are discouraged from using them."

Why is this post being upvoted?

I think this is a good decision -- crypto is not a standard library thing and including it in the first place was a mistake. I assume someone will take the liberty of mirroring the crypt library onto pypi and everyone can just install that if they're worried about their code breaking. In my experience upgrading python versions always has some friction so I don't think that's a big ask.

1. It is LITERALLY in the libc. https://man7.org/linux/man-pages/man3/crypt.3.html

2. No encryption in general, means also no http module, which means no REST calls, which means people would just use another language.

> It is LITERALLY in the libc.

Technically yes, in the sense that libc also includes a libcrypt.so, but the one in glibc is hot garbage, so most distros disable the libcrypt part of glibc these days and ship libxcrypt instead.

LOL. Std lib removes crypto package and the answer is “copy paste this code from this blog”? How predictably awful.

Don’t do this. Instead, carefully analyse and evaluate well known and well tested packages to do crypto with.

*python3.13

Reply on news.ycombinator.com