Hacker News Reader

Comment by strenholme | Hacker News Reader

I have seen multiple people post to Ycombinator claiming that porting to Python3 is always trivial and easily done. And, yeah, sure, for a 380-line script written after Python3 came out, porting the code so it runs the same both in Python2 and Python3 is trivial.

However, for an over 6000-line script written in the mid-2000s, before Python3 even existed, porting to Python3 is non trivial. Trust me, I tried.

First, the syntax sugar stuff was easy enough to fix: Do a simple search and replace so that all cases of stuff like

    print "foo"

Become

    print("foo")

I had to do something similar with the way this script calls raise(). Not to mention adding a hack so Python2 uses “xrange” while Python3 uses “range”. After doing all of this, the code still does not run in Python3. As far as I can tell, the issue is that Python3 does implicit float-to-int conversions differently than Python2; the code will not raise an error in Python3, but it will silently run differently, until the state is different enough that the Python3 version raises an error while the Python2 version works as expected.

At this point, I just threw my hands in the air and gave up.

For reference, the offending code is here:

https://github.com/samboy/misc-civ4-mapscripts/blob/master/T...

View on HN

No one has claimed porting to python 3 is easy in this thread. That said, if you start by "search and replace", then you are probably doing it wrong [0].

[0]: https://docs.python.org/2/library/2to3.html

There are lots and lots of people who have claimed that porting to python 3 is easy. Stop twisting the truth.

Hacker News gets some of the smartest people willing to jump into a comment section, but its also full of uninformed idiots talking out of their ass and trying to sound authoritative. I won’t defend them, but complaining about something someone else posted in another thread isn’t gonna get you too much sympathy either.

>Python3 does implicit float-to-int conversions differently

I don't think so? Might be floating point division.

> In Python 3.0, 5 / 2 will return 2.5 and 5 // 2 will return 2. The former is floating point division, and the latter is floor division, sometimes also called integer division.

https://stackoverflow.com/questions/183853/what-is-the-diffe...

I think an automated tool like `2to3` is probably a better bet for doing that kind of conversion. It should be a mostly automatic process for most scripts.

2to3 does NOT handle the change in division behavior. There is no automated way to do so because python2 only does floor division sometimes and it's decided at runtime based on the arguments being ints or not.

I believe you can get python3 behavior on a file by file basis with

    from __future__ import division

But it’s been a while since I’ve written anything serious in python.

Yes you can, but this person needs the python2 behavior in python3 which pretty much requires selectively replacing / with //.

Yep, you can use // to get python2 behavior unless the object redefines __truediv__ but not __floordiv__

That works for writing new code in Python 2.7. It would also have worked for going the recommended route of gradually migrating toward Python 3 compatibility while remaining on Python 2, _provided_ you're working with an organized project that's amenable to converting on a file-by-file basis.

I can see where it falls flat on its face when you're trying to migrate a monolithic 6000-line script. But, IMO, that example falls flat on its face when used as a rhetorical device. If you've got a monolithic 6000-line script with (presumably) no test coverage, you've made your own bed, and I don't think that the inherent difficulties of maintaining something like that should be blamed exclusively on the 2->3 transition.

In my experience in the business world, it is far too common to find a company’s critical infrastructure running on stuff more messy than this particular 6000-line monstrosity (which, for the record, I am not the original author of, and, yes, I have added a few test cases for code I have added to it like my own RNG).

I once was getting paid double my previous salary to maintain Visual Basic code which makes this 6000-line script look like well structured fully tested object oriented code. When my boss came in to the room and said the project was cancelled, I let out a sigh of relief that I would never have to look at that code again.

In terms of Python2, it will be maintained until at least 2024 because it’s part of CentOS/RHEL 7. I think it will be maintained until 2029 because I saw that package in CentOS 8, but I do not recall if it’s a core package with a guaranteed 2029 EOL.

Python 2 is not a core package in rhel8. You can get it via an extra collection that will be supported for the life of rhel7.

So using rhel8 won't be a solution. Rhel developers are probably eager to drop Python 2 support as soon as possible. Which is when rhel7 goes EOL. Of course, it wouldn't particularly surprise me if customer pressure makes them support it for longer.

I believe 2to3 hasn't been the entirely recommended way for a while - modernize (dual compatibility via the six module) and futurize (more magic to make as many python3 idioms work in 2 as possible) let you move towards python 3 without breaking compatibility from python 2, and give you ways of working in both versions at once - 2to3 isn't even mentioned on https://docs.python.org/3/howto/pyporting.html.

That said, syntax conversion is usually only a small part of the problem - unicode handling can be more problematic, especially if you've never spent much time thinking about or working with encoding/decoding.

It's definitely _much_ easier if you have a good grasp of both versions of the language and the changes that were made, and it isn't helped by the pervasive myth that "It's just brackets on print statements".

We recently converted a ~900kloc open source codebase to dual python 2/3 with many scattered interested parties and approached it in roughly a phased way - Using CI to prevent backsliding of syntax converted portions of the codebase and getting the tests running (and failing) in python3 early were crucial. (To add to this - a very _early_ step was getting the future imports in place - every file had future absolute_import, division imported at a very early stage. For division, you can also start python2 with -Qnew and it'll force all division to "modern" style).

Six is just trading dependency on py2 for dependency on a library that wouldn't otherwise be needed if you just made a code base forward compatible. It is more sensible to gradually work on getting 2to3 to do a clean conversion and then doing a final conversion when the py2 code is ready to be EOLed.

You are arguing for a hard cutover, which is how we got into this mess in the first place. By using a forward or backward shim, one can run on both. 2to3 is a CASE tool, it requires manual fixup, so it cannot be used to maintain a dual version codebase.

I read their argument as, getting the codebase to the point at which running 2to3 completes the transformation. I personally remain unconvinced that that code wouldn't require the equivalent of a compatibility library anyway.

And, do hard cutovers ever work well?

It's the same work as porting to a compatibility library but you don't have to carry any legacy long into the future once the port is finished.

This is a hard cut over from Python2 to 3. This presupposes that both upstream and downstream deps are on Python3. On which magical day does everyone decide "to have their port finished" ?

If it is the same work to to support both 2 and 3 as jettison support for 2 in favor of three. Then supporting both so the system can actually upgrade is far preferable. Applications and libraries not used in isolation. To jettison support while the rest of the dependency graph is flux is analogous to gimbal lock, you can't just get here from there. Without `six` the whole Python ecosystem would have bifurcated and Python2 would still be the dominant Python.

I'm reading "getting 2to3 to do a clean conversion" as having a python2 codebase but then running 2to3 to build a "python3" version before deployment. If I recall, that was touted as the approach in the early days of python3 and conversions. I haven't seen it used or talked about much, recently, although I'm aware that later tools like pasteurize attempt to do the reverse (but haven't tried them).

I worry that doing this would basically cause much of the pain of maintaining separate codebases - inevitable bugs that affect one or the other, with the problem of not being able to directly write in one of them, and getting to where you have to check both for new code. Plus, it doesn't cover everything (or can't - e.g. ambiguous cases were removed), and has a few places where it makes mistakes (have seen but cannot recall) - so you end up having to write cross-compatible code for some things yourself anyway.

I feel that extra dependencies, especially commonly known, understood and reasonably well documented ones, is an extremely small price to pay.

We decided early on for our large project that developing separately and having a version shear where we "switched" to python3 would probably be chaos. The slow and gentle way, you can make changes and adjust the other developers at a more reasonable pace, the point at which you drop python2 is merely a decision with an empty changelog (at which point you can start stripping stuff out).

Devil's advocate:

Many people aren't even bothered to try. The company I work for still writes new stuff in Python _2.7.3_ - not even a recent release of 2.7. Some of their stuff will be difficult to convert, but a lot of it really is trivial. They just don't care to put any effort into it until things start to break down.

I can understand why people get frustrated by situations like this and are tempted to call out programs still running Python 2 and those who advocate for keeping it.

I'm a huge fan of some of the features introduced by Python 3, so I switched and am not interested in going back. However, I think there is room for other interpreters to step in and continue supporting Python 2 even if the PSF wont.

Google?

If I were in that situation, I would make a reasonable effort to keep scripts as polyglots: Make sure they both run in Python 2 and in Python 3. In my experience, doing so is pretty easy to do when writing new code from scratch with Python 3 compatibility in mind.

The “no one is maintaining Python 2” issue is an example of open source economics in action: Everyone wants Python2 to be maintained, but no one is stepping up to plate to pay for it to be maintained, so it’s not getting maintained.

> Python _2.7.3_

Familiar number.

Maybe they tried, and stuff broke in weird and unexpected ways, so they gave up. For example, the first time we upgraded from that to (I think) .8, lots of strange errors started coming in from our webapp. Apparently kombu, one of the dependencies of celery, was importing an underscore-function from the standard library, which no longer existed in the new version.

This is definitely the case for a few apps. It's why I raise hell anytime I see someone use a private attribute or function from a builtin module.

However, I'm painfully aware that it does not apply to most of our apps and it does not justify using an outdated interpreter for writing new apps.

There are occasional cases where, with some Python libraries, I need to use a “hidden” _function to do my job. The one time I had to do that, I made sure to contact the authors to let them know what I did and why:

https://github.com/cobrateam/splinter/issues/591

Nobody is saying otherwise, but maintaining EOL software is also hard. The Debian project does have any able developers volunteering to do it for Python 2 in Debian's next release. That's all that there is to it.

    def a91a15d7(x):

Jesus. I didn't read much farther than this, but it does not surprise me that this codebase is difficult to work on.

It's so vitally important to write tests. Years ago I started doing full tests suites, even for my open source, work, once it got mature enough I felt comfortable with the patterns I established. Then I'd add tests before developing new features.

Unit tests help make upgrading so much easier. I was once on a huge Scala codebase that needed to go up from Spray.io to Akka-HTTP. Tests help save me on tons of potential bugs in migrating over. The few things that got through, I could write more tests for.

On another open source project I maintain, my functional tests helped me migrate from Scala 2.11 to 2.12 along with all my dependencies with relatively low friction.

If you're porting your Python2 to Python3 stuff now, it's never too late to start writing some tests. Write them over the weird division stuff you've found. Make sure they run in Py2 and fail in Py3 so you can isolate what it is. Even a couple of small tests in a large code base that only cover stuff you're finding now is better than no tests at all.

Have you tried https://docs.python.org/3.9/library/2to3.html? It should automate some of the syntax conversation.

You can also write custom transforms with 2to3 but I've found https://pybowler.io is a decent wrapper. I haven't used it for 2to3 conversion though.

There is no sure-er mark of a Python novice than the "2 to 3 conversion is trivial" myth. Python has a long history, there are lots of versions of 2, there are complicated data processing changes from all versions of 2 to all versions of 3. Not every conversion is hard, but they can be an enormous amount of work.

Python doesn't implicitly conver floats to in ints. As others have commented, there have been changes to how division is handled in that int / int may result in a float. But, to the beat of my knowledge, there's no implicit float -> int conversion. Maybe. Some libs convert floats to into, but its not the norm.

> the way this script calls raise()

That actually isn't a function call, despite the appearance. Python 3 raise is still a statement, not a function (unlike print). So this

    raise(ValueError, "some text")

does not do what you think; the proper way in Python 3 to spell what you probably expect is

    raise ValueError("some text")

Here is a transcript of a shell session showing the difference:

    Python 3.5.2 (default, Oct  8 2019, 13:06:37)
    [GCC 5.4.0 20160609] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> raise ValueError("some text")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: some text
    >>> raise(ValueError, "some text")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: exceptions must derive from BaseException

Note that the exception you actually get the second time isn't even ValueError, it's TypeError; it's being raised because "some text" is a string, not an instance of an exception class derived from BaseException.

In other words, you should not be treating raise in Python 3 the way print is treated. The 2to3 tool treats it properly.

Well, yes, porting large projects to Py3 is not trivial.

That's why they had, what, around 5 years to do it since the 2.7 EOL was extended to 2020?

I think most of the problems you encountered would be solved by the six library, btw.

But the most annoying bugs were those were some Python 2 quirk was "just working" (like comparing strings to integers in Python 2) or the conversion of ints and floats and then it correctly broke on Python 3.

Using the six library is one way to make a large project being written from the ground up run in both Python2 and Python3.

For this particular project, I would had needed to get the code to run in both Python3 and an ancient version of Python2 made back when boost.python was still a thing, so six would be out of the question.

I ported a website written in 2009 from Python 2 to Python 3, which was around 20,000 lines of code. It wasn't "trivial", but it took me (a single developer) a little less than a week. Most of the difficulty wasn't code changes, but upgrading dependencies.

Recently I had to port some code from the legacy version of an internal DSL to our newer version. I just wrote a primitive parser that would understand 98% percent of the code (in a safe way) and lowered that into the new DSL. Worked like a charm, and I didn't have to rewrite ~2000000 lines (a colleague tried pattern matching, but that was unusable for complex expressions).

I wonder if/why there is no such ~~thing~~ [edit] effort for python2/3?

If you're only accepting programs that P2 accepts, you can even get away with being lazy and "understanding" more than P2.

This isn't from experience (I have yet to do a conversion myself), but one thing I've heard is that the str/unicode change revealed that a lot of codebases didn't understand the difference in py2, were using the wrong one at API boundaries, and required a lot of manual work to figure out what it should be for each one so py3 could handle it.

Ouch. GP mentioned int/float conversion differences, and coders using the wrong [string/unicode] types doesn't help.

I wonder if these are symptoms of duck typing?

Edit: 2to3 is what I'm talking about https://docs.python.org/2/library/2to3.html

Maybe it's not trivial, but if you'd migrated two lines of code a day, you'd be done by now. Code requires some amount of maintainance.

> After doing all of this

You hopefully mean after spending 1 minute running py2to3 right? I mean I get that some people feel that it’s tedious or difficult to convert if they spend ages manually changing thousands of lines of code in a completely automatable fashion. Which is why it’s nice that you don’t have to actually do that. You just run py2to3 and the focus on differences that matter.

The primary choices for distribution are to either demand maintainers that they updates their scripts, or to take up maintaining the python 2.7 code base themselves, a code base which is written in C and mostly before 2000.

Is your code in Debian right now, or was your comment a general question in regard to the status of python 2.7?

Out of curiosity, how is this issue handled for languages like Algol or Fortran? The gfortran codebase was also written significantly in C and mostly before 2000.

I've read elsewhere in this thread that those implementations are still being maintained, so they are not in the same situation ad Python 2.

The Python 2 -> 3 transition has been unusually bad.

In most programming languages, a change to the language is practically always backwards-compatible. The update may add new capabilities, but the old capabilities and syntax remain in place. If the language changes semantics, it's usually in bizarre edge cases of the language. If a basic data type was a mistake, for example, a new data type is added & people are encouraged to use it instead... but the old one keeps working. If an existing data type or method should be renamed, the new name is provided & the old name keeps working for a long time so that people can incrementally update.

In short, the normal expectation in programming language updates is that existing code will keep working, and that updates can be done slowly and incrementally. That's historically been true for Python as well; updates typically didn't break much. In the few cases where there were significant backwards-incompatible changes, languages typically give 10+ years of warning.

The Python 2 -> 3 transition has instead been an example in how to do things badly. Pretty much no Python 2-only code will work without change; you have to change print to print(), division's semantics have changed, string semantics have changed, common names have changed (xrange to range), and so on. The original automated transition tool originally envisioned could never work reliably, because Python's dynamic nature ensures that there's not enough information to work without help. So the demands on people to make changes were unusually large for a language update.

A related problem was that there was never an implementation that supported Python2 and Python3 at the same time. That meant that if you depended on 100 Python2 libraries, you had to wait for all of them to transition before you could transition. If 99.99% of the libraries you depend on are available in Python3, you must use Python2. You can easily create C or Fortran programs that call on code written long ago unchanged; that is not true for Python3.

So in the Python3 transition all code had to be modified simultaneously across the ecosystem, and tools couldn't do it well.

Eventually the Python3 folks made enough changes to Python3 so that it started becoming practical to write code that worked on both Python2 and Python3. That should have happened from the beginning.

The only language I can think of that handled a transition worse is Visual Basic. When "Visual Basic for .NET" came out, it was so grossly incompatible with the previous Visual Basic that it was called "Visual Fred". In numbers I saw long ago, about 1/3 stayed with the old unsupported Visual Basic, 1/3 changed to a completely different language (that was less likely to abuse them), and only 1/3 of the previous Visual Basic users transitioned to Visual Basic for .NET.

I think that Python3, all by itself, is a fine language. The problem in this case is that the developers of Python made it unnecessarily hard to transition to Python3. That was both unfortunate and unnecessary.

Perl 6 botched the transition so badly they're not even going to call themselves Perl anymore, and Perl has basically fallen off the map for new projects. The Python 2->3 move was rough but the language seems to be thriving despite that.

Perl 6 is really a different language from Perl 5, so I think the new name is what should have happened from the beginning.

But I basically agree with you that using the name "Perl 6" for Raku led to most people abandoning Perl 5. Hopefully people will think more carefully about transition in the future. But too many technologists ignore history, so I don't have high hopes.

It is often easy, but not always.

Write some tests around the problematic functionality before attempting the final port, it should help in this situation. Should be broken into multiple files as well.

The number of atrocities committed in that code are mind numbing -- this is what you get for writing such unmaintainable code.

Could have been said more kindly, but the point is accurate. This reads like C code that happens to be written in Python. There is a lot of fiddly bit-twiddling operations.

The main issue — besides not using 2to3 — is that / in Python3 is interpreted as floating point division, whereas it used to be integer division if operands were integers. Replacing / by // in the relevant code blocks might help? Or getting another implementation of the RNG?

It really isn't that bad. I regularly see much, much worse, and much more unidiomatic python, from people who have gone on to work at FAANG.

The code base in question is not one I wrote; it is one I decided to maintain years after the original author lost interest in it. [1]

This is not too different from what I have seen in multiple jobs, where the new hire doesn’t get to write all the code they use from scratch, but instead has to maintain someone else’s mess. Often times, management and market conditions demand more from programmers than they have time to properly implement with test cases, well structured code, good comments, etc.

Technical debt is a big problem in code used by businesses, and the technical debt this particular 6000-line script has is a lot less than some stuff I have seen and have been paid to maintain.

The fact of the matter is that most open source code out there is quite a bit cleaner and maintainable than the code which is behind firewalls, running important businesses.

[1] Edit: To the original author’s credit, this particular code base is an incredibly popular and successful map generator for Civilization 4, with some 40 thousand downloads; it got messy because the author was more interested in adding new features requested by users than in keeping the code base clean. It is a single script because that made it a lot easier for users to install. And, newer versions of the same generator are written in Lua and not Python because that is what newer versions of Civilization use for scripting.

No, this code "got messy" because the person writing it has no clue what they're doing in Python. This code was never clean.

Please keep https://news.ycombinator.com/newsguidelines.html in mind when commenting.

A) Not your job to police this site, B) just because your feelings were somehow hurt about code you didn't even write, doesn't actually mean I violated commenting guidelines, and C) maybe you ought to read the guidelines yourself, considering one of them specifically says not to comment when you think a post is inappropriate.

You want a "good critical comment"? Here:

* Try breaking your code up into more than one file. There are other/better ways of releasing Python besides "one file".

* Lay off the code commenting, it's vastly overdone here.

* Use version control tools to version your code, not comments.

* Python convention is to use underscore_case, not camelCase.

* Variable/function/class names should be descriptive, not random letters/numbers/incomprehensible abbreviations.

* Delete commented out code -- it's stored in version control, you're not going to lose it.

* The "initialize" func is a terrible concept and represents a fundamental misunderstanding of how Python classes work. This whole misadventure is a cautionary tale in, "Why a light architecture pass matters even in 'throwaway' scripts."

* Add tests.

* The massive if/else chains can be replaced in a number of different, more readable and therefore superior, ways.

That's some of what I saw in 2 minutes of review. This code is in very poor condition, and if you think the state of this code is an indictment on how hard it is to convert python from 2 to 3, you're more than just a little mistaken. You would need to pay me substantial amounts of money to spend more of my time going deeper into what's wrong with this code, as there is a lot and it is not generally enjoyable to work with code so thoroughly broken as this.

The script (somehow) works. That matters. However, this is not a good representation of why migrating from Python2 to Python3 is hard; this code is in no shape to be used to represent anything other than how not to write Python.

How do you know that?

Reply on news.ycombinator.com