Show HN: Python library to overload functions based on interpreter version | Hacker News Reader

Show HN: Python library to overload functions based on interpreter version

github.com

45 pointsttymck36 comments

I don’t want to discourage the author, but this seems like a misguided use of decorators. Maybe there’s some scenario where it’s practical to do version checks only at function boundaries, but why limit yourself that? It’s already easy (and clear, and non-magical) to check sys.version yourself in an if-statement. You can do that inside a def block to modify a function slightly, or outside a def block if you truly need the function definition to change by version. Given that, I don’t see how this library adds enough value to justify it as a dependency.

The project should also maybe note that it doesn’t help at all with Python 2 compatibility. A newer Python user might assume it enables freely mixing Python 2 and Python 3 in the same file, but even without reading the code I’m pretty sure the library doesn’t (and couldn’t) support this.

I wish Python had come with some equivalent to the "BEGIN {}" blocks that Perl and Awk have.

Lacking that, there's no way to write a (single file) script where a Python 2 interpreter would just get a "You need Python 3" error instead of a syntax error. Because testing for the interpreter version happens too late...after the syntax error already happened.

Perl's "BEGIN {}" block lets you easily test for versions or language features before the script itself is parsed.

You couldn't be more mistaken, most libraries supported both python 2 and 3 for years. Doing what you describe is extremely trivial. If you thought it was a good idea you could write a script that would exec trampoline itself into a different interpreter version entirely.

You can also catch a SyntaxError like any other exception.

Put an f-string into a script and get python2 not to choke on it, or try catching that SyntaxError as you describe...without using eval, futures, or similar. You can't catch something that already happened.

Yes, people are able to write backwards compatible scripts... by avoiding some features, or by using more than a single file.

What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script.

What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script.

  import sys
  if sys.version_info.major < 3:
    raise Exception("this script requires python version >= 3")

plus an extra line or two if you're worried about minor versions.

Now put an f-string (or other new feature that pops a syntax error) below that:

print(f"whee")

And run it with python2, or a python3 < 3.6.

ahh, I see now. I would add an edit to my original comment, but it's too late.

  > cat test.py 
  import sys
  if sys.version_info.major < 3:
      raise Exception("this script requires python version >= 3")
  
  print(f"test")
  
  > python2 test.py
    File "test.py", line 5
      print(f"test")
                  ^
  SyntaxError: invalid syntax

Doesn't seem that simple

It needs to also do:

   from __future__ import absolute_import, division, print_function, unicode_literals

This will give you syntax much closer to Python 3 in Python 2.7. The remaining differences can be papered over with function and class helpers. Well, except for metaclasses, but I doubt you'd have need for those in a single-file script.

This isn't pretty, but it's possible:

  import sys
  if sys.version_info[0] < 3:
      print("Needs Python 3+")
      sys.exit(1)
  else:
      prog = r'''
  
  def help():
      print("Help: ...")
  
  def do_spam(*args):
      print("spamming")
  
  def main(args):
      match args:
          case [subcommand, *args]:
              return globals()[f'do_{subcommand}'](*args)
          case _:
              return help()
  
  if __name__ == '__main__':
      try:
          status = main(sys.argv[1:])
          sys.exit(int(status if status is not None else 0))
      except Exception as e:
          import traceback
          traceback.print_exc(e)
          sys.exit(1)
  '''
      compile(prog, '', 'exec')
      exec(prog)

Sure, a variation on eval. I still think some sort of BEGIN equivalent would have been nice. But, the optimal time for that has passed, so I'm just complaining anyway :)

i meant exec as in a shell script to replace the current PID, not python exec()

It doesn’t seem common to write single file libraries in Python; and it doesn’t seem to be too unreasonable to add an entrypoint script for version checks.

Yes, I didn't mean for libraries, but rather for single file scripts.

Not ideal, but in my code, I intentionally provoke syntax error near the beginning of the file, with a comment explaining what's going on, e.g.:

    0_0  # Python >= 3.6 is required

I have a collection of such incantations for many Python versions here:

https://github.com/jwilk/python-syntax-errors

Ah, thanks. That's short and does almost what I want.

That's hilarious. Which PEP made

0_0

syntactically okay? I typed it into an AST visualizer (I'm on mobile) and if I'm not mistaken this is equivalent to `0`. But why?

Underscores in Numeric Literals. (PEP 515) https://www.python.org/dev/peps/pep-0515/

Your question about 0_0 is just 00 which is 0.

(This is really common in other modern languages. Readable example: 1_000_000 vs 1000000)

Sure to please ~17% of the people, PEP 638 -- Syntactic Macros => https://www.python.org/dev/peps/pep-0638/

(It's only Standards Track at the moment)

>Created: 24-Sep-2020

I imagine it would have been more popular than 17% if it were available when v2 -> v3 pain was at the high point.

Disagree. The breaking changes then were a tough pill. More would not have been better.

I wasn't proposing adding yet another change at the time. I'm saying if the capability were already in place, dealing with breaking changes would have been easier.

The whole reason this thought popped in my head is that I still have people asking today: "Tried to run your script, and I got a syntax error...is it broken?". When the issue is that the default python on their box is 2.x.

Excellent point.

My way to work around this is to make a zipapp that contains several files, hence the entry point can do the checking and import another file that contains the code. But since its a zipapp, it behaves like a single file script: https://docs.python.org/3/library/zipapp.html

Nowaday, I use shiv to produce the zipapp:

https://shiv.readthedocs.io/en/latest/

It's handy because you can also now embed 3rd party dependencies, even for small quick scripts if you wish.

Awk's BEGIN blocks are not parsed separately from the rest of the program. Awk's Yacc-based implementations (like the original One True Awk and GNU Awk) parse the entire input in a single call to yyparse() before executing anything.

Therefore, I suspect it is not possible to write a test in the BEGIN block which avoids a version-dependent syntax error elsewhere in favor of terminating with a graceful error.

Ah, good catch. They do work in Perl, perhaps in Ruby also.

There's an interesting extensibility point in Python in form of codecs and the encoding comment:

   # -*- coding: ... -*-

The trick is that encoding names come from an extensible registry of codecs (https://docs.python.org/3/library/codecs.html#codecs.registe...), so you can register your own. And when you look at codec API, you get the raw byte stream as input, and spit out strings - so it's basically as powerful as reader macros.

The problem is getting that registration code to run before your script starts. Easy if you have multiple scripts, but it requires .pth hacks if you go for single-file.

If Python had macros this library would have a very different implementation indeed.

I think 200LOC and a complex registration/namespace/cache system is a bit much to avoid an if statement around a def for a handful of functions. This is especially true since this just turns the if into a decorator, and imports will still need to be done the old way for compatibility. The cognitive overhead of this is not a good tradeoff.

You're absolutely right! I built this mainly to see if it could be done. I noticed uvicorn's main function uses an if-statement like you described, and I wondered what it would look like as a decorator.

My next goal is to see if I can make the resolution "static", i.e. Once all the functions are loaded, avoid the dictionary lookup at runtime, and just assign the appropriate function version as the bare module attribute.

It's fun to do things like this in python, simply because you can. But it's also important to point out that there's a lot of things you _shouldn't_ (this may be one of them!). I'm glad to see this sparked some discussion!

If Python simply parsed and evaluated top-level expressions one by one, you could test the value of some version variable and act accordingly. Like for this range of versions, load this file, otherwise load that one.

If I think of this problem in a Lisp frame, it does not scream "use macros" at me.

That looks great, maybe some libraries can leverage this to offer a wider range of compatibility -- even if "recent" syntax changes, such like the walrus operator or the match/case statements cannot be covered by this.

Btw, I've taken the liberty to post your project to my Python News aggregator here: https://news.python.sc/item/ff8f92de-666e-4913-8aa3-27af90df...

I'm not sure I get the point of this. If you already have to write the code using the equivalent old syntax, and keep maintaining it, why write the code a second time with the new syntax?

Old versions get deprecated in libraries, you don’t just cut over to a new way and break your callers on upgrade.

If you own all the callers of your api in the same repo, you probably don’t want this, you just replace all the old callers along with the implementation.

What I mean is (looking at the example in the README): when `asyncio.run()` was added, `asyncio.get_event_loop().run_until_complete()` didn't stop working.

What is the point of making two versions of the functions, when one of them already supports every Python you target?

The ecosystem is a bit broken, but the owners want to use the latest stuff. The outcome is obvious.

What is the obvious outcome?

How are you "using the latest stuff", which here is a syntax shortcut, if you have to write out and maintain the old syntax too?

Reply on news.ycombinator.com