The PEPs of Python 3.9
lwn.net
lwn.net
this article sums it up better than I ever could https://www.xanthir.com/b4wJ1
...then str.strip and variants could be cleanly and logically extended to allow this functionality, because passing a string and a sequence of strings would be distinguishable.
Alas, clean and logical function design can be hard to do late in a languages life.
PEP 593 and PEP 585 are clean and logical... glad to see that :)
[0]: https://mail.python.org/archives/list/typing-sig@python.org/...
if isinstance(msg, str)
I agree that iterating on string is not proper design however. It's not very useful in practice, and the O(1) access has other performance consequences for more important things.
Swift did it right IMO, but it's a much younger language.
I also wish we stole the file api concepts from swift, and that open() would return a file like object that always gives you bytes. No "b" mode. If you want text, you to open().as_text(), and get a decoding wrapper.
The idea that there are text files and binary files has been toxic for a whole generation of coders.
It really is nonsense isn't it? Its like asking a low level api for opening files as a .doc, or as a pdf. Why would that be part of the file io layer?
Now I’m more familiar with it I’m careful to be explicit with the decoding when using text to make it super obvious what’s going on.
It just never appeared to me that the default mode of writing the file would _not_ write the array I was passing it.
It’s much more important for beginners to be able to learn clear recipes rather than having double standards with a bunch of edge cases.
EDIT thanks, I knew that didn't look quite right. "Mojibaked" - such a great term.
Even languages like Java now recognise the need to provide convenient access to text files as part of the standard API, with Files.readAllLines() in 7, Files.lines() in 8, and Files.readString() in 11.
Besides, Python has pathlib now, which allows you to do Path("foo").read_text() for quick and dirty text handling already.
if isinstance(msg, str)
As a second point, do you have more on your last sentence? ("The idea that there are text files and binary files has been toxic for a whole generation of coders."). I have been thoroughly confused about text vs. bytes when learning Python/programming.
The two types are treated as siblings, when text files are really a child of binary files. Binary files are simply regular files, and sit as the single parent, without parents itself, in the tree. Text files are just one of the many children, that happen to yield text when their byte patterns happen to be interpreted using the correct encoding (or, in the spirit of Python, decoding when going from bytes to text), like UTF8. This is just like, say, audio files yielding audio when interpreted with the correct encoding (say MP3).
Is this a valid way of seeing it? I have to ask very carefully because I have never seen it explained this way, so that is just what I put together as a mental model over time. In opposition to that model, resources like books always treat binary and text files as polar opposites/siblings.
This leads me to the initial question of whether you know of resources that would support the above model (assuming it is correct)?
For me the toxic idea about text files is that they're a thing at all. They're just binary files containing encoded text, without any encoding marker making them an ideal trap. Is a utf16 file a text file? Is a shift-jis file a text file? Have fun guessing edge cases. We've already accepted with unicode that the "text" or letters are something separate from the encoding.
If you don't know the Encoding? Just write a quick detect_bom function (should be part of the standard library, no idea why it isn't) and then open it with that encoding. I.E.:
encoding = detect_bom(fn)
with open (fn, 'r', encoding = encoding) as f:
...
Ultimately, putting the responsibility for determining the CODEC on the user at least makes it clear to them what they are doing -opening a binary file and decoding it. That mental model prepares them for the first time they run into, say, a cp587 file.
I understand why Python doesn't do this - it adds a bit of complexity - though you could have an "auto-detect" encoding scheme that tried to determine the encoding schemes, and defaults to UTF-8 - not perfect, as you can't absolutely determine the CODEC of a file by reading it - but better than what we have today - where your code crashes when you have a BOM that upsets UTF-8 decoder.
I finally wrote a library function to guess codecs and read text files, inspired by https://stackoverflow.com/a/24370596/1637450 and haven't been tripped up since.
But Python does not make it easy to open "text" files - and I know data engineers who've been doing this for years who are still tripped up.
This is, of course, utterly BS.
All files are binary files.
Some contains sound data, some image data, some zip data, some pdf data, and some raw encoded text data.
But we don't have a "jpg" mode for open(). We do have higher API we pass file objects to in order to decode their content as jpg, which is what we should be doing to text. Text is not an exceptional case.
VSCode does a lot of work to turn those bytes into pretty words, just like VLC into videos. They are not like that in the file. It's all a representation for human consumption.
The reasoning for this confusing API is that reading text from a file is a common use, which is true. Espacially on Unix, from which C is from. But using a "mode" is the wrong abstraction to offer it.
If fact, Python 3 does it partially right. It has a io.FileIO object that just take care of opening the stuff, and a io.BufferedReader that wraps FileIO to offer practical methods to access its content.
This what what open(mode="b") returns.
If you do open(mode="t"), which is the default, it wraps the BufferedReader into a TextStream that does the decoding part transparently for you, and returns that.
There is an great explanation of this by the always excellent David Beazley: http://www.dabeaz.com/python3io_2010/MasteringIO.pdf
What it should do is offering something this:
with open('text.txt').as_text():
This completly separates I/O from decoding, removing confusion in the mind of all those coders that would otherwise live by the illusionary binary/text model. It also makes the API much less error prone: you can easily see where to the file related arguments go (in open()) and where to text related arguments go (in as_text()).
You can keep the mode, but only for "read", "write" and "append", removing the weird mix with "text" and "bytes" which are really related to a different set of operations.
with open('text.txt', 'w').as_text(): with open('text.txt','w').as_text() as f:
f.write("text")the next thing will be a bunch of "open" functions:
with open_binary("filename") as f:
...
with open_text("filename") as f:
...
with open_text("filename").writeable() as f:
...
Python uses text mode by default to avoid surprising beginners on Windows. If you only use Unix-like OSs you will never have this problem.
Opening a file should return an object that gives you bytes, and that's it.
This "mode" thing is idiotic, and leak a low level API that makes no sense in a high level language with a strong abstraction for text like Python.
Text should decoded from a wrapping object. See my ohter comments.
However I think strings being iterable is one of the core ergonomics in the language and basic types of Python that make it so nice for many applications. Scripting, scraping, data cleanup, data science, even basic web development, all benefit hugely from little features like this. Without this sort of thing Python would be a different language with different uses.
While I normally like safety and types, I’m personally happy with things like this because it fits with Pythons strengths.
for char in my_str.chars(): foo()
That wouldn’t sacrifice any ergonomics, being consistent with how we already iterate over dictionary contents with d.items(), and it’d address all the concerns in the parent comment link
> And an obvious "fix" for this is worse than the original problem: Common Lisp says that strings are composed of characters, a totally different type, which doesn't implement the same methods and has to be handled specially. It's really annoying.
It seems to me this contradicts most of what the article says. Sure, strings are rarely collections, so they should not be iterable by default. But the final solution offered admits that sometimes they are, and then you want to be able to iterate over something. For most instances of something, It does not make sense for the individual "elements" to be strings. Bytes are clearly not strings, code points are clearly not strings, grapheme clusters are clearly not strings. Each of those will provide very different methods, because they are very different things. Only after that point (words, sentences, etc.) does the idea of the element being the same type start making sense again.
Clearly the concept of a "character" is too ambiguous, and there is no clear "default" for what it should mean, but the idea of a string consisting of some kind of element that is not string appears obviously correct.
The basic idea is to have sum(string.foobars()) == string. Bytes and characters ('grapheme clusters') are then a specific subset of strings, that can therefore support additional operations like byte.ord(), the same way eg positive numbers support num.sqrt().
In Haskell, String is an alias for [Char].
Maybe it's not changing String rather, encouraging other types which don't have this difficulty.
assert(not isinstance(x, str))> > I think name choice is easier if you write the documentation first:
> > cutprefix - Removes the specified prefix.
> > trimprefix - Removes the specified prefix.
> > stripprefix - Removes the specified prefix.
> > removeprefix - Removes the specified prefix. Duh. :)
I actually don't agree that it's so obvious, since it returns the prefix-removed string rather than modifying in-place. I think Fahlgren's argument would work better for `withoutprefix`.
So there is no possible confusion.
For those situations, bytes() + memoryview() or bytearray() can be used, but then you are on your own.
And honestly, I would be rich if I got a dollar every time a student does this:
msg.upper()
msg = msg.upper()
if title == user_input.upper():
With compiler, you mean mypy or linters?
>>> def up(v):
... v.upper()
...
>>> dis.dis(up)
2 0 LOAD_FAST 0 (v)
2 LOAD_METHOD 0 (upper)
4 CALL_METHOD 0
6 POP_TOP
8 LOAD_CONST 0 (None)
10 RETURN_VALUE
>>> def up(v):
... if v.upper() == "HelloWorld":
... return True
...
>>> dis.dis(up)
2 0 LOAD_FAST 0 (v)
2 LOAD_METHOD 0 (upper)
4 CALL_METHOD 0
6 LOAD_CONST 1 ('HelloWorld')
8 COMPARE_OP 2 (==)
10 POP_JUMP_IF_FALSE 16
3 12 LOAD_CONST 2 (True)
14 RETURN_VALUE
>> 16 LOAD_CONST 0 (None)
18 RETURN_VALUE
msg.upper()
In fact, any program can replace anything on the fly, and swap your string for something similar but mutable.
It's the trade off you make when choosing it.
You can definitely catch most cases at runtime. I’ve done something like this, in an library, to catch a case where people were treating the copy of data as a mutable view.
interface[address][slice] = new_values # fancy noop
byte_values = interface[address]
byte_values[slice] = new_values
interface[address] = byte_values
x if y else z
y and x or z
The existence of mypy and other static type checkers for Python disproves that; given their existence, warning of an expression producing a type other than “any” or strictly “None” was used in a position where it would neither be passed to another function or assigned to a variable that is used later should be possible. Heck, you could be stricter and only allow strictly “None” in that position.
> msg.upper()
> Instead of:
> msg = msg.upper()
> And then call me to say it doesn't work.
On this, isn't the student's reasoning sensible? E.g. "If msg is a String object that represents my string, then calling .upper() on it will change (mutate) the value, because I'm calling it on itself"?
If the syntax was upper(msg) or to a lesser extent String.upper(msg) then the new-to-programming me would have understood more clearly that msg was not going to change. Have you any insights into what your students are thinking?
It's just something to be told. A design decision, like there are thousands to learn in IT, that you just can't guess.
That was the original syntax [0], before the string functions became methods. I agree that a method more strongly implies mutation than a function does.
Also, for consistency with list methods like `reverse` (which acts in place) and `reversed` (which makes a copy), shouldn’t the method be called `uppered`?!
[0] https://docs.python.org/2/library/string.html#deprecated-str...
Also, it looks like that’s the name that Swift uses.
The details for mutation of data in scheme go beyond that, though. Sometimes procedures are "allowed but not required to mutate their argument". Most (all?) implementations do mutate, but it is still considered bad for to do something like:
(define a (list 1 2 3))
(append! a (list 4))
(display a)
(define append! append)
It always creates a new object but it doesn't necessarily copy the contents (even shallowly).
For instance slicing a `memoryview` creates a subview which shares storage with its parent.
>>> a = [254, 255, 256, 257, 258]
>>> b = [254, 255, 256, 257, 258]
>>> for i in range(5): print(a[i] is b[i])
...
True
True
True
False
False
>>> x = 257
>>> y = 257
>>> x is y
False
>>> def f():
... x = 257
... y = 257
... return x is y
...
>>> f()
True
OTOH it’s recommended to use `is` rather than `==` when comparing to “singletons like `None`”.
https://www.python.org/dev/peps/pep-0008/#programming-recomm...
class C:
def __eq__(self, other):
return True
print(C() == None) # True
print(C() is None) # FalseSince you said "always", I'll be "that guy"...
For the builtin types, yeah. But not in general; user code can do whatever it wants:
class MyType:
def __getitem__(self, *args):
return self
my_obj = MyType()
my_other_obj = my_obj[3:7]
assert my_other_obj is my_objYes. And doing otherwise is pretty risky as the Java folks discovered, ultimately deciding to revert the optimisation of substring sharing storage rather than copying its data.
The issue is that while data-sharing substringing is essentially free, it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour.
Languages which perform this sharing explicitly — and especially statically (e.g. Rust) — don't have this issue, but it's a risky move when you only have one string type.
Incidentally, Python provides for opting into that behaviour for bytes using memory views.
You can get around that with a smarter garbage collector, though. On every mark-sweep pass (which you need for cycle detection even if you use refcounts for primary cleanup), add up the number of bytes of distinct string objects using the buffer. If it's less than the size of the buffer, you can save memory by transparently mutating each substring to use it's own buffer. If it's not, then you actually are saving memory by sharing storage, so you should probably keep doing that.
That complexifies the GC (already a complex beast) as it requires strong type-specific specialisation, more bookkeeping, and adds self-inflicted edge cases. Even the JVM folks didn't bother (though mayhaps they did not because they have more than one garbage collector).
I think this will go a long way toward making type annotations feel less like a tacked-on feature.
Do you mean you'd been using that in comments?
Just to be clear, this isn't about ad-hoc syntaxes for use in comments, this is about syntax that parses when used in python code, and which can be used by type-checkers.
[int]
{str: int}
{'x': float, 'x': float}
def f(xs: [int]) -> {str: int}:
...
def g(x: 1+2+3, y: what('ever')) -> foo / bar:
...
i'm pretty sure you could even write a mypy plugin that'd interpret `[int]` into `list[int]` etc
In practice though we should probably all write annotations that do work with an existing type checker. False negatives are bad enough in mypy without people writing annotations for non-existent type checkers! (IMO --check-untyped-defs should always be used; mypy is misleading without it.)
oh, sure! "freeform annotations" that break mypy in a published library should be a punishable offense ;)
Given the recent success of anonymous functions and it's widespread use and impact in other languages, like Javascript, C#, Java, C++11, maybe it's time to re-evaluate that opinion. I mean, what would javascript be today without promises and multi-line anonymous functions?
But, it’s still unlikely that we’ll see full anonymous functions in Python. They just don’t work with its significant-indentation syntax.
Guido laid out the challenges of multiline lambdas on a mailing list [1] and then followed up with a blog post [2] [3]. His chain of thought is worth reading in full, but the crux is lack of “Pythonicity” and his gut feel that named functions avoid the complexity and possible ambiguity of multiline lambdas:
def callback(x, y):
print x
print y
a = foo(callback)
[2]: https://www.artima.com/weblogs/viewpost.jsp?thread=147358 "Language Design Is Not Just Solving Puzzles"
[3]: https://news.ycombinator.com/item?id=20672739 "Language Design Is Not Just Solving Puzzles (2006) | Hacker News"
You may already be aware, but not everyone is: they capture variables from outer scopes in exactly the same way that lambdas do.
Lots of people want them (lots of people don't, too), but no one has come up with a great syntax that plays nice with the rest of Python and saves you much over named functions.
x('.b', onclick=func ev:
print("clicked")
end)
of course it matters, there are design decisions in languages. a certain pattern or syntax may feel just right in one language but very much wrong in another.
Throwing in an `end)` like that feels wrong in python
Once you have the scaffolding of the syntax you can turn it over to the bikeshedders on the mailing list to make it pretty.
def begin(*args):
return args[-1]
begin(
func := lambda x, y: begin(
z := int(input()),
x + y + z
),
func(1, 2)
)I'm full of hopes for this feature, but it's going to be a slow hard work, and we'll only rip the benefit on the long run. So no use to rush it.
I feel like we rushed asyncio and type hints, and it took years to make them usable after they were introduced.
{str: [int]}
The ipaddress library also has no support for calculating subnets. It is quite hard to go from 2a00:aaaa:bbbb::/48 to 2a00:aaaa:bbbb:cccc::/64. It would be less weird if the essence of the documentation didn’t make it sound like the library was otherwise very thorough in the coverage of its implementation.
Can anyone write a PEP? Maybe I should get off my behind and actually submit a patch for proper IP calculations? Or maybe I missed it in the documentation (which, aside, I wish wasn’t written with such GNU-info style formality.)
original_net_48 = ip_network("2a00:aaaa:bbbb::/48")
desired_subnet = ip_network('2a00:aaaa:bbbb:cccc::/64')
subnets_64 = original_net_48.subnets(16)
print(f"{desired_subnet} is one of the computed subnets: {desired_subnet in subnets_64}")
#=> 2a00:aaaa:bbbb:cccc::/64 is one of the computed subnets: True site = ip_network(‘2a00:aaaa:bbbb::/48’)
subnet = f(site, 64, 0xcccc)
Edit: yes! But it’s a bit slow...
But on the other hand, you can make it WAY faster by doing it with bit manipulation instead like you're writing C:
Such a feature requires support from the parser.
It's also just a fun video on language parsers in general.
However, I also found that parsing is rarely a performance bottleneck so that wasn't a big plus.
PEGs are however are easier to reason about, no ambiguities, so that was a good enough reason.
Yes and no.
PEG parsers are definitely easier to implement than any of the LR(k)-and-ilk parsers. So if you're writing or debugging a PEG parser, that will be easier.
However, while shift-reduce conflicts are confusing, they are there to give the strong guarantee that the grammar is unambiguous. And the parser generator will tell you this as soon as you've defined the grammar, before you've even used it. PEG grammars instead remove the guarantee, and let you deal with any confusion that arises much later.
Here are some methods of reasoning that standard parsers will give you that PEG parers will not:
1. A ::= B | C is exactly the same as A ::= C | B.
2. If A ::= B | C and you have a program containing a fragment that parses as an "A" because it matched "B", then you can replace that fragment with something that matches "C" and it the program will still parse.
Neither of these rules hold in PEGs.
Here's a practical concern that (2) helps with. Say you have a grammar for html, and a grammar for js. And you want to be able to parse html with embedded JS. So you stick the js grammar into the html grammar at the right places. If you're using a standard (e.g. LR(k)) parser, and you don't get any shift-reduce (or other) conflicts, then the combined grammar works. In contrast, if you're using a PEG grammar, it's possible that you've ordered things wrong and there are valid JS programs that will never parse because they're clobbered by html parsing rules outside of them. Or vice-versa.
Also, realistically if you're using a PEG parser you'll want one that handles left recursion, because working without left recursion turns your grammar into a mess. And left recursion in PEGs can have some weird behavior.
I frequently hear this mentioned ("PEGs don't have ambiguity"). It is literally true, but I don't think it's true in the sense that actually matters.
I've blogged about this in the past (https://blog.reverberate.org/2013/09/ll-and-lr-in-context-wh...), but I'm not the only person saying this:
> PEG is not unambiguous in any helpful sense of that word. BNF allows you to specify ambiguous grammars, and that feature is tied to its power and flexibility and often useful in itself. PEG will only deliver one of those parses. But without an easy way of knowing which parse, the underlying ambiguity is not addressed -- it is just ignored.
https://jeffreykegler.github.io/Ocean-of-Awareness-blog/indi...
Python broke compat once in 25 years and gave 13 years to migrate.
It's a very conservative language.
It might seem like a quibble, but it seems better to describe Python 3 as a different language versus Python 2. Newbies seem to get that.
(Or, alternatively, "How many Python 2 scripts will run on a Python 3 interpreter?" Answer: "None of them.")
That is completely false. So many libraries have support both for 2 and 3. That's code that run just as well under both interpreter.
Googling quickly, I find this, which does a bit better than 2to3. I suppose one could write to a somewhat constrained intersection of Python2 and Python3, if one is willing to make at least some boilerplate changes to the original Python2 code.
https://python-future.org/overview.html
That said, if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.
Yes you often need some precautions like "from __future__ import" statements and sometimes libraries like `six`, but it's been perfectly normal practice for most of the last decade.
In fact, the vast majority of popular libs had a 2/3 compatible code base for a few years.
The hard part was not the syntax in fact. It's pretty trivial: the language are not that different.
The hard part is the I/O stack, because the stdlib is very different, espacially for this part.
Sure, as long as it doesn't contain any syntax or spellings which are incompatible between the two. That's a fairly large subset of the language.
> if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.
But this will work:
from __future__ import print_function
print(x)
Also, as I said above, there is a pretty large subset of the Python language that has the same syntax and spellings in both Python 2 and Python 3, and any script or module or package that only uses that subset will run just fine under both interpreters. You are drastically underestimating both the size and the usage of this subset of the language.
Nearly every py3 feature was backported to 2 you just need to write it in a compatible way. I'm seeing some drop py2 support now though. Which I'm fine with, I haven't written python2 code in maybe 6 or 7 years now.
That's obviously untrue. For example, consider the following Python 2 script:
with open('a.txt', 'r') as a, open('b.txt', 'w') as b:
for line in a:
b.write(line)
My guess is that if you sweep GitHub for Python2 code and push it into Python3, that proportion would be under one percent.
For some projects I did bigger refactors for 2.6/7 (exceptions) and 3.6 (fstrings).
I did two migrations of >500k loc projects in an afternoon each, and admittedly some days of testing to gain confidence since there where few unit test. But i found it to be very smooth sailing.
I was very familiar with both projects, so that helped a lot.
EDIT: I also want to add that i did this using python3.5, when to ecosystem seemed to be at a sweet spot of dependencies supporting both 2 and 3 mostly. I guess if one has been waiting until now, the divide between library versions will be a lot bigger.
int main(int c,char** v)
{
printf("Hello, World!\n");
return 0;
}
That is not true though, a lot of 2 scripts will run no problem, especially if import future is used
And there are really few that are even near Python popularity, or used with such diversity as Python.
I mean, you can get away with keeping AWK the way it was 2 decades ago, nobody is going to use it for for machine learning or to teach computing in all universities in the world on 3 operating systems, utilizing C extensions, or processing Web API.
Among the few that would even compare, there are the ones that have accumulated so much cruft that they became unusable from today's standard (E.G: bash). Then you have those who have done like Python (E.G: perl 6). The ones that just tried and failed (PHP 6). The ones that broke compat and told everybody move or die (Ruby in a point release, gave basically 2 years). And the ones that created a huge pile of horror they called full stack to keep going (E.G: JS). Also those that got hijacked by vendors and just exploded in miriads of proprietary syntaxes (E.G: SQL) or completely new concepts (E.G: lisp).
At least, in Python you CAN write Python 2/3 compatible code, and you have a LOT of tooling to help you with that, or migrating.
So, yes, the Python 2 -> 3 transition could have been better. Insight is 20/20.
But I'm struggling to think of any other language in a similar situation that has done better.
If Python 3 had been introduced as a separate language, I'm pretty sure everyone would have said "Why isn't this just called Python 3? It's 99.9% the same as Python and it's by the same people and they're deprecating Python in favor of it."
Perl 6 (now Raku), of course, but they at least had the decency to admit it.
I argued on the list that these should be kept for major releases for planning reasons but they appear to be convinced it is too hard.
I know some Python Libraries break backwards compatibility (Pandas being a big one) - but, for the most part, hasn't the language been backwards compatible since at least Python 3.4? (And possibly further back, for all I know).
https://docs.python.org/3/whatsnew/index.html
Keep in mind they have deferred a number of them because of the impending EOL of Python 2.7. There are fewer breaking changes during the latter 3.X series, which should resume in 3.9 or 3.10 now that Python2 has passed on.
Here's a commonly mentioned one:
Changes in Python Behavior: async and await names are now reserved keywords. Code using these names as identifiers will now raise a SyntaxError. (Contributed by Jelle Zijlstra in bpo-30406.)
Note: I think this is a bad idea, I'd rather all these small breaking changes and parser be deferred to 4.X. But they need to be small breaking changes, of course, not a new language.
Great naming...missed their chance to make the functionality of strip/lstrip/rstrip more clear by name the new methods stripword/lstripword/rstripword. Which would also had the benefit of consistence.