Understanding Python through its builtins
sadh.life
sadh.life
One subtle point that the post gets wrong:
> So where does that come from? The answer is that Python stores everything inside dictionaries associated with each local scope. Which means that every piece of code has its own defined “local scope” which is accessed using locals() inside that code, that contains the values corresponding to each variable name.
The dictionary returned by `locals()` is not literally a function's local namespace, it's a copy of that namespace. The actual local namespace is an array that is part of the frame object; in this way, references to local variables may happen much more quickly than would be the case if it had to look each variable up in a dictionary every time.
One consequence of this is that you can't mutate the dict returned by `locals()` in order to change the value of a function-local variable.
Another, less-subtle error in the post is this:
> int is another widely-used, fundamental primitive data type. It’s also the lowest common denominator of 2 other data types: , float and complex. complex is a supertype of float, which, in turn, is a supertype of int.
> What this means is that all ints are valid as a float as well as a complex, but not the other way around. Similarly, all floats are also valid as a complex.
Oh, no no no. Python integers are arbitrary-precision integers. Floats are IEEE 754 double-precision binary floating-point values, and as such only support full integer precision up to 2^53. The int type can represent values beyond that range which the float type cannot.
And while it is true that the complex type is just two floats stuck together, I would very much not call it a supertype. It performs distinct operations.
> Accessing an attribute with obj.x calls the __getattr__ method underneath. Similarly setting a new attribute and deleting an attribute calls __setattr__ and __detattr__ respectively.
Attribute lookup in Python is way more complex than this. It's an enormous tar pit, too much so to detail in this comment, but __getattr__ is most often not involved, and the `object` type doesn't even have a __getattr__ method.
And I have no objections to the article's description of the bool type.
Spot on. Python is widely described as a simple language, but the complexity of attribute lookup is one thing that shows that's not true at all.
Many things in Python are easy, such as adding `@property` above a method definition to turn it into a getter. But `@property` is far from simple - the way it actually works is very complex (for example, properties have to be data descriptors, because non-data descriptors cannot override object attributes of the same name).
The really cool thing about this, how descriptors have their __get__ called, is that methods are implemented this way. So when you access instance.method(), it’s a normal lookup for the attribute named “method”, which is (normally) itself a descriptor, so the __get__ magic is called and this binds the method to the instance at the moment it’s needed! Then you can just call it like a normal function. It’s incredibly elegant but extremely obscure. And vital to understand if you want to dive into monkey patching, which is an incredible skill to have!
* Special-method lookup (e.g. of `__add__` when you do `a + b`) works differently because it doesn't look at the instance `__dict__`, only the class hierarchy.
* Lookup on a class works differently because as well as looking at `__dict__`, it has to consider superclasses.
Much of the complexity relates to the different ways of handling the instance `__dict__`. By contrast, Ruby is able to have much simpler lookup rules because it never considers the instance, only ever the class hierarchy.
“But!” I can hear people say, “I heard you could attach methods to instances in Ruby! How is this possible if Ruby never considers the instance in resolving a method call?”
Well, that’s tricky. The thing is, Ruby instances have no public members, instance variables are private (for direct access, but not in any strict way, because there are public methods called instance_variable_get and instance_variable_get on Object which…do exactly what the names say.)
But every Ruby instance conceptually has (though it doesn’t concretely have one unless you add something to it) a unique class for the instance that is the first thing in its class heirarchy. And you can add methods to that class (the instance “metaclass”, which is different than a Python metaclass) for the effect of attaching them uniquely to the instance itself.
> Python is widely described as a simple language, but the complexity
> of attribute lookup is one thing that shows that's not true at all.
The complexity is all exposed to Python programmers, which makes the language hard to master and, unlike truly simple languages, too large for anyone to understand completely.
Yeah, calling float and complex "supertypes" probably wasn't the best idea, but I couldn't think of a better explanation that wouldn't take too long to explain. I'll ponder about that one.
the getattr thing seems like a huge rabbit hole, I'm totally going to look into this. Thank you :)
I once tried to catalogue all the stdlib operations which release the GIL, meaning if you use only those (well, only those "heavy" bits, you can still use other small blocking glue bits), you can do "real" multithreading.
It was a fun exercise!
Me too. When I used to write Ruby I read a lot of CRuby source code. I achieved a much deeper understanding of the language that way. Even answered some really fun stackoverflow questions.
Now the first thing I do when I see a new language is read its source code.
Edit: Found it! You're in for about 9 hours of quality watching. https://youtube.com/playlist?list=PLwyG5wA5gIzgTFj5KgJJ15lxq...
We still have -X dev and sys.flags but it's runtime only.
Would be interesting to scan pypi and check if this is still a thing.
Maybe create a bot warning lib authors.
While today's use case for assert is unit testing, the actual killer feature of the keyword is that it's removed only from prod.
The idea is that you can write things like function contracts, that are expensive, but only exist in dev.
Now, if one of you dependencies use assert for something they expect to still be in prod, which is the problem we are talking about in the first place, "-o" or "--remove-assertions" will strip their assert too, breaking their code, and hence, yours since it depends on it.
Another solution would be to accept the current role of assert as quick error checking and add in debug_assert to indicate a conditional error check. The biggest issue with that approach is that a majority will suddenly ask "Wait, python has a debug mode?"
The original feature is perfect. But there is 5 good years of educating users so it can be used.
So the problem is some vendors ship libs that have failing asserts so cannot run in debug mode?
I’ve been guilty of that in my own code when I just want to tell myself about an error.
get_president_inside_bunker()
assert president_is_inside_bunker()
close_bunker_doors()
assert bunker_doors_are_closed()
send_nukes()
Things like `list.append` modifying in-place might feel like a flaw to some, but I think Python is really consistent when it comes to its behaviour. If you ask a person who comes from an object-oriented world, they'll say it only makes sense for a method on an object to modify that object's data directly.
There's always ways to do things the other way, for example you can use x = [*x, item] to append and create a new copy, while being quite a bit more explicit that a new list is being created.
When you have immutability guarantees (like in many functional programming languages like ML or Haskell) you can avoid making copies by sharing the parts of the data structure that don't change.
If this kind of thing interests you, you should check out Chris Okasaki's book "Purely Functional Data Structures".
But immutability sure is nice when you can have it.
In my opinion, you should use whichever method makes your code easy to read and understand for your usecase.
first = [0,1,2]
second = [*a,3] # first is unchanged, second = [0,1,2,3]
Though, to me, it's asking for trouble later.
True, though you end up with things like:
' '.join(thelist)
thelist.join(' ')
JS:
Array.prototype.join.call(["one", "two", "three"], "|")
str.join("|", ["one", "two", "three"])Ruby does it by having a mixin (Enumerable) that anything meeting a basic contract (roughly equivalent to the Python iterable protocol) can include to get an enormous block of functionality; Python doesn’t have (or at least idiomatically use as freely; ISTR that there is a way to do it) mixins like Ruby does.
For example, scala has Iterable#mkstring (https://docs.scala-lang.org/overviews/collections-2.13/trait...)
str.join() and bytes.join() can support all iterable type arguments.
Better than trying to implement a join (or two) on all iterables.
Javascript, on the other hand, kinda does it worst, at least of the languages I regularly use... .join() is a instance method on Arrays and TypesArrays. But they forgot to add any kind of join for Sets, for example.
(["a", "b", "c"]).join("")
"abc" # alright
(new Set(["a", "b", "c"])).join("")
Uncaught TypeError: (intermediate value).join is not a function
([...new Set(["a", "b", "c"])]).join("")
"abc" # grmpf, have to materialize it into an array first.
python isn't alone either, by the way. C# has the static string.Join(...) that accepts "enumerables" (IEnumerable<T>), but no array.Join() or list.Join() or dictionary.Join(). Combined with Linq, especially .Select, that becomes quite handy. It has been plenty of times I did print-debugging by adding a one liner along the lines of
Console.WriteLine(string.Join("\n", dictionary.Select((key, value) => $"{key} = {value.SomeProperty}")));
`str.join(sep, ...)` works in Python as well, because `a.f(...)` and `type(a).f(a, ...)` are (almost) equivalent.
In pretty much all other languages that have them, the expected behavior of A+=B is exactly the same as A=A+B, except that A is only evaluated once. Now lets look at lists in Python:
xs = [1, 2]
ys = xs
ys = ys + [3]
print(xs, ys)
xs = [1, 2]
ys = xs
ys += [3]
print(xs, ys)
(Note that this is not the same as C++, because in the latter, the variables store values directly, while in Python, all variables are references to values.)
The worst part of it is that Python isn't even self-consistent here. If you only define __add__ in your custom class, you can use both + and += with its instances, with the latter behaving normally. But if you define __iadd__, as list does, then you can do whatever you want - and the idiomatic behavior is to modify the instance!
For comparison, C# lets you overload + but not +=, and automatically synthesizes the latter from the former to enforce the correct behavior.
After 'a = 300; b = 50000',
a += b
a = a + b
xs = [1, 2]
def foo():
xs += [3]
foo()
This would be horrendously inefficient without immutable data structures like Clojure's. Very few languages have that, so it's a strange assumption to make, especially for a language as old as Python.
Although it is a very nice feature of Clojure.
Rich implement his own brand of persistent data structures which makes Clojure's immutability a lot more efficient.
“persistent vectors” are certainly an interesting data structure that strike a compromise between fast indexing and being able to relatively quickly create a copy where only one element changes, but it's a compromise and indexing is made slower to allow for the latter. — They also take up more memory on their own but are allowed to share memory with their copies.
I will say that my ideal language contains them in the standard library alongside standard vectors that index in constant time.
Further, it should be noted that much of the performance talk is on the assumption that accessing from memory is truly random access; — with the existence of c.p.u. caches that assumption is not entirely accurate and accessing from contiguous rather than scattered memory in practice is considerably cheaper so one also pays the price for their being scattered more in memory.
But when you're indexing into the vector sequentially, the memory layout plays rather well with memory caching behavior, and most lookups are going to be in L1 cache, just like they would be in a conventional array.
So lookups are a bit more expensive, but not as much more expensive as one might imagine.
The actual data of a Pvector is not in contiguous memory but scattered however the JVM wills it, and on top of that in order to find which address to retrieve it, an algorithm that runs in logarithmic time with respect to the length of the vector must be used opposed to a constant time one.
How can most lookups end up in L1 cache if an element that is 32 indices removed is statistically likely to be arbitrarily far removed in memory?
Of course, all of that is not that material to begin with given that most elements will be pointers to begin with so the actual objects wither they point will already be arbitrarily scattered and it simply adds one more pointer indirection, but for unboxed types such as integers it does play a factor.
(loop [i 0]
(println (get my-vector i))
(recur (inc i)))
No doubt this is quite a bit worse than having everything nicely laid out sequentially in memory, but it's not as bad as you're describing.
Of course, all of that is not that material to begin with given that most elements will be pointers to begin with
I guess this is sort of true. If you're doing random lookups, then using a persistent vector instead of an array list mean 17 trips to main memory instead of just 1, so it's not totally inconsequential.
But I think (hope?) that modern JVMs can optimize collections of small immutable objects so that they're not represented as pointers to the heap. Surely ArrayList<Integer> x gets represented as int x[], and not int *x[], at least with the most optimizing JIT level.
The obvious question is why it can't return a reference to the list instead of returning None. I feel like if I've been using the language on an almost daily basis for ten years now and I still get burned by that all the time, then it's just a poorly designed feature.
Coming from a language with baked-in immutability, Python’s behavior in this regard was very difficult to get used to.
* `sort(arr)` returns a sorted copy of the input * `sort!(arr)` returns the sorted original
methods that return booleans end in `?`, like `arr.sorted?`
It's just a convention, but it's a nice way to let the writer know what will happen.
For example if I have a map function that applies a function f to a sequence, should I call it map! because I might pass in a function f that mutates the input? If so then it seems like any function that takes a function as input, or any function that might call a method on an object, should get marked with ! just in case. But if I don't mark it that way then the ! marking is not as informative: I might end up with a line consisting only of non-! functions which still mutates the input.
A very common case of this is mutating versions of non-mutating methods, but (1) mutating methods (in stdlib or other idiomatic code bases) that have no non-mutating equivalent are not named with a “!”, and (2) methods are sometimes named with “!” because they do dangerous things compared to a base method that are not mutating the receiver.
> `sort!(arr)` returns the sorted original
Python has a naming convention as well: `sorted(arr)` returns a sorted copy and `arr.sort()` works in-place (and returns None). However, I've always thought it's a bit odd that one is a function and the other is a method.
array = random.shuffle(array)
It would also enable chaining operations:
array = array.append(A).append(B).sort()
Perhaps recognizing this is really the job of an external linter. Sometimes I wonder if the future of enforcing canonical formatting on save like "gofmt" or "black" will extend to auto-correcting certain goofy errors on each save.
mypy would yell at you about this, but afaik type-checked python still isn't the norm.
xs.sort() # in-place
ys = sorted(xs) # copy
with array do
begin
append(A);
append(B);
sort;
end;
array append(A); append(B); sort.
I think it's pretty off-base to call this a "flaw". Immutable structures have their place and can be very helpful where appropriate, but making its core primitives work this way is far outside the scope or the philosophy of Python. If you want otherwise, you're really wanting an entirely different language. And there's nothing wrong with that! But I think it would be a "flaw" for Python to make these operations immutable, even though I love immutability personally.
https://github.com/django/django/blob/01bf679e59850bb7b3e639...
print([
f"{(not x % 3) * 'Fizz'}{(not x % 5) * 'Buzz'}" or x for x in range(1, 20)
]) >>> l = ['a','b']
>>> l[False]
'a'
>>> l[True]
'b'
>>> d = {0: 'a', 1: 'b'}
>>> d[False]
'a'
>>> d[True]
'b'
>>> # TIL!More pythonic maybe, but you can't have more than a single expression in a list comprehension without it becoming completely unintelligible. I also often miss other standard list features. Reduce, flatmap, indexed versions, utils like first of predicate, split, filternonnull etc
Python's creator, Guido van Rossum, doesn't like functional/functional-ish programming a lot. That's well-known.
Guido: "I value readability and usefulness for real code. There are some places where map() and filter() make sense, and for other places Python has list comprehensions. I ended up hating reduce() because it was almost exclusively used (a) to implement sum(), or (b) to write unreadable code. So we added built-in sum() at the same time we demoted reduce() from a built-in to something in functools (which is a dumping ground for stuff I don't really care about :-)."
Guido van Rossum doesn't appreciate FP and he really genuinely doesn't understand it. That's not to say he's dumb or not a nice guy or anything. It's just not his area. And this is reflected in the language.
His attitude of you have to be "really smart" to understand FP is a mistake.
https://developers.slashdot.org/story/13/08/25/2115204/inter...
https://blog.finxter.com/about-guidos-fate-of-reduce-in-pyth...
The problem with filter chaining is the final assignment and the actual value are separated by the rest of the chain.
i.e. x = strList.filter(parseInt).sum() doesn't tell me what type x is till I read the whole line.
Whereas:
x = sum( parseInt(v) for v in strList ) tells me as quickly as possible that I'm dealing with a single value output that will be a sum.
arrow(my_list [map args] [filter args] [filter args] [map args] [sum])
arrow(my_list [map map_func arrow.list_here])
Yeah, one thing I like about Ruby over Python is the fluent code th former allows of that style.
People talk about Guido not liking functional style, but that explains comprehensions over map/filter, but not function-sum() over method .sum().
Amen.
There are plenty of languages where your code ends up looking like an entry in an obfuscation competition without even trying. If you're using Python, and working for me, I expect the code to be readable by anyone.
And, no, I don't give a toss whether the code is three times the length it might have been if it was dangerously, and expensively, obscure.
Someone coming from, say ruby, or javascript would find list comprehension jarring. You can't compose them and you basically have to rewrite them when its time to extend them.
readable by anyone is pretty subjective.
Like, which of these make more sense?
strList.filter(isNumeric).map(parseInt).filter(x => x != 0)
[ x for x in [ parseInt(s) for s in strList if isNumeric(s) ] if x != 0]
filter(map(filter(strList, isNumeric), parseInt), lambda x: x != 0)
And it's not like Python doesn't have the language features to implement the first pattern. Map,reduce,filter,etc. could simply be added to the iterable base class and be automatically usable for all lists, generators and more.
Surprisingly, Python doesn't have an iterable base class! `list.__bases__` is just `object`.
[for x in [for s in strList: if isNumeric(s): parseInt(s)]: if x != 0: x]
[for s in strList: if isNumeric(s): if (x := parseInt(s)) != 0: x] >>> max(['aaa', 'bb', 'c'], key=lambda item: len(item))
'aaa' l = []
for a in range(10):
for b in range(10):
for c in range (10):
l.append(a + b + c)
l = [
a + b + c
for a in range(10)
for b in range(10)
for c in range(10)
]
class list:
def __eq__(self, other):
return all(x == y for x, y in zip(self, other))
# Can also be written as:
return all(self[i] == other[i] for i in range(len(self)))
It's probably simplest to compare `len(self) == len(other)` before.
Similarly, the set comparison will also be true if the first is a subset of the other.
I still wonder what was the reasoning for allowing creation of local objects with the same name as builtins.
Okay it can be nice to redefine pprint as print I suppose.
Still how many sum, list, min, max, dict(!) have been erroneously redefined in beginner tutorials and beginner code.
From my experience sum and list suffer the most.
Sure there are linters that will warn you but there should be a setting for the interpreter (as in -Werror in GCC) to disallow this silliness.
For students for example, I'll have to agree. Maybe having a flag or environment variable that teachers can set up for it would be a nice idea. You should start a thread on the python-ideas mailing list about this, and it might get somewhere :)
A good linter will catch these, so in production environments you usually don’t run into issues. I agree that it can be a beginner trap though!
This is better for novices, because otherwise you create a whole bunch of land mines for people who are desperately trying to get something done. If they aren't aware of the built-in then they aren't trying to use it. Insisting that they become aware of something they don't want right then will be frustrating.
It's also better for experts, in that they're generally aware they're overriding a built-in and are doing it on purpose, and if not they'll have an IDE or linter reminding them.
To me, I see tooling as a spectrum from supportive to controlling. Python is very much on the supportive end. It feels controlling when I get interrupted because some programmer who has never met me programmed a tool to insist I do things their way. That would very much include insisting I respect a bunch of names they decided long ago to put in the global namespace.
False = None = True
$ python2 -c 'False = None = True'
File "<string>", line 1
SyntaxError: cannot assign to None
$ python3 -c 'False = None = True'
File "<string>", line 1
SyntaxError: cannot assign to FalseI don't know any books for projects per-se, maybe HN will know!
That said, I think I had good luck with Writing Idiomatic Python.
When I learned Kotlin, I just read through the docs, and then knew of basically all the different concepts in the language.
For Python, the docs were comparably very bad. For instance, Decorators aren't mentioned even once in the "The Python Tutorial". In "The Python Language Reference" (if one even bother to read such a dry document) it's barely mentioned in passing. How should a new user know it's a concept and how to apply it? And the language reference links only to a glossary item, and none of them specify how parameters in a decorator is supposed to work.
Pretty frustrating experience, put me a bit off the language from the get-go.
@decorator
def f(...): ...
def f(...): ...
f = decorator(f)
Compare their ThreadPoolExecutors (Python's being inspired by Java's)
https://docs.python.org/3/library/concurrent.futures.html#co...
https://docs.oracle.com/javase/7/docs/api/java/util/concurre...
Long story short : I think both have a lot of value (but beware I'm quite biased on Robust Python)
The solutions presented typically include both a "basic" approach, a "as pythonic as possible" approach, and a brief discussion of the trade-offs between elegance and readability, etc.
https://github.com/satwikkansal/wtfpython
HN thread: https://news.ycombinator.com/item?id=26097732 (163 comments)
PS: found with a site I'm building: https://discussions.xojoc.pw/?q=Understanding+Python+through...
def outer_function():
x = 11
def inner_function():
nonlocal x
x = 22
print('Inner x:', x)
inner_funcion()
print('Outer x:', x)
And that's also usually what you want because otherwise a function would start altering variables in the enclosing scope if they happen to exist!
E.g.
foo = 42
def myfunc(bar):
foo = bar + 1
print(foo)
myfunc(6)
print(foo) # would print "7" if the "foo =" above took the nonlocal foo automatically!
I can appreciate this must have taken considerable effort, It reads really well. Thank you!
What is your definition of a primitive data types? All of these have object as a superclass, so I wouldn't call them primitive data types in python.
Maybe there is just 1 primitive type: type? Or none at all?
Merely having `__add__` (without `__radd__`) is enough to add two MyNumber classes together in your case.
>It mostly exists to support type annotations,
The link for "type annotations" is broken.
Just `__add__` doesn't work for me. I get:
TypeError: unsupported operand type(s) for +: 'int' and 'Number'
class Number:
def __add__(self, x):
return 42 + x
num = Number()
print(num + num)
(Pseudo code, let's call them num1 and num2 for better readability)
num1 + num2
= num1.__add__(num2)
= num2 + 42 (that's why the order is important)
= num2.__add__(42)
= 42 + 42
= 84I'm not sure about that - it's still `<a href="mypy-guide">type annotations</a>` which jumps to https://sadh.life/post/builtins/mypy-guide and then jumps to your homepage.
I've been using that instead of print debug, it's been great.
should be
> for index, item in enumerate(menu, start=1):
for the example to be correct :)