Sensible type-annotated python code could be so much faster if it didn't have to assume everything could change at any time. Most things don't change, and if they do they change on startup (e.g. ORM bindings).
class SomeClass
def init(self)
self.x = 0
def SomeMethod(self)
q = self.x
## do stuff with q, because otherwise you're dereferencing self.x all the damn timei don't understand what you think is nuts about this. it's an interpreted language and the word `self` is not special in any way (it's just convention - you can call the first param to a method anything you want). so there's no way for the interpreter/compiler/runtime to know you're accessing a field of the class itself (let alone that that field isn't a computed property or something like that).
lots of hottakes that people have (like this one) are rooted in just a fundamental misunderstanding of the language and programming languages in general <shrugs>.
And getting rid of descriptors would be a _fundamental change to the language_. An immeense one. Loads of features are built off of descriptors or descriptor-like things.
And what you're complaining about is also not true in Javascript world either... I believe you can build descriptor-like things in JS now as well.
_But_ if you want that you can use stuff like mypyc + annotations to get that for you. There are tools that let you get to where you want. Just not out of the box because Python isn't that language.
Remember, this is a scripting language, not a compiled language. Every optimization for things you talk about would be paid on program load (you have pyc stuff but still..)
Gotta show up with proof that what you're saying is verifiable and works well. Up until ~6 or 7 years ago CPython had a concept of being easy to onboard onto. Dataflow analyses make the codebase harder to deal with.
Having said all of that.... would be nice to just inline RPython-y code and have it all work nicely. I don't need it on everything and proving safety is probably non-trivial but I feel like we've got to be closer to doing this than in the past.
I ... think in theory the JIT can solve for that too. In theory
This is the fundamental issue and "elephant in the room" that everyone is seems to be overlooking, and putting under the carpet.
The extreme compiled type language guys going gung-ho with very slow to compile and complicated Rust (moreso than C++), while the rest of the world gladly hacking their shiny ML/AI codes in scripting language aka Python "the glue duct tapes language" with most if not all the fast engine libraries (e.g PyTorch) written in unsafe C/C++.
The problem is that Python was meant for scripting not properly designed software system engineering. After all it's based on ABC language for beginners with an asterisk attached "intended for teaching or prototyping, but not as a systems-programming language" [1].
In ten years time people will most probably look in horror at their python software stacks tech debt that they have to maintain for the business continuity. Or for their own sanity, they will rewrite the entire things in much more stable with fast development and compiled modern language eco-system like D language with native engine libraries, and seamless integration C, and C++ (to some extend) if necessary.
[1] ABC (programming language)
What something was meant to do has never, ever stopped people. People find creative ways to use tools in unintended ways all the time. It's what we do.
We can call this dumb or get misanthropic about it, or we can try to understand why people all over the world choose to use Python in "weird" ways, and what this tells us about the way people relate to computing.
I regret to inform you that there are _loads_ of multi-decades-old Python stacks at this point.
On the micro level I'll be like "ugh wish I wasn't paying the costs of Python" decently enough. But on the macro level I don't regret Python stacks. At least not when looking at the alternatives.
Tho I will admit I'm a bit mystified at data science stuff in particular persisting in Python. Lots of CPU churn even if the underlying libs are all C extensions.
Yes, like they did for JavaScript!
For example, numbers and strings are immutable objects in Python. If self.x is a number and its numeric value is changed by a method call, self.x will be a different object after that. I'd dare say people expect this to work.
now, functional languages don't have this problem at all.
This is a Python specific problem caused by everything being boxed by default and the interpreter does not even know what's in the box until it dereferences it, which is a problem that extends to the "self" object. In contrast in C++ the compiler knows everything there's to know about the type of this which avoids the issue.
struct S { const int x; int f() const; }; int S::f() const { int a = x; printf("hello\n"); int b = x; return a-b; }
The compiler can't reuse 'x' unless it's able to prove that it definitely couldn't have changed during the `printf()` call - and it's unable to prove it. The member is loaded twice. C++ compilers can usually only prove it for trivial code with completely inlined functions that doesn't mutate any external state, or mutates in a definitely-not-aliasing way (strict aliasing). (and the `const` don't do any difference here at all)
In Python the difference is that it can basically never prove it at all.
I would say it is part python being highly dynamic and part C++ being full of undefined behavior.
A c++ compiler will only optimize member access if it can prove that the member isn't overwritten in the same thread. Compatible pointers, opaque method calls, ... the list of reasons why that optimization can fail is near endless, C even added the restrict keyword because just having write access to two pointers of compatible types can force the compiler to reload values constantly. In python anything is a function call to some unknown code and any function could get access to any variable on the stack (manipulating python stack frames is fun).
Then there is the fun thing the C++ compiler gets up to with varibles that are modified by different threads, while(!done) turning into while(true) because you didn't tell the compiler that done needs to be threadsafe is always fun.
That's not the whole thing, what is going on. Every attribute access is a function call to __getattr__, that can return whatever object it wants.
bar.foo (...) is actually bar.__getattr__ ('foo') (bar, ...)
This dynamism is what makes Python Python and it allows you to wrap domain state in interface structure.
Did you miss the part where I explained to you there's no way to identify that it's a member variable?
> Nobody in the real world expects this behaviour
As has already been explained to you by a sibling comment you are in fact wrong and there are in fact plenty of people in the real world who do actually expect this behavior.
So I'll repeat myself: lots of hottakes from just pure. Unadulterated, possibly willful, ignorance.
"Did you miss the part where I explained to you there's no way to identify that it's a member variable?"
No, you you did miss the case where that in itself can be considered nuts - or at least an unfortunate early decision.
"this just how things are dunn around diz here parts" is not an argument.
Please explain to us then how exactly you would infer a variable with an arbitrary name is actually a reference to the class instance in an interpreted language.
Did I stutter when I wrote about "an unfortunate early decision"? Who said it has to be "an arbitrary name"?
Even so, you could add a bloody marker announcing an arbitrary name (which 99% would be self anyway) as so, as an instruction to the interpreter. If it fails, it fails, like countless other things that can fail during runtime in Python today.
"That's how it's been since forever, it's an essential part of country X"
"Yes, and it's a badly designed part".
"But now you are no longer talking about the way country X works, but the way you want country X to work - and that has nothing to do with country X."
See how the argument quickly degenerates?
This is not a side implementation detail, that they got wrong, this is a fundamental design goal of Python. You can find that nuts, but then just don't use Python, because that is (one of) that things, that make Python Python.
The language supports multiple threads and doesn’t have private fields (https://docs.python.org/3/tutorial/classes.html#private-vari...), so the runtime cannot rule out that the value gets changed in-between.
And yes, it often is obvious to humans that’s not intended to happen, and almost never what happens, but proving that is often hard or even impossible.
There is no such thing as 'successive references to the same member value' here. It's not that you look up the same object and it can change, it's that you are not referring to the same object at all.
self.x is actually self.__getattr__('x'), which can in fact return a different thing each time. `self.x` IS a string lookup and that is not an implementation detail, but a major design goal. This is the dynamism, that is one of the selling points of Python, it allows you to change and modify interfaces to reflect state. It's nice for some things and it is what makes Python Python. If you don't want that, use another language.
The name `self` is a convention, yes, but interestingly in python methods the first parameter is special beyond the standard "bound method" stuff. See for example PEP 367 (New Super) for how `super()` resolution works (TL;DR the super function is a special builtin that generates extra code referencing the first parameter and the lexically defining class)
A common approach is hidden classes that work much like classes in other languages. Reading a simple int property just reads bytes at an offset from the object pointer directly. Upon entry to the method bits of the object are tested and if the object is not known to be simple it escapes into the full dynamic machinery.
I don't know if those exact techniques would work for Python but this is not an either-or situation.
See also: modern Objective-C msg_Send which is so fast on modern hardware for the fast-path it is rarely a performance bottleneck. Despite being able to add dynamic subclasses or message forward at runtime.
If these analyses don't apply and the callee could do anything, then of course the compiler can't keep the value hoisted. But a function call has to occur anyway, so the hoisted value will be pushed/popped from the stack and you might as well reload it from the object's field anyway, rather than waste a stack slot.
There might be common scenarios where this had a real, significant performance impacts, E.G. use-cases where it's such a bottle-neck in the interpreter that it measurably affects warm-up time. Also, string manipulation seems like the kind of thing you see in small scripts that end before a JIT even kicks in but that are also called very often (although I don't know how many people would reach for Java in that case.
EDIT: also, if you're a commercial entity trying to get people to use your programming language, it's probably a good idea to make the language perform less bad with the most common terrible code. And accidentally quadratic or worse string manipulation involving excessive calls to trim() seems like a very likely scenario in that context.
You are right that it is not needed often, but there is often somewhere a part in the library stack that does exactly this, to expose a nice interface.
Javascript implementations do their own magic since most objects aren't constantly mutating their prototypes or doing other fun things. They effectively fast-path property accesses and fallback if that assumption proves incorrect.
Couldn't python tag objects that don't need such dynamism (the vast majority) so it can take the fast path on them?
Funnily enough I’ve found Python to be excellent for modelling my problem domain with Pydantic (so far basically unparalleled, open for suggestions in Go/Rust), while the language also gets out of my way when I get creative with list expressions and the like. So overall, still it is extremely productive for the work I’m doing, I just need to spin up more containers in prod.
And what prevents someone from designing such a language?
https://github.com/abilian/p2w
NB: some preliminary results:
p2w is 4.03x SLOWER than gcc (geometric mean)
p2w is 5.50x FASTER than cpython (geometric mean)
p2w is 1.24x FASTER than pypy (geometric mean)Definitely, but then it wouldn't be Python. One of the core principles of Python's design is to be extremely dynamic, and that anything can change at any time.
There are many other, pretty good, strictly dynamically typed languages which work just as well if not better than Python, for many purposes.
And when Python is a mainstream language on top of which large, globally known websites, AI tools, core system utilities, etc are built, we should give up the purity angle and be practical.
Even the new performance push in Python land is a reflection of this. A long time ago some optimizations were refused in order to not complicate the default Python implementation.
1. I doubt that many Python devs are reading my random comments on a random site.
2. I doubt that those Python devs that stumble upon my comments care about them.
I'm a nobody on the internet. In the past I would have commented that maybe I'm a dog, but now it's fashionable to be an AI agent.
> Most engineers never get to choose the programming language used for 90% of their professional projects.
If it was up to me, there are plenty of languages to choose from that meet my technical needs just fine, but the political friction of getting all of my colleagues (most of whom are not software engineers at all) to use my language of choice is entirely insurmountable. Therefore, I have a vested interested in seeing practical changes to Python. The existence or invention of other languages is irrelevant.
It is called type hints, and is already there. TS typing doesn't bring any perf benefits over plain JS.
class Foo:
__slots__ = ("a", "b")
a: int
b: float
* a user can define subtype `class my_int(int)`, so you cannot optimize the layout of `class Foo`
* the builtin `int` and `float` are big-int like numbers, so operations on them are branchy and allocating.
and the fact that Foo is mutable and that `id(foo.a)` has to produce something complicates things further.
Therefore Python has no use for TS-like superset, because it already has facilities for static analysis with no bearing on runtime, which is what TS provides.
1) Add TS like language on top of Python in backwards compatible way
2) Introduce frozen/final runtime types
3) Use 1 and 2 to drive runtime optimizations
From all posts it looks like what OP wants is a different language that looks somewhat like Python syntax-wise, so calling for "backwards-compatible" superset is pointless, because stuff that is being demanded would break compatibility by necessity.
Then it wouldn't be Python any more.
Even type annotations, though useful, can get in the way for certain tasks.Betting on things like these to speed up things would be a mistake, since it would kind of force you to follow that style.
Anything that accelearates things should rely on run-time data, not on type annotations that won't change.
As far as I can tell, it only ever existed to make PyPy possible, and was only defined/specified in terms of PyPy's needs.
You could make this clean break and call it Python 4 but frankly I fear it won't be Python anymore.
I think it's not so much that Pytest is using obscure language features (decorators are cool and the obvious choice for a lot of this kind of stuff) but that it wants too much magic to happen in terms of how the "fixtures" automatically connect together. I would think that "Explicit is better than implicit" and "Simple is better than complex" go double for tests. But things like `pytest.mark.parametrize` are extremely useful.
All the dynamism from Python should stay where it is.
Just JIT and remember a type maybe, but do not force a type from a type hint or such things.
As a minimum, I would say not relying on that is the correct thing. You could exploit it, but not force it to change the semantics.
TL;DR: SPy is a variant of Python specifically designed to be statically compilable while retaining a lot of the "useful" dynamic parts of Python.
The effort is led by Antonio Cuni, Principal Software Engineer at Anaconda. Still very early days but it seems promising to me.
Great idea, but I'm not convinced that they learned anything from the Python 2 to 3 transition, so I wouldn't hold my breath.
If you want a language system without contempt for backward compatibility, you're probably better off with Java/C++/JavaScript/etc. (though using JS libraries is like building on quicksand.) Bit of a shame since I want to like Python/Rust/Swift/other modern-ish languages, but it turns out that formal language specifications were actually a pretty good idea. API stability is another.
b = ComplexObject (...)
# do things with b
def foo (self, arg=b):
# use b
return foo
Should the semantic of this be any different? :
def foo (self, arg=ComplexObject (...)):
ComplexObject = listdef foo(self, arg=expression):
could, and should work as if it was written like this (pseudocode)
def foo(self, arg?): if is_not_given(arg): arg=expression
if "expression" is a literal or a constructor, it'd be called right there and produce new object, if "expression" is a reference to an object in outer scope, it'd be still the same object.
it's a simple code transformation, very, very predictable behavior, and most languages with closures and default values for arguments do it this way. Except python.
def foo (self, arg=lambda : expression):
> most languages with closures and default values for arguments do it this way.
Do these also evaluate function definitions at runtime?
The entire point of it being an executable statement is to let you change things on the fly. This is key to how the REPL works. If I have `def foo(): ...` twice, the second one overwrites the first. There's no need to do any checks ahead of time, and it works the same way in the REPL as in a source file, without any special logic, for the exact same reason that `foo = 1` works when done twice. It's actually very elegant.
People who don't like these decisions have plenty of other options for languages they can use. Only Python is Python. Python should not become not-Python in order to satisfy people who don't like Python and don't understand what Python is trying to be.
There might not be that many of them, depending on how you count, but they're not rare in the slightest. For example, you have to use `is` in the common case where you want the default value of a function argument to be an empty list.
def silly_append(item, orig=[]):
return orig + [item]
Using, for example, `()` as a default argument, and cleaning up your logic to not do those mutations, is commonly simpler and more expressive. A lot of the community has the idea that a tuple should represent heterogeneous fixed-length data and a list should be homogeneous; but I consider (im)mutability to be a much more interesting property of types.
It has nothing to do with whether the list is empty. It has nothing to do with lists at all. It's the behaviour of default arguments.
It happens at the time that the function object is created, which is during runtime.
You only notice because lists are mutable. You should already prefer not to mutate parameters, and it especially doesn't make sense to mutate a parameter that has a default value because the point of mutating parameters is that the change can be seen by the caller, but a caller that uses a default value can't see the default value.
The behaviour can be used intentionally. (I would argue that it's overused intentionally; people use it to "bind" loop variables to lambdas when they should be using `functools.partial`.)
If you're getting got by this, you're fundamentally expecting Python to work in a way that Pythonistas consider not to make sense.
It's just slightly annoying having to work around this by defaulting to None.
https://github.com/python/cpython/blob/3.14/Lib/json/encoder...
Default value is evaluated once, and accessing parameter is much cheaper than global
Still churning on it, will probably publish it and do a proper blog post once I've built something interesting with the language itself.