kenjin4096

Woops, I meant to write tail call optimization not tail calls. Shouldn't have used the term interchangeably. Yeah I'm aware that C doesn't have proper tail calls. Thanks for the correction though.

I think Anton is replying to me in that LWN article IIRC. I personally didn't know C only had tail calls that late and learnt something new there!

On the other hand, I am pretty new to the compiler space myself, and I count early 2000s as a pretty long time ago, though again it is not that far back considering how long other language implementations had tail calls like in ML or variants since 1980-90s.

This is an intuitive step and I kind of did something similar initially (I simplified the diagram a lot, but it was basically what you are suggesting). However, it slowed down the computed goto interpreter by 6% (though not the tail calling interpreter).

The other comment about interpreters being greatly affected by instruction cache is right. The other problem is that PGO in CPython's test suite could be improved and when you have the full thing as a switch-case and trigger the profiling mode, PGO then mistakenly thinks that the RECORD_INST_X section is hot (when it is clearly not). This causes a pessimization in the interpreter loop leading to the 6% slowdown.

Pablo is a great speaker and I love Tachyon! However, this is a different type of/target for profiling. Tachyon profiles the entire CPython interpreter, this blog post is just type profiling/recording and interpreter path profiling for the CPython core interpreter loop itself.

Update: I put up a PR to document the trace recording interpreter https://github.com/python/cpython/pull/146110

> I've never seen where the high level discussions were happening

Thanks for your interest. This is something we could improve on. We were supposed to document the JIT better in 3.15, but right now we're crunching for the 3.15 release. I'll try to get to updating the docs soon if there's enough interest. PEP 744 does not document the new frontend.

I wrote a somewhat high-level overview here in a previous blog post https://fidget-spinner.github.io/posts/faster-jit-plan.html#...

> does this mean each opcode is possibly split into two (or more?) stencils, with and without removed increfs/decrefs?

This is a great question, the answer is not exactly! The key is to expose the refcount ops in the intermediate representation (IR) as one single op. For example, BINARY_OP becomes BINARY_OP, POP_TOP (DECREF), POP_TOP (DECREF). That way, instead of optimizing for n operations, we just need to expose refcounting of n operations and optimize only 1 op (POP_TOP). Thus, we just need to refactor the IR to expose refcounting (which was the work I divided up among the community).

If you have any more questions, I'm happy to answer them either in public or email.

I implemented most of the tracing JIT frontend in Python 3.15, with help from Mark to clean up and fix my code. I also coordinated some of the community JIT optimizer effort in Python 3.15 (note: NOT the code generator/DSL/infra, that's Mark, Diego, Brandt and Savannah). So I think I'm able to answer this.

I can't speak for everyone on the team, but I did try the lazy basic block versioning in YJIT in a fork of CPython. The main problem is that the copy-and-patch backend we currently have in CPython is not too amenable to self-modifying machine code. This makes inter-block jumps/fallthroughs very inefficient. It can be done, it's just a little strange. Also for security reasons, we tried not to have self-modifying code in the original JIT and we're hoping to stick to that. Everything has their tradeoffs---design is hard! It's not too difficult to go from tracing to lazy basic blocks. Conceptually they're somewhat similar, as the original paper points out. The main thing we lack is the compact per-block type information that something like YJIT/Higgs has.

I guess while I'm here I might as well make the distinction:

- Tracing is the JIT frontend (region selection).

- Copy and Patch is the JIT backend (code generation).

We currently use both. PyPy uses meta-tracing. It traces the runtime itself rather than the user's code in CPython's tracing case. I did take a look at PyPy's code, and a lot of ideas in the improved JIT are actually imported from PyPy directly. So I have to thank them for their great ideas. I also talk to some of the PyPy devs.

Ending off: the team is extremely lean right now. Only 2 people were generously employed by ARM to work on this full time (thanks a lot to ARM too!). The rest of us are mostly volunteers, or have some bosses that like open source contributions and allow some free time. As for me, I'm unemployed at the moment and this is basically my passion project. I'm just happy the JIT is finally working now after spending 2-3 years of my life on it :). If you go to Savannah's website [1], the JIT is around 100% faster for toy programs like Richards, and even for big programs like tomli parsing, it's 28% faster on macOS AArch64. The JIT is very much a community effort right now.

[1]: https://doesjitgobrrr.com/?goals=5,10

PS: If you want to see how the work has progressed, click "all time" in that website, it's pretty cool to see (lower is faster). I have a blog explaining how we made the JIT faster here https://fidget-spinner.github.io/posts/faster-jit-plan.html.

Tier-ups for trace-based JITs have been explored before. You can find an example here https://dl.acm.org/doi/abs/10.1145/2398857.2384630 I know LBBV isn't technically tracing, but it's quite similar, so I think similar concepts apply.

> Generally not that much has happened in 5 years, sometimes 10-15% improvements are posted that are later offset by bloat.

Sorry but unless your workload is some C API numpy number cruncher that just does matmuls on the CPU, that's probably false.

In 3.11 alone, CPython sped up by around 25% over 3.10 on pyperformance for x86-64 Ubuntu. https://docs.python.org/3/whatsnew/3.11.html#whatsnew311-fas...

3.14 is 35-45% faster than CPython 3.10 for pyperformance x86-64 Ubuntu https://github.com/faster-cpython/benchmarking-public

These speedups have been verified by external projects. For example, a Python MLIR compiler that I follow has found a geometric mean 36% speedup moving from CPython 3.10 to 3.11 (page 49 of https://github.com/EdmundGoodman/masters-project-report)

Another academic benchmark here observed an around 1.8x speedup on their benchmark suite for 3.13 vs 3.10 https://youtu.be/03DswsNUBdQ?t=145

CPython 3.11 sped up enough that PyPy in comparison looks slightly slower. I don't know if anyone still remembers this: but back in the CPython 3.9 days, PyPy had over 4x speedup over CPython on the PyPy benchmark suite, now it's 2.8 on their website https://speed.pypy.org/ for 3.11.

Yes CPython is still slow, but it's getting faster :).

Disclaimer: I'm just a volunteer, not an employee of Microsoft, so I don't have a perf report to answer to. This is just my biased opinion.

Woops, thanks for noticing, fixed!

Yeah, I believe that statement and it seems to hold true for MSVC as well. Thanks for your work inspiring all of this btw!

Thank you very much for the kind words, that means a lot to me!

So it seems I was wrong, [[msvc::musttail]] is documented! I will update the blog post to reflect that.

https://news.ycombinator.com/item?id=46385526

Thanks :), that was indeed my intention. I think the previous 3.14 mistake was actually a good one on hindsight, because if I didn't publicize our work early, I wouldn't have caught the attention of Nelson. Nelson also probably wouldn't have spent one month digging into the Clang 19 bug. This also meant the bug wouldn't have been caught in the betas, and might've been out with the actual release, which would have been way worse. So this was all a happy accident on hindsight that I'm grateful for as it means overall CPython still benefited!

Also this time, I'm pretty confident because there are two perf improvements here: the dispatch logic, and the inlining. MSVC can actually convert switch-case interpreters to threaded code automatically if some conditions are met [1]. However, it does not seem to do that for the current CPython interpreter. In this case, I suspect the CPython interpreter loop is just too complicated to meet those conditions. The key point also that we would be relying on MSVC again to do its magic, but this tail calling approach gives more control to the writers of the C code. The inlining is pretty much impossible to convince MSVC to do except with `__forceinline` or changing things to use macros [2]. However, we don't just mark every function as forceinline in CPython as it might negatively affect other compilers.

[1]: https://github.com/faster-cpython/ideas/issues/183 [2]: https://github.com/python/cpython/issues/121263

Thanks for reading! For now, we maintain all 3 of the interpreters in CPython. We don't plan to remove the other interpreters anytime soon, probably never. If MSVC breaks the tail calling interpreter, we'll just go back to building and distributing the switch-case interpreter. Windows binaries will be slower again, but such is life :(.

Also the interpreter loop's dispatch is autogenerated and can be selected via configure flags. So there's almost no additional maintenance overhead. The main burden is the MSVC-specific changes we needed to get this working (amounting to a few hundred lines of code).

> Impact on debugging/profiling

I don't think there should be any, at least for Windows. Though I can't say for certain.

Got it. I'll try to set one up this weekend.

I'm the author of the thread you linked. Community stewardship is actually happening in some form or another now.

3.15 has some JIT upgrades that are in-progress. This has a non-exhaustive list of them https://docs.python.org/dev/whatsnew/3.15.html#upgraded-jit-...

Hey Maxime!

> whereas AFAIK the Python JIT project was lead by a student.

I am definitely not leading the team! I am frankly unqualified to do so lol. The team is mostly led by Mark Shannon, who has 10+ years of compiler/static analysis experience as well. The only thing I initially led was the optimizer implementation for the JIT. The overall design to choose tracing, to use copy and patch, etc. were other people.

> However they decided to ignore my advice and go with their own unproven approach.

Your advice was very much appreciated and I definitely didn't ignore your advice. I just don't have much say over the initial architectural choices we make. We're slowly changing the JIT based on data, but it is an uphill battle like you said. If you're interested, it's slowly becoming more like lazy basic block versioning https://github.com/python/cpython/issues/128939

You did great work on YJIT, and I am quite thankful for that.

Hi, author of the post here, stability indeed has been a priority. There are some points which are not exactly the case though:

> - Most of the work has just been plumbing. Int/float unboxing, smarter register allocation, free-threaded safety land in 3.15+.

The first part is true, but for the second sentence: none of that is guaranteed to land in 3.15+. We proposed to land them, that doesn't mean they will. Landing a PR in CPython is subject to maintainer time and reviewer approval, which doesn't always happen. I proposed a few optimizations for 3.14 that never landed.

> Most JIT optimizations are currently off by default or only triggers after a few thousand hits

It is indeed true we only trigger after a few thousand hits, but all optimizations that we currently have are always enabled. We don't sandbag the JIT on purpose.

This is a good point. We already observed this in our LTO and PGO builds for the computed goto interpreter. On modern compilers, each LTO+PGO build has huge variance (1-2%) for the CPython interpreter. On macOS, we already saw a huge regression in performance because Xcode just decided to stop making LTO and PGO work properly on the interpreter. Presumably, the tail call interpreter would be immune to this.

That's a fair question.

The blog post mentions it brings a 1-5% perf improvement. Which is still significant for CPython. It does not complicate the source because we use a DSL to generate CPython's interpreters. So the only complexity is in autogenerated code, which is usually meant for machine consumption anyways.

The other benefit (for us maintainers I guess), is that it compiles way faster and is more debuggable (perf and other tools work better) when each bytecode is a smaller function. So I'm inclined to keep it for perf and productivity reasons.

We don't normally test with bleeding-edge compilers on the faster cpython benchmarks page because that would invalidate historical data. E.g. if 2 years ago we used GCC 11 or something to compile and run a benchmark, we need to run it with GCC 11 again today to get comparable results.

Clang 19 was released last year. We only benched it a few months ago. We did notice there was a significant slowdown on macOS, but that was against Xcode Clang, which is a different compiler. I thought it might've been an Xcode thing, which in the past has bit CPython before (such as Xcode LTO working/not working versus normal Clang) so I didn't investigate deeper (facepalming now in retrospect) and chalked it up to a compiler difference.

TLDR: We didn't run benchmarks of clang 19 versus 18. We only ran benchmarks of clang 19 versus gcc, Xcode clang, and MSVC. All of which are not apples-to-apples to Clang 19, so I naiively thought it was a compiler difference.

EDIT: As to how we could improve this process, I'm not too sure, but I know I'll be more discerning when there's a >4% perf hit now when upgrading compilers.

Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython.

First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue.

Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug.

I posted an apology blog post here. https://fidget-spinner.github.io/posts/apology-tail-call.htm...