There was JS Engine performance war before Chrome v8 came around. From JavaScriptCore to whole bunch of monkeys [1] from Mozilla. Google v8 just makes the competition super heated and every one of them were working around the clock trying to out compete each other on the latest benchmarks. ( That was the dark era when Browser only cares about JS benchmarks scores and nothing in real world )
It would not be an exaggeration to say all the man hours on JS JIT is more than Perl, Ruby and PHP's VM combined. That is why I often said Ruby is the only Top 10 language that gets little to no funding and backing of FAANG. So both YJIT and Sorbet are much needed contribution from Stipe and Shopify. Along with help from Github and Gitlab ( hopefully :) ).
The VMs that gets more resources than JS would be JVM and .Net. And JVM is a monster on its own. Easily a multi billion dollar of investment over all these years. Or something using less man hours and resources like LuaJIT. But then Mike Pall is a super human.
[2] https://github.com/sagemathinc/JSage/tree/main/packages/jpyt...
MRI however implements some of them (integers, floats, symbols, true, false, nil, I might have forgotten one or two) using type tagged values instead of pointers to objects.
Type-tagging is easier to make fast for a highly gc'd dynamic language, as it reduces gc pressure substantially to not have to allocate lots of small objects without massive amounts of complex optimisations.
I think one could radically change the way Python objects work internally, and have the C foreign function interface (FFI) wrap every object passed to a C extension in an API/ABI-preserving facade (which itself would wrap any objects returned from its methods). However, this would probably greatly slow down C extensions, which are often performance-critical sections of Python applications. It's also possible that there are portions of the C extension API that expose enough details of object internals to even make such facades herculean to implement. (I've only written some small simple C extensions and am not very familiar with the API.)
V8 didn't have to deal with API/ABI compatibility with any preexisting C extensions that may have made too many abstraction-violating assumptions about how objects and the VM worked.
Breaking too many important C extensions would almost certainly send Python the way of Perl 6.
Edit: as an aside, a big difficulty with JS is that objects can have their prototype changed arbitrarily at runtime. Even with Metaclass programming in Python, the class of an object can't be changed after creation, making it much easier to cache/memoize dynamic method dispatch. On the other hand, high performance implementation of Python's bound methods require a bit more flow analysis than you need in JS. In Python, if you write f = x.y, f is a "bound method" (a closure that ensure x is passed as "self" to y). It's expensive to create closures for each and every method invocation, so a high performance implementation would need to do a bit of static analysis to identify which method look-ups are used purely for invocation, and which look-ups need to create the closures because the method itself is passed around or stored in a variable.
HPy is building an API abstraction layer which is designed to be used with both the CPython API and JITs. However, IIUC they are not proposing any changes to CPython itself, but rather to provide a smaller API surface and fewer JIT impedance mismatches when extensions are built against something other than CPython. The lead developer is a longtime PyPy developer.
If you're interested, I'd be happy to take look anyways and see if there are any easy, idiomatic performance changes that can made for the Julia code without changing the algorithm.
Other languages struggle in this regard. Comparatively I imagine far fewer developers working full time on Ruby/Python, not to mention they would have budget constraints to hire and retain talent.