You ignored an important lesson IMO.

Use the right tool for the job.

Your entire journey here is battling with Python and not the problem space. Your improvements are solely measured by how your code leaves Python problem solving space, so you end up writing what's probably unpythonic and unmaintainable Python.

Why?

Rewrite in Go, Rust or C++ or whatever.

If you need to go faster than Python, don't use Python. Python is great for everything else. Its overhead and sluggish execution are consequences of its greatest strength.