I think I'll just use Julia, since it has great libraries for doing matrix operations and mapping loops etc onto GPUs and handling indices etc. And if you need some Python library it's easily available using the fantastic PyCall library.

This is a big improvement over numpy, but I don't see much of a compelling reason to go back to Python.