I guess performance was considered less critical than clarity/flexibility. But it seems that people are discovering that complex code tend to be hard to read/modify no matter the language...
A lot of it is also asynchronous for performance: the Python code just enqueues more work to a queue which some native C++ code processes. For TensorFlow the Python code traces an entire computation graph that is stored a protobuf and then executed by a C++ native stack, potentially remotely/distributed. Serving ML with TensorFlow does not involve any Python code in many scenarios.
Python is still quite useful for scientist to quickly glue everything together, and to describe their dataset, or when they collect result and need to produce graphs or other data analyses.
Most quants do not want to learn complex build systems that have quirky behavior on different platforms, wait for very long compile times when making small changes to the code, dense and incomprehensible error messages, and a host of painful problems that one has to consider when writing C++.
Python just works really well on almost all platforms.
The biggest downside of Python is its parallelism, which means there is a lot of hackyness around writing parallel code. In most cases we can break things down and run different tests independently of one another, but in many other cases we have to make use of awkward workarounds, use multiprocessing, and other tricks.
Which C++ version you guys are using?
The reasoning wat the same, all the heavy lifting done with fast native code, and everything written in Lua was mostly glue code without real performance impact.
Turns out the engine was slow and difficult to maintain because of the many interfaces. They ditched it a few years later...
I think you could say the same thing about javascript on the web, this is mostly glue code, all the heavy lifting is done by optimized C++ or Rust code in the browser. And yet it is difficult to run a web mail client if your computer is a bit old...
This is probably related in some ways to Amdahl's law https://en.wikipedia.org/wiki/Amdahl%27s_law
https://www.gamasutra.com/view/news/213647/Epics_Tim_Sweeney...
However a couple of years later and Epic is bringing scripting back with Verse
https://twitter.com/saji8k/status/1339709691564179464?s=20
Because the problem is not the scripting, but how it is done.
I imagine that the engine you mention did not use any kind of compiler for Lua, e.g. LuaJIT, nor batch calls across the marshalling layer.
Same happens on the Web, JavaScript JIT can outperform WebAssembly if one keeps switching all the time between layers, instead of doing batch requests.
The code was a mess, absolutely no batching, quite the contrary.