> ... wouldn't it be a better option to rely on python scripting by providing a plugin API

NeoVim is using Lua for this very reason. It fits the bill nicely, and is quite fast.