At this point we might want to step back and wonder why this has become so hard.
TeX is a poor foundation to build upon, so I don't think LuaTeX, despite being the best solution currently, is the right direction to explore.
I still disagree with this characterisation. TeX is a really beautiful solution to a non-trivial problem, and its implementation in WEB is really quite interesting. LaTeX adds quite a lot of complexity and messiness, but does also add very useful functionality and abstraction away from low-level typesetting. Org-mode and similar 'mark-up' languages are very nice in more rigidly-defined domains, but they don't have the power of (La)TeX for academic writing. I don't see any real 'competitors' to (La)TeX that aren't themselves just adding layers on top of TeX (for better or worse). TeX is nearly as old as (original) Emacs and has similarly become complex in perhaps not-entirely-ideal ways, but - like Emacs - there's no substitute.
Imaging the capabilities of TeX with a good programming language.
I don't find WEB beautiful. In my opinion, literate programming (at least in this case) is a pretext to hide a terrible, unexpressive programming language.
As you're saying, Emacs had the good taste to use a Lisp back in those days, so it really is a pity that TeX didn't.