Martin Ruckert (of MMIX fame) has been working on something like this: http://tug.org/practicaltex2018/slides/ruckert.pdf / https://www.tug.org/TUGboat/Contents/listauthor.html#Ruckert... It seems he has a talk at the upcoming TUG conference to follow up on this.