pcwalton's project Pathfinder (https://github.com/pcwalton/pathfinder) may eventually lead to an efficient GPU-based vector renderer, which would make this feasible.
In any case, I suspect something like lyon (https://github.com/nical/lyon) would be a better fit than pathfinder, which is both a) not super portable and b) very focused on rendering text, which it does very well.