While I don't have a solution, I'm facing very similar challenges.
I really liked the object model of paper.js, but the very same layers/object model led to some performance issues ol large data sets. So im stuck with custom canvas implementation with no vector/high res export solution.