I completely agree with this problem. While I haven't solved the full problem you are describing, including graphs and diagrams, I think they will all fit nicely into this application model soon.

The biggest problem with pens is palm rejection, which some recent hardware has gotten right. This actually works find on a screen you can only write on with the pen, like a graphics tablet, but people love their mobile games, so even laptops get capacitive touchscreens in most cases these days.

I cannot wait for good tools for managing mathematical work like we have for prose today, I think it will be a real productivity boost for students as well as professionals. So many workflows get forces into Excel and Word because they are the tools that people know. While they are great at some things and pretty good swiss-army knife products, they just don't work for managing a bunch of floating diagrams/images and lots of equations.