I experimented with this approach on my blog[1]. I had an image of the handwritten blog post and used a feature in Inkscape to get SVG and make clickable text. I decided to place a little plain text in there to help Google search crawlers know what content is on the page. I really would prefer handwriting more.
My problems boil down to just that flow of the tools sucked. I didn't want to have to attach multiple pages together. Inkscape was honestly a bit clunky to work with and I didn't have any personal tools to generate the full html from the svg.
I might just be inspired to try again though.