You can get halfway there if you combine Xvfb's -fbdir option and Chafa's --watch option. Chafa supports the XWD format Xvfb puts out.

Have to do input with xdotool, though :)