Access to primitives such as lines, polygons or pixels, you say? Canvas, SVG, and WebGL fit that bill.
Either way, WebGL/canvas operate as child elements in an html page, you still need to use javascript, and SVG isn't GPU accelerated. It's still, overall, an ugly mess. But definitely, with improvements, canvas, svg and webgl are all candidates for my aforementioned "next steps."