So just do it with VRML5 like HTML5.
<cube id="myCube" x="0" y="0" z="0" width="10px" height="10px" length="10px" onClick="someFoo()" onMouseOver="someOtherFoo()" onTouchStart="someBar()">
<video src="blah.mp4" rotateX="45deg" rotateY="30deg" autoplay="true" loop="true" onClick="document.querySelector('#myCube').rotateX(45);">
...
No reason the above can't be done. But many people will come up with lame excuses about why we shouldn't have nice things.