To get a sense of how this could work, try sketch mode in Autodesk Inventor (there's a free 30 day demo) You can specify that a point must be coincident to an edge, that an edge must be coincident to an edge, that something must have specified dimensions, that some dimension must have a numerical relationship with another dimension, etc. Inventor goes further, supporting constraints on diagonal lines, circles, arcs, ellipses, etc. Whether layout should support curves is an interesting question, but the technology exists to make that work.
The people who designed HTML5 and CSS thought procedurally, not geometrically. It shows. Designers think geometrically, and a geometric constraint system would make sense to designers.