Whoah, that's so awesome. The M&M Man manipulation is nuts.
Jeff Han demos something really similar -- almost to the T -- in his TED presentation of the Microsoft Surface technology. He even mentions "the mesh," which leads me to wonder if it's the same rigid-as-possible approach. See http://www.youtube.com/watch?v=QKh1Rv0PlOQ at around 7:30.