There's a whole chapter dedicated to HSMs in Security Engineering [2], which is available online. There are clever ways to attack them, and yes, the booby trap idea has been done, typically by using something light-sensitive. I'm not aware of any concrete-encased HSMs, however...
It's an interesting topic. That are lots of challenges around them too. It will probably have a battery backup, so how do you allow someone to replace the battery without wiping the keys? Or can only people with access to the keys replace the batteries? That won't work if you're doing mathematical secret sharing, however, since there's no physical way to do that.
[1] http://en.wikipedia.org/wiki/Hardware_security_module
[2] http://www.cl.cam.ac.uk/~rja14/book.htmlAs far as some of the issues you raised like battery replacement, I would treat them as disposable. In the next few years a complete computer will be available for so little that we will consider it disposable. Prepare a server once, enclose it, make sure the only way in or out is an SSH connection. When it's time to set up a clone to replace it, clone everything via an SSH session and trash the first server.