It's called a hardware security module (HSM) [1]. The basic idea is to store the private key in the tamper-resistant hardware, along with a microprocessor to perform the basic encryption options you need. They can be relatively cheap, like a smart card + usb smart card reader, to very expensive, like an IBM 4764. Yubikey sells a usb HSM if you want to run your own authentication server for around $500.

There's a whole chapter dedicated to HSMs in Security Engineering [2], which is available online. There are clever ways to attack them, and yes, the booby trap idea has been done, typically by using something light-sensitive. I'm not aware of any concrete-encased HSMs, however...

It's an interesting topic. That are lots of challenges around them too. It will probably have a battery backup, so how do you allow someone to replace the battery without wiping the keys? Or can only people with access to the keys replace the batteries? That won't work if you're doing mathematical secret sharing, however, since there's no physical way to do that.

  [1] http://en.wikipedia.org/wiki/Hardware_security_module
  [2] http://www.cl.cam.ac.uk/~rja14/book.html