The real downside is that there's a better, proven way to do the same effective thing, which is make a database-only compromise require additional work, without rolling your own crypto. It also supports doing things retroactively for real (not some of the hacks being discussed in this thread) and key-rotation. All the upsides, with none of the downsides.

Please do not ever consider "rolling your own crypto" a walk in the park. Unless you have a serious security background and some actual cryptography education and research never, never, NEVER do this. It is not negligible, and it is not safe.