A theoretical performance penalty, which, in practice, you just right there told us was a non-factor, right? If you are connection pooling. Which you should be.
But yeah, in just about _any_ sort of application using SSL, if you're not connection pooling your performance is going to be seriously sunk. I demonstrated the same thing ruby HTTP libraries and SSL here: http://bibwild.wordpress.com/2012/04/30/ruby-http-performanc...
MySQL, http, anything -- SSL connection establishing is expensive. If you are making lots of SSL connections, it's going to be expensive. If you can pool and reuse SSL connections, you always always want to, it will make a big difference.