I think there would be a different answer depending on who you ask, but some of the biggest downsides as of now:

1. Lack of DANE/TLSA support in browsers

2. Lack of SSL since the CAs aren't issuing for Handshake Names. This is independently solved by #1.