You might want to try booting without graphics and accessing it via a serial port, to see if that makes a difference. See this e-mail thread for a discussion:

https://www.mail-archive.com/smartos-discuss@lists.smartos.o...