They didn't do it in a smart way. Part of the Self group joined forces with some other researchers to create a company called Animorphic which would build the world's "fastest Smalltalk" using the Self technology. It was called StrongTalk and had other interesting features like optional type declaration and extensive use of traits and mixins.

Given the buzz around Java they did a quick demo showing that language running on top of their technology, and in 1997 Sun bought them to use that HotSpot technology in their main Java implementation. So they paid a lot for what they previously already had but ignored.