I think the most important part of growing was focusing on high quality content. 50% of the visitors are entering iconfinder.com in the browser or clicking a bookmark. The site has grown because it was recommended by the existing users. The user base is homogenic (web designers and developers) who often work together in teams and therefore talk a lot together.
It was only recently (2010-2011) I started really focusing on SEO.