I took a VERY similar approach for generalized universal lego sorter and got to about 120 classes of parts recognizing acceptably well for about 70 of the classes. I used Mathias' articles as a guide.
I was picking and choosing physically varied lego to find difficult pieces to sort. Beveled pieces tended to be the worst. Small pieces hid important information maybe? I suspect any solution will require mirrors to get better information.
I also got discouraged because I wanted to sort into about 2-300 different "buckets" and found the mechanical piece of that intimidating and got distracted by some other fun project. Maybe I'll return to it.
edit: I followed up by reading the longer linked article. I can't believe how similar the software the two of us wrote was. There are some labeling enhancements that I were probably the next step that I appreciate he took.