Hacker News Reader

Comment by jawns | Hacker News Reader

I would imagine that this is a hobby project and you're losing cash on it. But what would be the parameters of a profitable business? At what level of scale (if any) would it have to operate? And is there a lot of competition in this space?

View on HN

Well, yes. It is definitely not going to make money if I count my time. But I learned an awful lot about machine learning and the present state of computer vision, far more than if I had tried to do that on something abstract. And when I look at the Lego bought-and-sold it seems to work out ok.

As for competition, yes, plenty, but all manual. Scaling up now that I have the software working is definitely an option but I have a good set of very well paying customers and not much can compete with that.

Frankly, the "present state of cv" as you call it, is far far ahead of the 5% classification errors you are running into on a standardized textbook problem such as this - the Legos are all blocky discrete plastic with super-saturated colors and very distinct features. You can easily drop the error by taking multiple pictures from different angles, using ImageDataGenerator during training ( since you are using Keras), especially with horiz and vert flips, and the usual set of tuning tricks - going deeper, dropout, l2 etc. No, the real takeaway is that you have built something very impressive - a tangible physical classifier that uses vgg to get the error down to 5% with very little effort. Going from there to 0.5% would be quite easy if you post the training set so a bunch of people can try different things, see what works best for this situation. Last week I trained a car to drive around a circular track with just a stupid mnist classifier, only 9 classes, 10 lines of Keras! State of cv is very far ahead these days.

That 5% is based on a relatively small set of images that is expanding rapidly (4K or so per run), so I expect that error to get lower quickly. Also, these images are stitched and so are far from perfect, the colors are all over the map due to discoloration and so on. Then there are certain nasty aliasing effects where a part will look exactly like another part from the angles the camera can look at the part. This is fixable but will require (yet) another redesign of the optics.

Dropout is at .5 both for conv and connected layers.

All those things you mention have already been done and yet the error is as high as it is. But this does not surprise me at all, the difference is that the pre-trained models actually do very poorly on this dataset and so I'm having to train them up from scratch. That's getting there though, another week or two and I should be in the tens of thousands of images to train with and then life will get a lot easier.

Data augmentation works well and has been enabled right from the get-go.

I'll definitely do another write-up in a while about the software end of things, once it crystalizes to something a bit more stable, I'm still hacking on it daily.

I will post the training set at some point, first it needs to become large enough. But with the machine helping now that is getting there, rapidly.

> Also, these images are stitched and so are far from perfect

Maybe you could use a push broom scanner? I don't know if there's any hardware available for consumers, otherwise that seems ideal for a conveyor belt application.

This is a very rough equivalent of that. The area outside the stitched part is used to keep the belt synchronized if there are not parts in view, hence that wavy line on the belt in the video.

thoughts on tumbling/turning the parts on the belt once identified for more pictures? It wouldn't add anything to the sort speed but it'd get you way more pictures. Thinking of that machine that tumbled dice automatically. Does the addition/removal of the belt in the background come into the recognition? Otherwise in air shots from a tumble tower or belt with trip on it would get you a lot more images.

The belt movement is continuous, parts that do roll around on the belt are coming out crazily warped because in each image the belt has moved 3 to 5 mm and the part has moved the same but probably in a different direction than the one the belt is moving in. Those images are useless. This scheme would work if you could always get the parts out of a single frame on the camera but they're much too long for that (24 studs long is the maximum).

What cameras are you using? I couldn't see it very well in the writeup. Also thanks very much for this post it's very cool. You should cross post to hackaday blog.

http://aggregator.foolab.org/node/73382

> You should cross post to hackaday blog.

I post my own stuff on my own blog, I don't feel like making accounts in 20 different places. HN and Twitter are my exceptions.

I only mentioned them as they're very hardware focused.

So why micro imaging instead of macro?

I haven't done any of the nn cv stuff yet, can you pair different image sources of the same item?

> So why micro imaging instead of macro?

Because the details between some of the parts are only very minor and yet that can the difference between the part in its normal version or (for instance) its technic version.

> I haven't done any of the nn cv stuff yet, can you pair different image sources of the same item?

I took a chance on that and it seems to work. I just stack the images and present them as one larger image. Quite possibly that's 'wrong' and I should present the two images separately to two different nets which are then recombined at the end but this trick seems to work.

Huh I would have tried a 20mp dslr with a nice lens instead figuring it could pick up the differences. You can get Canon ones for about $450 on cl with lenses now.

Good to know it worked on the composite. Thanks for the info!

We have a local Lego re-seller (https://bricksandminifigs.com/) which I visit occasionally. One time I spoke to the owner about the business economics. They try to buy complete kits. It is not economical to re-create kits from parts if you sort manually. When they buy unsorted Lego collections from collectors/enthusiasts, they do a limited manual sort to pull out the most valuable pieces and then they store the leftovers. When I spoke to her, I believe she said she had 2 storage units filled with Lego that aren't economical to sort further. Run-of-the-mill bricks and pieces have no value other than scrap. The value is primarily in mini-figs and rare specialty pieces like clear acrylic canopies. Obviously, kits are valuable, but only if you don't have to create them by manually sorting. Also, used kits are much more valuable if they are sold already assembled as proof that all of the pieces are included. This assembly labor must be factored into the business model.

Buying lots off of Ebay, it seems like you risk buying presorted Lego which have already been picked over for the most valuable pieces.

Reply on news.ycombinator.com