AI-powered Lego sorting machine built with Lego bricks | Hacker News Reader

AI-powered Lego sorting machine built with Lego bricks

brothers-brick.com

94 pointsGys38 comments

First AI-powered Lego-sorting machine, eh? jacquesm@ would like to have a word with you.

http://jacquesmattheij.com/sorting-two-metric-tons-of-lego/

He mentions this in the video which is titled "World's first UNIVERSAL lego sorting machine" - emphasis mine. It seems the emphasis was shifted to AI by someone else.

And mine is just as universal, and likely also not the first but the first one that got wide visibility.

This guy did a great job of addressing some of the shortcomings of my machine, specifically in the number of output bins (18 instead of 12), but it is a regression in other ways (built out of Lego so it will break frequently, mechanical part movement from the belts, which will lead to frequent jams and will wear as well as that it is slow and more timing sensitive).

All in all a nice effort and very nice video production but not nearly as much of an improvement that would make it reproducible/mass marketable.

To properly build a machine like this will take a significant budget and a lot of hours behind cad software to make it out of industrial parts rather than Lego. I you want to make something mass marketable then you have a very long road ahead of you.

I guess someone could consider made-from-Lego to be part of the fun rather than a weakness.

It is odd to claim a first whilst also crediting you. He seemed to emphasize the universality in several places (I guess you didn't explicitly state this?).

I thought the photos of the parts bins spoke for themselves.

At some point the guy that bought Bricklink wanted to buy out the project and I told him to f-off because of the way Bricklink has been run, no surprise to see him sell it, likely he got many times over what he paid for it.

I'm happy to see someone continue with the project though, for me the fun was over as soon as the software worked and it turned into an engineering/business model affair. Bringing this to market is a lot harder than making a prototype.

> Bringing this to market is a lot harder than making a prototype.

Not realizing this for almost any product is one of the biggest mistakes technical founders make.

https://xkcd.com/678/ 5 years

I realize this doesn't add to the conversation, but I love the fact that you can mention someone on HN and they might show up for the discussion.

I believe what he means by "universal" is the ability for the machine to recognize parts it has never physically encountered before (since it's trained on 3d models).

It seems like increasing the number output bins shouldn't really matter much, since with either machine you could just reprogram it and run parts through multiple times, right?

You can train it on three D models but that's just a stand-in for the parts as though you have them. But - and this is quite important - if you want to achieve some efficiency with the 38000 or so parts that Lego has made over the years then you're going to have to cut a couple of corners and one of those is to give some more weight to the parts that you do encounter and to train your model to the point that it will come up with the right category even if it has never seen the part at all, virtually or IRL.

That's roughly what I got it to with showing it samples of wheels but not all wheels and it detected wheels that it had never seen before just fine and so on. This is a lot easier than it might seem, you'll get a good match on a class but no good hit for any of the subclasses: that's a new part right there.

And yes, that's exactly how I did it, multiple passes. But more bins is certainly useful, I just ran out of room, the machine was already getting quite large, and the belt and frame I used was fixed in size and dictated a lot of the dimensions of the machine (it was a re-purposed running trainer).

Hey, Daniel West here, the creator.

First of all, as I mention in the video, this project is inspired by the machines built by akiyuky and jacquesm@. So I'm certainly not claiming this is the world's first LEGO sorting machine!

In the video I try and focus on the unique aspect of this machine, which is the Universal part. Because the CNN is trained on synthetic data, the machine is able to recognize any LEGO part that has ever been produced. This is with the notable exception of ignoring printed parts, mold variations, and oversized parts, as well as parts that are not present in the LDraw part library. These restrictions bring the tens of thousands of part categories down to just under 3000. Accurately classifying 3000 categories is prohibitively difficult using manual labelling alone, simply because it's very hard to get physical access to all these different parts (not to mention the labelling process itself is quite laborious).

Is it overkill? Totally. But one goal of this project is to be a testbed for the use of synthetic data in a real-world classification system, and I'm quite happy with how well it ended up working.

I have produced another video explaining the AI system in detail: https://youtu.be/-UGl0ZOCgwQ

It has also been noted that the machine would not be mass marketable in its current state. That is totally accurate! This machine should be considered a proof-of-concept/prototype and I have no plans to try and create a mass-produced version.

Very neat project, congratulations on seeing it through. Note that being able to sort down to single parts is less useful than it may seem simply because of the number of passes required and that you'll likely reach your 'universal' stage a lot earlier simply because neural nets are good at classifying stuff they've never seen before as long as enough of the bits and pieces match.

How long does it run between fixes? How many kg of lego have you put through it?

Beware of lots people that will bug you to give them all your work, to this day not a week goes by that someone contacts me with a thinly veiled request to get a copy of the hard parts 'just for their personal curiosity' and then when you search for a bit on Google you'll find them associated with large parts resellers.

Highly annoying.

Best of luck with your project, curious about what you'll do next with it.

Thank you Jacques! Indeed, for practical sorting purposes going down to individual parts is less useful. But there are intriguing possibilities, like for example using it to keep track of your collection, or extracting sets from a collection.

The machine is quite robust to jams so long as you're careful not to insert oversized parts. That being said, I'd say there's a problem every 30-60 minutes, so certainly not something you could leave overnight. Running for hours and hours means that it will eventually grind itself into dust anyway (I've already burned out a motor!)

I'll probably end up sharing everything one way or another. I'm sure you'll agree that any real commercial application of something like this is pretty far-fetched so I'm more than happy to just treat this as a contribution to the body of domain randomization application research.

"And there are entirely non-LEGO machines that could do it, although what fun is that?"

That's not so much about fun as it is about long term viability, building it out of Lego will cause it to break very frequently. Typical Lego parts in complex setups like these will cause the machine to break once every couple of hours. Lego is first and foremost a toy and not an industrial component, but still, it is surprising how far you can get with it.

I built my V0 out of Lego and it worked but it was very quickly evident that it needed a more industrial approach if you wanted to drop 10 Kg of Lego in the hopper and go have lunch.

I just wanted to thank you for putting out such quality documentation of your effort and making it seem attainable. I gave it a run similar to this guy and got pretty decent results. Before I saw what you did, I didn't think I had much hope in trying this stuff out and getting any sort of positive results, but overall it was a great experience, even if it didn't end up with that "10kg of Lego in the hopper" end product.

I'm aware of several 10's of people that have re-built it from the ground up, some of them considerably better than the one posted here but most of those are just quietly making some money. Just showing that something can be done has value, it will convince others that their quest isn't hopeless.

It's a pity I don't have the time - or the funds - to really see this through but I have a pretty good understanding of what it would take to do this properly and to date nobody has done one that I would say is ready to be used by anybody but its creators. Once that does happen there will be a rush of follow-ons.

"World's first (AI-powered Lego sorting machine built with 10k Lego bricks)" not "(World's first AI-powered Lego sorting machine) built with 10k Lego bricks".

The previous HN discussions of that must also be worth a look, and I don't think there's a link to it

https://news.ycombinator.com/item?id=14226889

https://news.ycombinator.com/item?id=14280569

Is it just me that would like this machine to be used to find parts for a specific model/box. We've tried to (re)build my sons police station a number of times and it would be perfect to have a machine that goes through the box of lego parts and spits out just the parts we need.

I would love to work on something similar for sorting plastics for recycling. I wish there was an organization like Uber for organizing groups to do this in every city, along with finding funding to pay developers (> $100/developer_day). I would devote considerable effort to this full time if there are any angels out there or people who know anyone that could raise funds.

Also, does anyone know of a group working to improve the recycling symbol? It's too small/shallow to even be read by humans, much less machines, on many plastic containers. Short of that, we could break down the task into 1) isolating the symbol on the container and 2) reading the symbol maybe with extra wavelengths like infrared/ultraviolet or their combination with visible light.

Here in Boise, we have regular recycling that only accepts types 1 and 2, then they make participants sort 4, 5, 6, and 7 into an orange bag that gets sent to Nevada to be turned into diesel fuel. I see a ton of recycling in the trash because people aren't motivated to sort it manually:

EDIT: Added JPG: https://imgur.com/a/puSmAzO

PDF (old): https://www.cityofboise.org/media/1632/detailedplasticsguide...

I have thought about this a bit. It would be awesome if there was a camera next to the recycling bin that could recognize the container and give advice on what to do with the material and any rules on it. Such as cardboard can be recycled if it's clean but composted if it has any grease. Some places don't have compost so there it'd say to put it in the trash.

It would be nice just to know what to do with specific materials rather than guessing half the time.

Oh that's a good idea. I was thinking that there should be an open source object recognizer that can sort any material (kind of like a 3D printer). But having cameras on recycle bins would be great. Used cell phones could do it. There are probably unlimited amounts available for $5-10 each:

https://www.ebay.com/sch/i.html?_nkw=lot+cell+phone

Also image recognition has been a solved problem since about 2010, so any roughly 1 GHz processor can do it in under a second for something like a plastics recycling symbol.

I just can't believe that there is no money available for projects like these. I was thinking about it just now at the gym and I'd like to see society move away from donating time at a personal level, and towards organizational donating where the workers involved are paid the going rate on their labor. So instead of working a full-time job and being expected to put in a couple nights and weekends per month on volunteering, we'd instead donate money to foundations that could go deep on problems with real backing. It seems to me that this would also be a good fit with open source in general. These problems are so much easier than the yak shaving we do at work that we could solve most of them in a matter of a year or two.

Not bad! I'm curious what the accuracy is for parts. I'll look further than this short article. (edit: on follow up, not quite the level of detail I was hoping for, I'll keep an eye out)

I took a VERY similar approach for generalized universal lego sorter and got to about 120 classes of parts recognizing acceptably well for about 70 of the classes. I used Mathias' articles as a guide.

I was picking and choosing physically varied lego to find difficult pieces to sort. Beveled pieces tended to be the worst. Small pieces hid important information maybe? I suspect any solution will require mirrors to get better information.

I also got discouraged because I wanted to sort into about 2-300 different "buckets" and found the mechanical piece of that intimidating and got distracted by some other fun project. Maybe I'll return to it.

edit: I followed up by reading the longer linked article. I can't believe how similar the software the two of us wrote was. There are some labeling enhancements that I were probably the next step that I appreciate he took.

Jacques Mattheij's setup, I'm so sorry about butchering the name.

Just a though on the 200-300 buckets - perhaps you could create buckets in a grid and then just have the grid or dropper move around - once you know which bin is most appropriate for the part, the process of placing in the grid at that point seems relatively simple with just X and Y movement :)

Yeah, this type of setup is fairly reasonable but slow. I've considered droppers into banks of containers, but I haven't really gotten that far into it yet.

When he was talking about the sim-to-real problem it reminded me of a talk I went to on neural networks. The presenter's company identifies retinopathy (diabetes damages the retina). When they first were training the CNN, they only jpg images. However a clinic used slightly different equipment and uploaded pngs and it caused it to not work at all (false positives and false negatives).

Very cool project from a technical point of view, but I can't resist thinking how unnecessary this is in the real world. I remember a blog post of one of those first sorters, which ended with the guy owning a huge stack of sorted legos and no buyers.

Well done nonetheless, seems to work pretty smooth (though I see a lot of mixed materials in the buckets?).

Buyers aplenty but not enough to warrant shutting down my other company so that I can focus on Lego, there just isn't enough money in it once you start accounting for your time.

IME, the ecosystem for buying Lego is not very mature.

I tried to buy a bunch of 44 specific lego pieces (the parts were listed from a book and fairly common) through Brickseek and couldn't do it.

Brickseek has a "Easy Buy" where you can add items to your list and have it find sellers, but it didn't work at all for me. Manually trying to find each piece is not worth the effort.

It would seem then that the real business opportunity here is probably in reselling used legos on an as needed basis.

You buy or just get donated legos from people that no longer use them and then you can sell custom sets for people by running through the semi sorted bins and creating the order. So you don't want to sell the machine but rather, sell Legos. Perhaps the main issue is whether resale rights are allowed but seems solvable.

I came across a vendor on eBay just the other day that sorted Legos into full Lego sets and sold those (sans mini-figs), with a link to the PDF instructions, and a warranty that they would send any missing parts.

I would suspect this is a nice little business selling 40% discounted complete Lego sets.

It must feel empowering to be able to turn of the shelf toys for children into state of the art sorting machine. Hardware is still much harder to do than software projects. Lego lowers the barrier for many people.

just quit it with the "AI" adjective already.

Well, for once an ML label would be appropriate.

It's not real human-level AI until the machine can truly experience what it is to step on a lego with a bare foot.

Fancy meeting you here ;)

It's been a subject of some research and here is the most damning piece of evidence:

https://9gag.com/gag/aBWA6LP/this-is-what-legos-do-when-were...

For safety, an AI Lego sorter should have a whole fleet of these:

https://robots-blog.com/2018/03/21/finally-never-again-step-...

Reply on news.ycombinator.com