A DIY M&Ms and Skittles sorting machine | Hacker News Reader

A DIY M&Ms and Skittles sorting machine

willemm.nl

581 pointsjoeguilmette121 comments

Here's the commercial version.[1] This machine is sorting peas by color. Peas. Individual peas. Each individual pea is examined by cameras for size, color, and looking like a pea. Rejects are kicked into the reject hopper by an air jet. There are machines like this for most fruit. Typical throughput is a ton per hour. Most fruit and berries go through such machines today. That's why the fruit at the supermarket is so consistent.

The process looks like magic. Color-mixed items go in, and single-color items come out, on a line going so fast that no human can see what's happening. It's amazing to see computer vision systems that fast.

These machines work by putting the items on a conveyor belt, then dropping them on a much faster conveyor to spread them out. The fast conveyor goes past cameras, and at the end, launches the items into free flight for a few inches. While in flight, computer-controlled air jets knock out the rejects.

[1] https://www.youtube.com/watch?v=JyGR6A5MWG0

A friend of mine works for a consulting company specializing in optics. They invented a machine for one client which sorts wheat grains. The machine does spectrography of each individual grain, looking for moulds and fungi as well as heavy metals and more. It's a big rotating drum with small chambers for each grain. After doing the spectrography it shoots each grain off into either the selected or rejected direction.

But it doesn't just reject every bad seed, instead it will optimize to keep within legally accepted limits.

> But it doesn't just reject every bad seed, instead it will optimize to keep within legally accepted limits.

This is just... I guess the emotion you feel when reading the last paragraph defines you as either technical guy (outrage / resignation, depending on your age and experience) or managerial guy (pure delight).

I am guessing there is a switch that makes it sort properly, you know, for VIP?

So if someone reacts with delight to that last paragraph, then they must lack technical knowledge? This seems inaccurate.

You seem to have missed the point of the remark. It isn't about technical knowledge, it is about differing priorities.

In all fairness, wasting food needlessly can have important environmental impact issues, and if the legal limits are well calibrated, getting the "allowed" dose of fungus might be a good thing from an immunological perspective. Heavy metals not so much...

Arguably, separating out grains, etc., that would be actually unsafe to eat is not really wasting food. Throwing out food that is perfectly edible is wasteful.

I know, it's horrible, right?! But also perfectly obvious.

I haven't seen the machine but I envision a panel of knobs you can turn to adjust the acceptable levels of mercury, ergot fungus, mildew... ;-)

It seems far more reasonable to me that increasing the true positive rate (bad grains rejected) will also increase the false positive rate (good grains rejected), so it's not the value of the bad grains included (which is almost certainly very small) but instead the value of the good grains which would accidentally be destroyed.

According to my friend, no; it could definitely do a "better" job to select the best grain. But obviously no one is interested in rejecting grain that could be legally sold.

Further, I'm guessing that there will always be some trace amounts of heavy metals, poisons and such in any grains so it's necessarily about setting a limit and optimizing the mix for the largest clean yield (in itself an interesting problem since the regulations are not for each grain but instead for all of the grain taken together).

There's a similar issue, IMO, with new home construction. There's a minimum legal quality limit for having a home be sellable, and anything past that basically doesn't get done. It's made worse by the home evaluation metrics - square footage, bathrooms/bedrooms, and location are most of what matters for getting a mortgage (and thus for bidding on and pricing a house).

Like, if you look closely at a 100 year old house, you'll find details like "the awnings over the window are just long enough to shade the window in the summer and short enough to get sun in the winter" that basically don't make it into modern homes.

A list of which would make for an interest design book.

You might really enjoy The Timeless Way of Building.

That's a great suggestion. I flipped through A Pattern Language a few years ago and found it fascinating.

I don't really think this has to do with legal limits; you could probably sell a hut as a "house" in most places if you wanted to, as long as it was compliant with the building codes. The problem isn't that the government won't let you call it that, the problem is that nobody will buy just a hut, they want an actual house.

While it may or may not be the case that 'house quality' has declined over the decades, that's driven by economic factors: do house buyers want to pay 1.2x or whatever for those extra details? I would conjecture that home developers are not stupid, and that they have tried adding those (presumably expensive) details, and found that they were unable to recoup those costs in an increased price.

>as long as it was compliant with the building codes

Huts aren't compliant with building codes - and regardless, there's specific criteria you have to fulfill to get various statistics (square footage, bedrooms, bathrooms, kitchen). Plus there are zoning restrictions on minimum building sizes, on top of regulations on minimum room sizes.

>do house buyers want to pay 1.2x or whatever for those extra details?

Even if they do, they often can't. If the bank appraises it as a $200k house, that's often going to mean that the buyer can only ever get a mortgage for $180k with $20k down. If the buyer wants to pay 1.2x, that means that they have to come up with $40k in cash.

>I would conjecture that home developers are not stupid, and that they have tried adding those (presumably expensive) details, and found that they were unable to recoup those costs in an increased price.

They aren't stupid, and they have tried adding details. What you wind up with are granite countertops, because that sort of thing impresses average home-buyers (and is an easy thing to point to for getting your appraisal adjusted).

The big issue is that American homebuyers, as a class, have lost the ability to distinguish good home design from bad. "Why" is a long essay, but the "just trust me on this" is that the vast majority of home-buyers have little to no experience in the sort of work that goes into building a home.

Is that a problem?

If the masses don't know or care then it's not a problem to them.

Whereas you, who do both know and care, can snap up superior quality houses at bargain prices.

Seems like a good situation to me.

> The big issue is that American homebuyers, as a class, have lost the ability to distinguish good home design from bad. "Why" is a long essay, but the "just trust me on this" is that the vast majority of home-buyers have little to no experience in the sort of work that goes into building a home.

and therefore this is not a legal issue, but an economic one. (or aesthetic, or whatever you want to call it, but not legal)

My house has a number of details that would never be put in a spec house, because only someone like me would be willing to pay for them. I expect when this house is sold, it will garner a 0% premium for those features.

For example, it has GFCI breakers for all circuits, not just the bathroom ones. For another, it has a stainless steel sill plate (which keeps wood boring insects from coming up through cracks in the foundation).

> For example, it has GFCI breakers for all circuits, not just the bathroom ones.

I did that to every house I ever lived in. It's good practice and as far as I'm concerned it really ought to be code.

Do you also have the habit of installing multiple utp runs into every room ;) ?

that seems implausible to me, for two major reasons:

1. you're suggesting that the machine has a zero percent false positive rate, which I believe given the circumstances is physically impossible. for this type of machine, there must necessarily be some increasing function relating false positive to true positive rate. perhaps doubling true positives from (making up some numbers) 0.0001% to 0.0002% only wastes 0.01% of the available grain, but either way, I refuse to believe that increasing the true positive rate is truly "free".

2. you're effectively stating that the grain processors take out most of the bad grain, then dump it back in. given that the allowable percentages of "bad material" are (as far as I know) quite low, I don't really see why they would bother reducing the amount thrown away from, say, 0.0001% to 0.00008% to save that tiny amount of money.

Speaking of legally accepted limits, a college friend told me of his first engineering job as a chemical engineer where the factory refined sugar and graded it based on impurities sold at different prices. Sometimes a batch turned out too good so his manager would insist on pouring an amount of sand into the mix to make sure it would meet the agreed upon impurity target.

The legal limits probably don't specify rejection criteria on a per-grain basis - they probably say something like "No more than x.x% trace of fungus Y in a given batch", so being able to calibrate on the fly is pretty important.

In the 1970s, the USA sold a large amount of wheat to the USSR in a famous deal.

The contract stipulated that the wheat would contain something like <1% sand. US wheat at the time had effectively no sand at all, so they mixed in 1% of pure sand, staying just within the contract terms.

(I heard this from a friend whose friend knew something about it; in searching I can’t find a source online, so the story might be apocryphal. I also might be off on the precise percentage.)

That doesn't sound like a very nice thing to do. I wonder if capitalism is to blame, as I would certainly blame capitalism, or good old fashioned egotistical rivalry that I'm told was common in many areas of life then between the USSR and the US.

I'd say that feeding humans or even livestock is more important than profits. Or at least, it ought to be.

Or classical capitalism, because 99 ton of wheat plus one ton of sand is one ton of wheat cheaper than 100 tons of wheat. Sand is heavy and cheap.

Perhaps it's 1% by volume instead of by weight.

I just happen to have bought some sand recently, and it isn't that cheap. Especially not the type of sand that you'd feel comfortable mixing into food.

Where I am, a ton of sand aggregate using in construction is ~$100 per ton. The better sand used in gardens, the brown sand (it's used to repel water so you don't over-water crops) is more than than.

Considering the wheat/ton spot price is $160 per ton at the moment, I don't think the actual prices would be too different.

[0] http://www.indexmundi.com/commodities/?commodity=wheat&month...

Unfortunately for some profit is still more important than human health or the environment [0]

"Trafigura, Vitol and BP exporting dirty diesel to Africa, says Swiss NGO

"Traders blend cheap fuel with sulphur levels many times the European limit for sale in African countries, says Public Eye"

[0] https://www.theguardian.com/world/2016/sep/15/trafigura-vito...

A good compromise would have been to put a small bag of sand in the bottom of the container.

Though we'd want that to be okay. The Russians might then reject saying you sent us 999 kgs of wheat and 1 kg of sand.

Also, diplomatically, this might be more difficult to defend. Although allowing this would've been in the best interest of both nations. Cue the human condition :)

Grain elevator contracts contain an allowed foreign material allowance. Farmer's bringing in their grain get docked for any foreign material. Then the elevators add foreign material to the grain as train cars are loaded. It's crazy but that's just how the industry has always worked.

Do you have a source for this? What kind of material do they add?

Sounds illegal - https://www.gipsa.usda.gov/fgis/grainhandling.aspx

Except -

"Recombine or add broken corn and broken kernels to whole grain of the same kind, provided, that no dockage or foreign material, including dust, has been added to the broken corn or broken kernels;"

There's also the 'story' of the Japanese firm who included 10 broken 'X's' with the delivery of X, since the contract stated 10 broken X's per 10000.

I think you'll find neither story is true.

Like most Urban Legends it's a bit silly, people are people whatever the culture. They have common sense.

Usually they include 10 Xs more for the broken ones.

In the Google SRE book they say that if a service has reliability much better than the stated SLO they artificially introduce errors to get closer to the error budget.

This is to prevent over-reliance on the measured SLO rather than the stated SLO in upstream services.

This sounds very very odd?

Why ride the line? It'd take one major issue then you're way over you error budget?

For testing/simulations I can see why you'd introduce the errors.

Netflix has Chaos Monkey. The purpose is to find unexpected flaws and risks.

One major issue and you stop artificially inserting errors, which are inserted at a rate such that you could turn off the error inserted within some timely manner and still stay within budget.

My understanding is that they don't bring the service to exactly the SLO... To prevent overreliance on a service, it can be sufficient to introduce some level of failure, which may still be well above the SLO.

http://danluu.com/google-sre-book/#chapter-4-service-level-o...

Well, kind of both. As an engineer I applaud the fact that it optimizes for maximum usage while remaining in-spec. I'd still hope they preferentially discard the really nasty stuff. A few slightly discoloured grains per kg is much better than an entire dead mouse every 100kg.

You'd be surprised how disgustingly high the legal limits are for food to be considered defective in the US, though.

http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocuments...

I have relatives in Australia who worked for the sugar industry.

Turns out that unrefined sugar is kept in gigantic open barns, and moved around with bulldozers --- not a clean process. Of course, birds and animals get in, and frequently die there, and then the sugar does... sugar things... to them.

As a result there is apparently a legal limit as to the maximum permissible number of crystallised parrots per tonne in Australian sugar.

I think it probably has to be your first gig as a technical guy to not feel resignation. I mean, "good" grains and "bad" grains? They don't come labeled. It's a noisy, messy spectrum.

I apologize in advance for my ignorance. Why are the different kinds of wheat grains mixed in the first place?

Because a field of wheat isn't uniform. The goal is to make all the wheat perfect, but nature foils that by spoiling some of the wheat with mold/fungi; other parts of the field may grow over soil that has a higher heavy metal content. As truckloads of this non-uniform wheat are processed, the machinery sorts the grains to get a mostly uniform end product, rejecting grains that fall outside of an acceptable threshold.

The fact that the air jet can target individual bad peas out of that torrent is incredible. Computationally, probably not that difficult, but the timing required must be ridiculous.

I took a tour of a company where they build machines like this last year, it's amazing. One guy was telling about a customer who bought several machines from them, then buys rejected products from big manufacturers who have less sensitive machines, and find the good product out of those rejects and sells that again. Almost literally finding needles in haystacks, and making money from it (and reducing food waste). (not sure if the guy was supposed to talk about this, so keeping it vague on what the product was)

I just fell into a serious rabbit hole watching every video I could find on this. The speed and accuracy of these things is wild, thanks for sharing it.

That's incredibly fast. The precision timing to actuate the puff of air from above is super impressive, a bit too long or too short or mistimed and a whole pile of good stuff gets rejected or bad stuff ends up in the product. Amazing video, thank you for posting that.

https://www.youtube.com/watch?v=Lz88nsWL4kw there's also similar sorting machines for larger items (tomatoes in this case) that use mechanical paddles instead of air bursts to shoot down unwanted items.

Speaking of air bursts, here's an example of the sort of valve that gets used to actuate (with precise timing constraints) the air burst:

http://www.matrixairvalves.com/sorting-manifolds.php

Long ago I visited a company that made agricultural sorting equipment. They were trying to develop a new machine for sorting ears of corn. Among the requirements, the machine had to be capable of rejecting a dead rat.

My partners and I were bidding to design the machine... we didn't get the job.

So live rats are OK? :O

Another one for grape sorting: https://www.youtube.com/watch?v=AamACDnwRts

You feed the whole grape clusters, machine remove each individual berry from stem and then rejects all the unripe/damaged berries and little debris. All in speed of many tons per hour.

All of those are incredible machines, I wish I can buy it for my winery, but even small machines costs something like mid sized house.

There is a technology similar to this for sorting cells based on biological properties and fluorescent markers called Flow Cytometry. The machines typically allow "user defined" gates and schemes through interactive software.

This one is cool also (conveyor belt that can do all sorts of fun stuff): https://www.youtube.com/watch?list=PLKfWL8IXgKBte4TfD53pLaHO...

I worked on a cranberry farm one summer where they used a cranberry grader to sort remove the fruit from the vines and separate the good from the bad. It is basically a bunch of boards at a slight angle so that any damaged or very small cranberries would bounce out, away from the good cranberries. Next the fruit rolls down a large comb like board where it is sorted by size. It is generally sold at that point and Ocean Spray would sort it by color depending on the final product. It was amazing how well such a simple and easy to repair machine worked

What a nice job this person did.

Coincidentially I'm working on something similar at the moment only with an order of complexity that is several magnitudes larger than the one on display here (39000 different shapes, several 10's of possible colors). But my contraption doesn't nearly look as nice as this one and definitely is not ready for any kind of production.

I've been working on this for the last two years or so, it has just about every bit of my skills exercised (optical, mechanical, software, electronics) and every time there is a minor breakthrough I feel like throwing a party.

Likely this piece of gear will never see the light of day in a commercial setting but it's the most fun I've had in a long long time.

Disillusioned with web programming (security really spoiled the fun I used to have making web stuff) I figured I should do something that will make programming fun again and at least on that count I have succeeded.

And on another note, I've gained a lot of respect for the visual cortex and it's preprocessing capabilities.

Just remember that the moment you stick an Ethernet port or WiFi connection on it, security becomes just as important again.

Networking this particular device is useless so not planning on that, but yes, you're 100% right. Many producers of SCADA stuff and industrial controllers that worked just fine as long as they were isolated have found this out the hard way over the last couple of years. And I don't doubt that there are many still to come, insecure protocols, world-open ports whose only protection is that probably nobody knows what sits behind that port.

> [...] and every time there is a minor breakthrough I feel like throwing a party

Yes!

It's become a deliberate strategy of mine to set the subgoal complexity in my personal projects such that I get a small "yes! it works!"-rush about once per week or so. It's what keeps me going on larger projects, even if occasionally some part takes longer to do.

If you can share: what are you working on, out of interest?

Sorting large volumes of Lego ... call me crazy :) (several metric tons worth)

Nice project idea. After you do that have a go at sorting out the Lego Mindstorms bits and pieces!

Already part of it. It's all lego pieces, not just the bricks, basically any part lego ever produced (hence the approximately 39k shapes).

The problem is anything but trivial but I'm making good headway, proof of concept took a few months and for the last year or so I've been slowly making progress with a more robust and capable version.

It's a superficially trivial idea, any toddler could do it but to have a machine that does this with any degree of reliability is fairly complex.

If anybody is ever going to try something like this I'd give you just one piece of advice: control your inputs.

Anything you have to deal with in software can eat up capacity very fast so by conditioning your inputs the software can get simpler in ways that really matter.

I thought it was Lego at first, but 39000 shapes seemed a bit much. I thought there were only 8-10k shapes?

  mysql> select count(*) from parts;
  +----------+
  | count(*) |
  +----------+
  |    38516 |
  +----------+
  1 row in set (0.00 sec)

Though I'm sure there is some overlap it's definitely not 70%.

That's really shapes and not counting colors/prints multiple times? (Not that it really matters to the scope of the problem ;))

No, if you multiply by colors it gets much worse. 100's of thousands of possibilities...

Of course not all parts exist in all colors so that helps (a bit), but it is quite an interesting problem to work on. Every assumption you make will be challenged.

Prints and stickers are counted separately but that's not a really huge number and they should be correctly identified (so the surface decoration matters as well in the classification).

Do you also plan to detect counterfeit pieces ?

That's a really hard problem. In many cases there are subtle hints (color, the writing on the studs).

Spectrography might give a hint here due to the different formulation of the plastics but some of the knock-offs are now so good it can be very hard to tell them from the real thing.

I'm not really sure if 'counterfeit' is the right term, the companies selling these are not making pieces labeled 'lego', and in fact the Lego brand started out by copying an English product.

https://en.wikipedia.org/wiki/Kiddicraft

Damaged pieces and discolored pieces are also of interest and a very hard category to detect.

I didn't know about Kiddicraft, thanks TIL.

It might not be true at the piece level, but some sets on some websites(ali, etc.) are exact replicas of lego sets, bar the brand. Down to the manual. Hence the use of "counterfeit".

Ouch, that's bad. I wasn't aware of those, and yes, I agree, that's a good use of the word counterfeit.

I was hoping it was Lego sorting! Every kid's dream machine. I got to talk with Nathan Sawaya at an event where he brought boxes of mixed Lego for us to play with. (including some of the giant Lego boulder from the Mythbusters episode) He said paying his assistants to sort bulk used Lego was more expensive than ordering new bricks en masse. (In the context of his sculpture projects at least)

> He said paying his assistants to sort bulk used Lego was more expensive than ordering new bricks en masse.

That's roughly the economics of it. There is a cottage industry where people will sort lego in bulk by hand, effectively it's probably better to flip burgers on a $/hr base.

cool! So why are you doing this? to sell them? and where did you get metric tons of lego?

> So why are you doing this?

Excellent question :) Because I'm mad ;)

> and where did you get metric tons of lego?

Auctions.

Top marks for: 1. Showing a diversity of maker skills 2. Making something that actually works 3. Industrial design savvy 4. Entertaining video

You've given me some motivation to get off of HN and work on one of my side projects for the rest of today.

Mechanically, it could have been done a bit simpler. This is an old project that has been around a long time:

https://www.youtube.com/watch?v=Ku9PKRoH1CE

Now that green skittles have changed from "lime" to "apple" (my lawsuit regarding calling the flavor assortment "original" will be filed any day now), I need a machine like this to sort the green ones into the trash where they belong.

I would counter sue the shit outta you.

Flavour-based sorting? That's Version 2.0!

Your video is even more compelling: https://www.youtube.com/watch?v=ceGlMV4sHnk

So is the write up. My favorite part:

> I started working on this machine in May ’16 and only finished it in December. It took a lot of time to design and build the machine, and I kept optimizing the parts and software after the first prototype was done. Including all prototypes and spare parts, I spent nearly €500 on this machine. Well worth it, considering everything I learned.

https://willemm.nl/mm-skittles-sorting-machine/

Thanks! Url changed to that from http://imgur.com/a/M539W.

Google cache is here:

http://webcache.googleusercontent.com/search?q=cache:https:/...

Bonus points for the How It's Made-esque robo-industrial synthpop soundtrack. Reminds me of Mr. Wizard.

Fantastic job! Functional and beautiful at the same time.

This is the kind of stuff that I love to read about in the mornings.

I know this machine will not solve world hunger or bring about world peace but I know this would bring peace to my family.

I apologize in advance but I am going to "borrow" your design and work with my kids to recreate this.

Thank you for giving me something worthwhile and productive to do with my kids today. Atleast give them something inspiring and fun to look forward to.

This is really cool! I love how it lights up with the color of the candy it's sorting each time.

I'm curious, have you tried putting both Skittles and M&M's in the same batch? I'd be interested to see if it determines that the purple Skittles and the brown M&M's are the same color, for example

This is still an awesome project, but anyone else notice the machine makes a mistake? @ 1:10, in the background, you can see a purple skittle with green ones.

(Technically it we don't see it make the mistake, but it probably had made a mistake...)

Awesome job! Any stats on its error rate?

I wish I had a link, but this reminds me of a machine that was at the OMSI Maker Faire in Portland last summer that measured the shininess of pennies and then directed them into a large board with columns (kind of like a giant connect-four board).

The machine sorted the pennies to match a greyscale image given as input, so that the final output is a penny mural ready to be encased in epoxy.

I've often idly thought about such a machine but this is much more beautiful than anything I would come up with, well done! I expect you'll be working on some statistical analysis on M&M color distribution now that you've got this part finished?

Also, what's the difference between processing M&Ms and Skittles? Is it just the expected colors?

Here's the google cache: https://webcache.googleusercontent.com/search?q=cache:nt1C4j...

The perfect machine for when you're hosting Van Halen concerts: http://www.snopes.com/music/artists/vanhalen.asp

I bet he could sell a budget version which only removes one color and sell it to every concert venue in the world.

It's brilliant! Well done for such a fantastic job designing the hopper and sort units. Plus the sound it makes while sorting is very pleasing. :)

I remember doing the same thing in my Digital Electronics class in junior high. Except it was marbles, and we used BASIC.

You could probably sell one of these into every concert theatre in the world. I'm wracking my brain and exercising my google-fu, but I can't seem to find/remember which famous musician wanted only one color of candy on their rider. Or was it everything except the green ones. Or something like that.

van halen -

apparently they don't want the green ones : http://www.snopes.com/music/artists/vanhalen.asp

silly me, forgot to check snopes. Thanks for the link!

It was just a test for the venue provider, which is a really clever way of sneaking in a kill-switch as well.

> So, when I would walk backstage, if I saw a brown M&M in that bowl ... well, line-check the entire production. Guaranteed you're going to arrive at a technical error. They didn't read the contract. Guaranteed you'd run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

You know what would be even better? Sorting the difference between M&Ms and Skittles :)

I didn't see it anywhere but are you willing to provide the 3D printing files?

Mars, Inc. could save us all a lot of trouble by pre-sorting them at the factory!

If you are into this kind of things, check out the pebble sorting artistic installation, Jller: https://vimeo.com/167126696

Am I the only one who read the headline and thought the machine was created to undo this monstrosity?

http://imgur.com/kXfGutB

A lego sorting machine by shape and color would be so great

by length: https://www.youtube.com/watch?v=nlCSlJuD65A

This one is pretty impressive. :-) https://www.youtube.com/watch?v=m7Gs6-6p7qw

Impressive indeed ! I meant something that can be used at home. Like this M&M one

Coin sorting via weight or size would also be cool.

Technically something with just different sized slots for the coins will work, and let gravity take care of the rest...

Indeed, I had these as a kid in the 80s. Coins are simply inserted in the top right and roll down before falling into a stack. Hours of fascination! https://uk.pinterest.com/pin/303781937345711623/

For our high school end of year CDT (Craft, Design and Technology) project, my best friend built a snooker ball sorter. It was awesome.

This is very cool. Well done and great write up.

I don't think I could punch that much effort into something i wasn't going to commercialize.

Anyone here found a good pill sorting machine yet? All the ones I've found just seem to be vaporware.

Very cool - thanks for sharing!

Thank goodness this problem is finally solved.

This one is so fast! Nice Job!

Very cool

Reply on news.ycombinator.com