How computer vision is changing manufacturing in 2023
voxel51.com
voxel51.com
(Not Cognex)
edit: Oh, hey! Opteon still exists! I'm sure their products are much better than those first generations.
[0] scare quotes because I had the completely free choice between two otherwise identical degrees, one of which had a mandatory year in industry and the other did not, because the UK council tax system demands a different rate if you're a student on a course with a mandatory year in industry
[1] also a cheat, I worked for an academic research lab
My favorite machine vision use case is the tomato sorter. This is one I found from YouTube, not affiliated
I think it's funny that by you providing a major employer where you don't work nerd sniped a couple of us into guessing the other players in the space.
And here's documentation on machines which sort grapes intended for wine production:
https://www.food.fraunhofer.de/en/beispiele12/Produktschutz/...
I work in heavy industry (Steel Mill) most people that get hired around here have Engineering degrees. Mining, Oil and Gas is quite similar.
We do engage with data science people etc (Usually as consultants) but there can be a lot of friction as oftentimes the conversation gets lost in translation. If you can fluently speak 'engineer' and 'data scientist' you would make yourself a very attractive hire.
You could also look into the lab automation sector, that is taking off in a big way recently for materials and chemistry (round 2 for chemistry i guess). Chemspeed is the big name but there’s a bunch of smaller companies doing cool vision and robotics stuff in this area
Toothbrushes, drink bottles, oil bottles, more labels than you can imagine, pharmaceuticals, credit cards, fast food packaging, retail packaging, wine bottles, liquor bottles, beer cans, injection molded plastics, IBM bottles, insert molded bottles, cosmetics, raw foods, countless barcodes, and many, many more.
Honestly though, the suits i worked with, were all very dated and used hand constructed feature filters etc. to detect flaws. Usually, it was easier to adapt the environment (exclude external light etc.) instead of lengthy tuning sessions for the installer.
Usually the industrial cameras were also designed, so that local maintainers could readjust them, which excluded complex programming and happened in simple wizards or excel like programming surfaces. There was no time planned in to "retrain" further once the line was running. And it was cheap and good enough that way.
Thus the "cutting" edge tech seemed to be eternally 20 years behind the cutting edge in other sectors relying on machine vision.
Though with recent developments in machine learning, the case for PC-based solutions is a lot easier now than before. Behind all the fluff and shiny marketing, the incumbents are very stagnant.
They gave me plenty of swag, but if I wanted to play with one of their cameras I'd have to go out on the production floor.
The promise is - as you say yourself - in systems that are easily maintained by non-vision experienced engineers. As I noted in another post, these are usually controls engineers that overwhelmingly prefer ladder logic on their PLCs and have little exposure to modern software engineering practices such as source control. Obviously it's not "up in an afternoon" - any sales rep that promised that got sent away (Keyence, I'm looking at you) - debugging consists of a lot of product test runs and more mechanical/controls work and definitely takes more than a day.
I tried on more than one occasion to put forward PC-based systems, but the customers wanted the smart cameras. Though I did frequently use OpenCV for batch image analysis in-house, I ought to write an article or two about that bit.
>I tried on more than one occasion to put forward PC-based systems, but the customers wanted the smart cameras.
Oh, all those that you listed also use PC-based systems. I know because all of them are also customers of ours.
Sidenote, I feel partly responsible because we bought a ton of systems from McGarry's previous company Acumen, then I guess he took off and created the InSight. Colorful guy, I remember him showing up with his fancy Porsche around that time...
I wound up writing an in-house tool that pulls the program files from each camera on the machine LAN over FTP and commit them to a local Git repo. There was also some futzing around with XML to get the backup metadata to work seamlessly, but it's not too hard to figure out.
Now getting co-workers to use Git and not various combinations of "Copy of (1) Copy of visionproject (FINAL) 3-2-16 2a.zip" was a different challenge.
That demo of real-time blob detection and sorting by color filtering was doable in 1998. Earlier than that, even. I've found about 90% of the work in vision applications in industrial packaging is in the product handling and scene setup - focal length, lens selection, exposure time, etc. - all things familiar to a photographer. The last 10% is almost always handled by bog simple algorithms that can be more or less cobbled together from OpenCV's examples and boilerplate, the most complicated usually being OCR.
The value-add of these dedicated industrial vision systems is in integration. Fanuc's iRVision is good at sending spatial data back to the robot controller, but the interface itself is a horrid kludge that specifically requires Internet Explorer and in-person training at their own (admittedly very nice) facilities and promises of litigation if you so much as think about sharing documentation with co-workers.
Recording images during trial runs with their native tooling was impossible, as their under-powered processor couldn't handle saving 640x480 images at 10fps while also running the vision application. So we resorted to recording test runs by feeding the live view OBS, and everyone thought I was some kind of wizard for even considering that.
At least Cognex's In-Sight has the ability to simulate their weird spreadsheet-based vision programs without a camera. With Fanuc you need the whole $30,000+ robot+controller+camera setup and with real-time applications the only way to debug it is to run it in situ.
Now my most recent industrial vision experience is from 2019, so maybe some things have changed. But these are folks that often don't even know what source control is and will run screaming for the hills at the first sign of anything that's not Excel or ladder logic, and balk at the idea of paying an experienced engineer more than $100k all the while wondering why they aren't finding any talent.
I’m hugely enthusiastic hobbyist that would love to chat more about robotics, in particular how a hobbyist could get started with it (a robot arm + camera maybe?). I’d love to buy you virtual coffee, get in touch if you’re up to it!
The finances on pure software are just so much better. Better margins, better scale, better return on equity. Since ROE is always going to be better (because you're not touching atoms) you'll always have better valuation on the stock market, and be able to pay programmers better.
It's less a "gap" in the market and more "the market functioning correctly". There's no law of the universe that says programming a robot has to pay as well as programming a SAAS webapp.
Think about scale. If you teach programming at a middle school, you have maybe 100 customers at a time. If you work for a hardware company, you have 1,000,000 customers. If you work for facebook, you have 3,000,000,000 customers. Which one of these will pay the most?
If we want to not lose (or should I say restore?) the ability to manufacture stuff in the US this trend has to be broken somehow.
I did a project with Lego sorting marbles in 2009 doing this in High School. By that point, anyone with $100 and a few hours of spare time could put together a rudimentary sorting system.
Reasons why I think it shows off their best features:
- two regular industrial cameras 1.5m away
- shiny and slippery parts
- vacuum gripper (not magnetic)
- cramped picking environment
- works even when things move around (no scene caching)
- fast (video is not sped up)
https://www.valcomelton.com/industrial-products/inspection-c...
As for PC-based systems, I would be very surprised if deep learning models weren't being used in production somewhere. But in a factory environment you can go a very long way with primitive feature recognition and good control over the scene and lighting, and the customer just cares that whatever you're doing just works and any new method will have to be enough of an improvement to be worth the cost of development time.
They definitely are. ~5 years ago I built a PC-based system that detected grain direction of wooden boards (looking at the end of the board).
Initially I resisted the ML approaches and my first attempt was basically hand-crafted image analysis pipeline- split the image in segments, apply Gabor filter with kernels of various angles and try to fit a curve to results. It kind-of-worked but I wasn't entirely happy with it's performance on the test data.
Even the simple classifier models that could execute on a fanless PC without a GPU outperformed my solution, and after a few more training runs the handcrafted code was replaced by #include <tensorflow.h>.
This year I'll have to extend the system with on-site training mode, where an operator has a pushbutton to label the images and re-train the model.
I don’t recognise the truth in what you are writing.
> there are supposed to be 22 pepperonis on the pizza, but only 19 were found
Instance segmentation is a solved problem. A properly constructed and trained neural network can tell you exactly how many pepperonies it sees and exactly where. Telling if that is the right number is a trivial problem from there.
> the diameter of the pizza is out of tolerance by 8mm
Here too, the neural network can recognise the edges of the pizza and then you can fit a shape to it. You can do this second step either with classical algorithms or with a machine learning one. (I would use a classical algorithm if the pizza is meant to be circular or rectangular shaped, and a machine learning algorithm if they are aiming for something weird, like an Italy shaped pizza or something.)
> Machine learning leans towards "it's not a good pizza"
Sounds like you have only heard of simple classifier models.
I see! Thank you for the explanation. That now makes sense.
Basically you were talking about what is available on the market as a product, and I was talking about what the state of the art is in machine learning. Now obviously if you actually want to put a factory line together you care about the available products, not what is possible in theory.
It is kind of like asking someone if it is possible to travel to the moon. If you are asking a physicist they will do some calculations with the rocket equation and will tell you that it is perfectly possible. If you ask the same question from a travel agent they will tell you it is not possible because they can’t sell you a moon holiday right there and then. They are both right, just in different contexts.
> If those systems have capabilities like you describe, they have not been well presented during their sales pitches.
All I can tell to those companies is that they should “git gud”. :)
Thank you for your explanation about the context you were talking about!
- Agriculture and food processing, which cannot be offshored as easily, requires very challenging machine vision solutions. Dirty environment, unpredictable lighting, unpredictable object appearance.
- Proto and small scale high tech manufacturing, pre-offshoring or sensitive IP, requires machine vision solutions that are both sophisticated and quickly adaptable
I also wish robots would do the menial labor that I do not enjoy and would take care of all of my basic needs.
But this is an article about basic computer vision beginning to impact basic manufacturing. What you're talking about is decades in the future if ever. I'm very confused.
Edit: The OP originally talked about an agricultural robot that could charge itself, do all the home chores, and fix things around the house. Now it's just one sentence.
Computer vision in a very constrained environment is much much different and often isn't even suitable for many "simple" tasks that aren't constrained quite enough.
We can't produce an electromechanical device that is capable of the kind of fine motor control 99% of animals are capable of, let alone doing it on an industrial scale. We're not even at the "promising proof of concept" stage and what use is more advanced software when we're not even close with the hardware.
One of the ideas around cognition is that a lot of what we regard as intelligence in the physical domain, including intelligence below human levels, involves creating physical models about how the world works, which AIs are literally not able to do at all. You can instruct them in various ways, e.g. Boston Dynamics, but they have no way to internalize and actually understand novel physical world situations.
Some very smart people suspect that ML is a very powerful technique but that "better ML" only gets you so far.
I don't think this is true. I work for a US company producing industrial equipment based heavily on machine vision. Our products (along with those of our competitors) have changed the entire industry we support, for the better.
Ours is only one specific part of the manufacturing space, but I fully expect the impact to spread to other parts as well.
Being able to identify molds reaching end of life prior to parts failing QA for being out of tolerance is also huge for American manufacturing.
Where it's way less important is when you are spitting out eraser tips or other 'high scale' manufacturing.
In the U.S. the number of people employed in manufacturing is lower than ever but the value of manufacturing has been steady at 12% of GDP since World War 2 ended.