Translating a Visual Lego Manual to a Machine-Executable Plan | Hacker News Reader

Translating a Visual Lego Manual to a Machine-Executable Plan

cs.stanford.edu

75 pointsnl27 comments

Would be interesting to see a solution of the reverse process i.e. given an image or definition of a final model, create a bill of materials and (within various bounds) output step-by-step instructions to create it.

Connect this to DALL-E, and instaneously create new Lego sets with text prompts.

So that's how the world ends, a computer going full paperclip maximizer trying to build Lego sheep that are in some way "electric"

I think LEGO used to have a publicly-available CAD which could give the building steps, given a model.

Lego did have this, and I think they still do, but the building instructions it created were fairly formulaic, layer-by-layer instructions. The manuals you get with a normal Lego set are much better.

Now they just need a robot that can grab the required parts out of a box full of Legos and execute the plan. But I guess that's left for the next paper...

"Two more papers down the line..." - dr Karoly Zsolnai-Feher

Jacques Mattheij has got the first part covered (kind of): https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/

https://jacquesmattheij.com/sorting-lego-many-questions-and-... for a follow up and more links.

More importantly, a robot that can pick out required parts from a bunch of sets mixed all together. I would so kill for something like that.

And then, progressing upstream, the next version will spring into a Lego store, grab the required box(es), and sprint back out again.

This is awesome and cool!

It's hard to say how well it would translate to other kinds of manuals. I say this just knowing how exquisitely high quality the design is for lego manuals, and how much effort went into ensuring 3yr olds can accuracy understand them.

It is interesting how it is done by Lego themselves. Given the quality and accuracy of the manuals I always assumed that they are machine generated.

Lego manuals exist much longer than computer could do the task. I mean manuals from the 60s or 70s had to be like hand drawn but still look the same as today.

Sure. I never had a chance to assemble Lego from those days, but from unboxing/assembling videos [1] it seems that the manuals were either hand drawn or photos, so definitely it was a manual work.

1. https://youtu.be/xuJ2OulhC1M?t=564

I remember them offering a software (or maybe a webpage, though it would've likely been a Java of Flash applet at the time) that allowed users to create official-looking manuals for their own creations by interactively specifying the build steps.

The article is not about generating manuals. It is about reading them.

Given that this approach needs a manual written by somebody who already understands how to assemble a lego kit, this is not practically useful. The person who wrote the manual can also specify a set of machine-readable instructions (for example, in a programming language).

In short, this looks cool, but it's pointless.

Many more people can put together a manual of step-by-step pictures than can program.

Yes, but if the step-by-step procedure is known, it can be programmed manually and does not need to be machine-learned from examples. It doesn't have to be the same person who writes the manual and does the coding.

An approach like the one in the article above would be useful if it could derive a procedure we don't know how to code manually.

Couldn’t possibly disagree more; that’s akin to saying that feature identification is not useful, because if we really wanted to we could just deliver a list of identifiable features along with every photograph.

The point of this, obviously, is not to make it easier for machines to construct LEGO sets. It’s just a really clever problem space to do research into improving machine vision and image interpretation: it’s complicated enough to run into meaningful challenges, but constrained enough to make progress achievable. (The “Minecraft” dataset mentioned sounds like a simpler baby-steps version of the same, since there’s a much more limited set of possible shapes and connections to work with.)

Plus they get to be the fun lab with all the LEGOs in it

>> Couldn’t possibly disagree more; that’s akin to saying that feature identification is not useful, because if we really wanted to we could just deliver a list of identifiable features along with every photograph.

These are not arbitrary photographs but images of legos. The features needed are few and finite and there's no need for machine vision to find them.

This is pointless and has no real-world usefulness, other than to look like "the fun lab with all the legos", like you say.

Again: you somehow appear to have missed the point that this is a research project, not a SaaS proposal. It is not intended to have "real world usefulness", it's intended to advance the science of feature identification and image recognition. You know: science. You're a PhD student yourself, which makes it especially confusing that you appear to believe research has no value in its own right.

You are of course correct that these are not arbitrary photographs. Again: it's a domain containing enough complexity to offer meaningful challenges, but constrained enough to allow achievable progress.

> The features needed are few and finite and there's no need for machine vision to find them.

Finite, sure. Few, not so much; there are thousands of distinct types of LEGO block. If you're going to try to make the case that these diagrams can just be trivially parsed into the corresponding parts, orientation, and connections, while also accounting for occlusion of elements and model rotation across diagrams... well, you're probably going to have to try a lot harder to make that case than by just stating it as fact.

>> Again: you somehow appear to have missed the point that this is a research project, not a SaaS proposal. It is not intended to have "real world usefulness", it's intended to advance the science of feature identification and image recognition. You know: science. You're a PhD student yourself, which makes it especially confusing that you appear to believe research has no value in its own right.

Your comment is aggressive and a personal attack, and for this reason I'm not engaging you further.

If only the persons making invoices and receipts would have specified a set of machine readable values for the fields, a whole task in NLP would have disappeared.

Why NLP? Reading (hand-written) receipts and identifying field contents is a machine vision problem, not an NLP problem.

Anyway this is a much easier problem than either NLP or handwritten digit recognition and the solution is already known, so there's no need for machine learning. Ordinary, manual programming can do the job just as well. For a bit of perspective, industrial robots are programmed by hand and they don't build stuff with legos.

Edit: so what kind of non-lego application would you expect of a model that can put together lego assembly plans from lego assembly manuals? I'm curious.

It's a problem in the middle between CV and NLP. It used to be solved with LSTM sequence tagging models, now solved with combinations of CNN and transformer with 2d coordinate embeddings. It's far from being solved, not even receipts work 99% correct, real scores between 89..92%. Manual programming solutions are very bad, they usually do regex matching or template matching.

If you want to extract 5-10 fields from a document then you'll have at least one or more errors per page, unfortunately not "automating" too many documents out of 100 yet, with any method.

To be honest, 89-92% sounds too optimistic for a real-world application.

But I was talking about applications of the lego assembly model. My intuition is that lego is a special case: the pieces are made to interlock with each other and there are only a small number of ways in which two pieces can lock together. Compare this with building a prefab house, for example, or a car, where elements must be bound together with nails, mortar, soldering, or glue etc, in essentially unpredictable ways.

I say that the approach described is pointless because I can't see any real-world applications of extracting assembly plans from a visualisation of an already-known assembly plan. Even less so when the object assembled is just a toy.

And it's been possible to assemble toys using machine vision, given a pre-determined set of instructions, from the 1970's already:

(Donald Michie presenting the Edinburgh Research Purpose General Assembly Robot, a.k.a. "Freddy"):

https://youtu.be/03p2CADwGF8?t=1668

More info of Freddy here

https://www.aiai.ed.ac.uk/project/freddy/

Freddy was tediously hand-programmed with a full plan but it used vision and touch to find the pieces of the toy it assembles in the video.

Reply on news.ycombinator.com