LegoGPT: Generating Physically Stable and Buildable Lego
avalovelace1.github.io
avalovelace1.github.io
I'm far from an AI expert, but I've long felt that this is one of the most interesting ways to use AI: to generate and optimize possibilities within a set of domain-specific constraints that are programmed manually.
For example, imagine an AI that is designed to optimize traffic light patterns. You want a hard constraint that no intersection gives a combination of green lights that could cause collisions. But within that set of constraints, which you could manually specify, the AI could go wild trying whatever ideas it can come up with.
At that point, the interesting work is deciding how to design the problem space and the set of constraints. In this case it's a set of lego bricks and how they can be built (and be stable).
Well, yes, we've been doing this for several decades, many people call it metaheuristics. There is a wide array of algorithms in there. An excellent and light intro can be found here: https://cs.gmu.edu/~sean/book/metaheuristics/
> What is a Metaheuristic? A common but unfortunate name for any stochastic optimization algorithm intended to be the last resort before giving up and using random or brute-force search. Such algorithms are used for problems where you don't know how to find a good solution, but if shown a candidate solution, you can give it a grade.
That sounds like "the AI came up with a solution where cars can crash, let's give that solution a bad grade."
I was hoping for something more like "the problem is specified such that invalid solutions aren't even representable, so only acceptable solutions are considered."
How on earth would one come up with a model where "crashing cars isnt't representable"? I don't think you recognize how ill-defined and nonsensical this expectation is. Especially when you consider that a such a car may encounter a situation where a crash is unavoidable, where there's certainly room for damage control. Sliding scales ALWAYS work better for optimizations anyways, since regression is so powerful.
I think it would be fairly straightforward to enumerate, given a set of lights at an intersection, which combination of lights can be green without allowing cars to cross paths. In other words, we're ruling out combinations that are fundamentally unacceptable and would never be seen in the real world (like "all lights are green at the same time").
That gives the AI a set of acceptable combinations that can be considered. Essentially the AI is choosing an integer in the range 1-max for each intersection at each point in time.
This doesn't eliminate the possibility of car crashes if someone runs a red light. But it lets us constrain the optimization problem to the set of green light configurations that are actually feasible to deploy.
It my (roughly) work this way. For example, when you do hyperparameter tuning, you specify upper and lower bounds (so that "invalid solutions aren't even representable").
The problem is, you often have no idea what will work and what not, and e.g. your HPO algorithm might hit the bounds, suggegsting that it might make sense to extend them before the next run.
The same mechanism that underlies JSON Schema support can be applied to any sort of validation and correction, and yeah, I'd love to see more of this kind of thing!
Ask an LLM: "Say the word APPLE", but modify the code so the logits of the token for Apple/apple/APPLE is permanently set to -Inf - ie. the model cannot say that word.
The output ends up like this:
"Banana. Oh, just kidding. Banana. Oh, it's so tasty I said it wrong. Lets try again: Orange. Whoops, I meant to say grape. No I meant to say the tasty crunchy fruit known as a carrot".....
Ie. a smart model, knowing it cannot say a word, will give the next best solution - for example maybe saying "A P P L E" or maybe "I'm afraid I'm not able to do that".
However, a constrained model does not know or understand its own constraints, so keeps trying to do things which aren't allowed - and even goes back and tries to redo these things which aren't allowed, because to the model it is a mistake which needs correcting.
I'm guessing I'm not that different from the average human and I can 'feel' something physically while I'm searching for the word. I've always wondered what that was.
The real value is upstream: defining a problem space so well that the model is boxed into generating something usable.
For example, if you give a text-to-SQL bot access to the same idea (e.g., error feedback from the SQL provider), it is much more likely to succeed in generating valuable queries.
If you want to be safe do not use the word LEGO. Use Bricks or in German "Klemmbausteine".
Many people have had to deal with LEGO's lawyers and it ain't pretty.
Where there is gray area is in them not clearly stating they are unaffiliated with LEGO the company.
OTOH, they also don't seem to be looking to monetize anything, so they are at lower risk from LEGO having a plausible claim that they are hurting their sales.
But then again IANAL, and that is just how I understand the American law, and every country is different.
EU being EU, I can only imagine there's a bunch of particular rules around research that may or may not work in the authors' favor.
https://www.inta.org/fact-sheets/fair-use-of-trademarks-inte...
https://en.wikipedia.org/wiki/Hilary_Page#Kiddicraft
LEGO is based off of earlier designs of interlocking bricks, they are well known because they got really good at affordability, high tolerances, and durability not because they invented the concept. Beyond that the original functional patents have long expired.
1: Never, ever, sell modified Lego bricks: https://www.brickfanatics.com/lego-wins-court-case-against-c...
Also, they don’t tend to go after fan-made things like this, based on some googling they typically throw the book at counterfeit producers who are eating into their profits.
That's where Nintendo is fundamentaly different.
Also don’t forget that Sega was “originally an importer of coin-operated arcade games to Japan and manufacturer of slot machines and jukeboxes”
Sega was mostly into normal arcade games, and Sammy baught them for their expertise to improve Sammy's much more profitable gambling machines. It's Sammy's CEO that took the lead, and Sonic and console games became a mere side business.
They just won the market because historically they reused existing locking bricks concept from a company called Kiddicraft, found a way to make it more lockable... and patent it before the original company and other companies could implement it.
We can say that they became famous half fir engineering reason, and half from their legal department...
In the casinos?
It feels like a hand-crafted algorithm would get a much better result.
(Totally feasible with today's technology, but you'll need to train your own specialized models.)
annoying that this is the default behaviour on iOS though
I just wish scientists would start by solving problems that actually exist in the real world. There’s real value — and real money — in that.
World’s first AI-powered LEGO sorting machine built with 10,000 LEGO bricks https://www.brothers-brick.com/2019/12/06/this-fan-created-t... (2019) https://news.ycombinator.com/item?id=21741834
If anyone else was searching for the dataset, it is at https://huggingface.co/datasets/AvaLovelace/StableText2Lego
It contains " contains 47,000+ different LEGO structures, covering 28,000+ unique 3D objects from 21 common object categories of the ShapeNetCore dataset".
Local inference instructions are over at their github page - https://github.com/AvaLovelace1/LegoGPT/?tab=readme-ov-file
https://youtu.be/Ca-SoKzjh4M?t=110
SMT component placement isn't that different to placing bricks. Conventional wisdom is that if you can design a PCB that requires no manual work, its assembly cost is more-or-less location independent. SMT pick and place can hit speeds of 200,000 components per hour [1]. That's about 50 components per second.
Skynet isn't goanna attack you with Terminators wielding a "phased plasma rifle in the 40W range", but will be auto-rejecting your job application, your health insurance claims, your credit score and brain washing your relatives on social media.
There’s a difference though. The “cool” Terminator Skynet pursues its own goals, and wasn’t programmed by humans to kill. The “boring” insurance-rejecting Skynet is explicitly programmed to reject insurance claims by other humans, unfortunately.
So still, no need to worry about our AI overlords, worry about people running the AI systems.
I don't see how you could possibly think this is true. Physical automation is easier to scale since you only need to solve a single problem instance and then just keep applying it on a bigger scale.
Robots work for highly predictable high speed tasks where dexterity is not an issue, like PCB pick and place.
Using bricks other than 2x2 and 2x4 blocks creatively to make interesting things is really important, i’m not sure what type if algorithm would best auto generate beautiful MOCs however? Was thinking of doing a $50000 kaggle comp for this, what do others think?
Keep in mind that these sites are run by AI researchers, not dedicated UX teams at major tech companies—so the interface can feel a bit rough around the edges. That said, your critique is still valid; it’s just fair to cut them a little slack given their priorities.
But it also shows the weirdness of the solution - in places where larger bricks make sense, multiple smaller bricks are used instead. In a section where a 2x6 should be repeated, in on instance of the repetition it uses tow 1x6s. It’s weird.
Cool idea.
It kind of makes me suspicious of the integrity of Carnegie Mellon if they will allow trademark infringement of this type because, well, it does make me feel like I can shit in a bag and call it a Carnegie Mellon Socking Stuffer without consequence.
However, the model "A high-backed chair" has some floating pieces in the middle of the seat, that are fastened from above. Can these robots handle building these?
Can it produce an ample bossom made of lego? And indecent protrusion? Weapons?
I guess I learned a word today...
Sometimes the amount of money and energy that are spent in "recreation" projects just amazes me.
This is interesting and seemingly quite applicable base research and we move forward by being curious.