AI for generative design: Plain text to 3D Designs | Hacker News Reader

AI for generative design: Plain text to 3D Designs

blog.insightdatascience.com

239 pointstf_1248 comments

I was literally thinking about this in bed before falling asleep last night, but I was imagining scenes rather than objects.

I'm in no way a ML developer (I'm not even a developer), but I was wondering how hard it would be to generate an apartment / house layout from a (somewhat detailed) description.

I'm sure the first use that springs to mind is for architects and real estate developers / agents / brokers, but one of my pet-projects-I'll-never-really-do is to recreate memories in 3D from descriptions and old photos. Imagine being able to relive your childhood memories! Just throwing it out there in case one of you smart folks want to take that and run with it...

Somebody will eventually build it in Grasshopper/Wallacei.

Wallacei X Cybertruck: https://youtu.be/bLZf-MNRoyg

That exists! The graphics lab at my old university published a paper on this exact thing: https://manyili12345.github.io/Publication/2018/T2S/index.ht...

Thanks! That is really cool and I appreciate you sharing it!

FYI I tried clicking on "Slides" but got a 404 error, so thought I'd flag in case you know the authors.

Tea, Earl Grey, Hot!

I really want to try the live demo and make some melty furniture of my own, but I got this error:

    ModuleNotFoundError: No module named 'plotly'

Thanks! It got overloaded with the traffic from my post but I restarted it and it's back online now so feel free to give it a try!

ModuleNotFoundError: No module named 'plotly' Still getting this error

Honestly surprised I had to scroll this far to find a Star Trek reference.

The scene where the crew describes their mass dream into a horror dentist chair is forever burned into my brain as a technology I want in my lifetime.

I loved that scene! It was actually a direct inspiration for my project here. For anyone looking it was Star Trek TNG S06E05 (Schisms)

Really cool idea, I wonder why nobody tried this earlier.

However, the main problem I have with this approach is the voxels. They model the geometry and only the geometry. The far more important aspect of topology is left out. Thus, the results suffer the same problems like 3D scanning / photogrammetry does: It is practically unusable as it can not even be textured, yet alone animated or used in fabrication (except for Lego-models I guess). So point 5 of the future work is the biggest one in my opinion.

Well they have at least for certain parts. I'd guess based on what they use, the whole workflow here was really only possible in the last couple of years.

I'll update this comment another time but back in 2016 my company needed something like this and there was already research using GANs to generate objects from basic parameter inputs. The text portion wasn't there.

I believe that research came out of Stanford actually.

Voxels are directly analogous to 2d pixels, which means they benefit from the myriad of CNN based techniques developed over the last few years (GAN in particular)

3d manifolds are sparse, and more analogous to 2d vector graphics. There are approaches for dealing with this type of data (eg. spectral graph NN) but they don't work as well for 3d topology as CNNs do for dense pixel data, as far as I know.

In the near term, it might be better to explore approaches that use voxels, then generate the topology heuristically.

Electronic Arts used 3D scanning / photogrammetry in the new Star Wars Battlefront II.

You're right - the topology is a huge problem - but we're starting to see specialized tools that can take minimal input from the artist and automate the topology creation workflow.

Here's the GDC talk where the E.A devs go over how they converted the scans to game usable 3D models. https://youtu.be/U_WaqCBp9zo

Nobody tried what, generative design? https://www.autodesk.com/solutions/generative-design/manufac...

Or did you mean the way the model is specified?

Ya using voxels was definitely a big drawback to my approach here but was more intended to just see what's possible.

I think a better approach would be something similar to the StructureNet paper I mention in the post and use graph based models to actually attempt to capture the topology. But they did it with super explicitly defined part trees as training data the hard part would be finding a way to do that in a unsupervised manner so you could actually make use of the massive amount of unlabelled 3D models available.

There were a lot of evolutionary design algorithms in the 1990's with similar results. This is fun - but I don't think it is going to go anywhere.

yeah, since all they wouls show were rather cruddy voxel shapes and didn't start the text with "It actually includes the physics from the start" I didn't bother read the whole thing.

But it very much looks like a very advanced way to produce something slightly less useful than a sketch on a napkin, since the latter was made with an actual understanding of what the description is supposed to mean.

Very short sighted. The underlying technology is fundamentally more powerful. Also the whole tool ecosystem has improved, so going from voxels -> surfaces is better.

Amazing! I do work in 3D computer vision and I've really wanted to get into 3D generative asset design as a side project at least. If the author or anyone else is interested in collaborating, I'd be down to talk and shop something.

Sure! I've been wanting to incorporate this exact thing for a "And let there be light simulator!"

Essentially, people in VR would be able to point and speak to create objects in a shared space.

Cool! I've got some Unity VR experience, though i don't know much about incorporating ML pipelines into it. I'm guessing there's resources out there? So basically you'd want speech -> text -> object?

Exactly!

I've got some of this working in Python, outside of Unity. Last time I looked at integrating with Unity there seemed to be some Google API requirements that were hard/annoying to meet in my few hours of digging. Should take another look. All the VR HMDs have mics.

This would be really incredible! Could maybe be a way to quickly make video game assets for testing and such. Although to make the meshes performant in a game engine you'd need to decimate them quite a bit as the marching cubes I used here produces really dense meshes.

Do you think the resulting meshes are amenable to auto-decimation tools? Also (directed to the previous poster), while VR applications sound cool, i must admit my interest really lies in improving the asset generation tools themselves. The work done in the original post is super inspiring, I'd love to see a more robust asset generation pipeline built up from tools like that.

Oh wow, count me in. You could use Google Cloud Speech to go from voice -> text, and then Google Cloud Natural Language to go from text -> simple command interface.

I've got both working in Python, but don't do anything with the parsed text.

I want to do things like "create sphere" "move that up" "no, bigger"

etc.

Great work! I wonder if using SDF as a 3d model representation would result in higher resolution model generation (rather than voxels), some recent work in that area here: https://github.com/marian42/shapegan Adversarial Generation of Continuous Implicit Shape Representations : https://arxiv.org/abs/2002.00349

I hadn't seen that paper, looks really interesting! It definitely would improve the resolution but I think fundamentally it still runs into the issues I had with voxels where it mostly just learns low level correlations and not high level topology.

Very cool. But, seems like a lot of work to order to conclude that tables are rectangular flat surfaces and cups have handles and hold liquids.

Here's the github repo for this interesting work: https://github.com/starstorms9/shape

(I couldn't find it in the linked article, might have missed it)

This is fantastic. Much better than my idea of just using text recognition to search a database of models. I hope this research continues so in the years to come I can slap on a VR headset and generate scenes like the holodeck.

It would be interesting to train the 3D Designer on more classes. Then have the text generated from something like AI Dungeon 2 feed into it and see what 3D designs it generates.

I've been thinking along similar lines on high level - this approach will likely be very general and can be used to create different kinds of media, as well as potentially behaviours in a later stage.

https://metapresent.org/creation-engine

Will be interesting to base it on a decentralised open platform that could be "built-in" in the Internet.

    Importing libraries...
    ModuleNotFoundError: No module named 'plotly'
    Traceback:
    File "/usr/local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 314, in _run_script
        exec(code, module.__dict__)
    File "/app/shape/streamlit_app.py", line 15, in <module>
        import plotly

Is anyone else getting ModuleNotFoundError: No module named 'plotly'?

Yes

Fun. Here are some sentence that come to mind.

Flat surface with bevelled edge and 4 Ionic columns as legs that fade into nymphs at the base.

Table with caryatid legs.

Crescent wrench that doubles as a corkscrew.

Stapleremover.

This work is inspiring and ground breaking. I can see a lot of text description => X solutions coming up in the future. Absolutely awesome work!

This will be extremely useful in product search.

"sofa with 3 cushions and a round arm" should generate a model and match with similar looking products

Design is not just about how a thing looks or what kind of shape it has, but the intelligence that goes into how it works.

I thinked about it so many times... Really coil. I hope someday will be possible convert a sentence to a full scene

That's possible even today, kind of. [1] [2]

[1] https://t2i.cvalenzuelab.com

[2] https://towardsdatascience.com/text-to-image-a3b201b003ae

Can’t wait till this becomes applicable to something like AIDungeon!

Stunning work, congratulations. That was a pleasure to read.

The ultimate tool for story-telling.

Woot for Insight Data Science!

This is amazing.

Nifty!

Reply on news.ycombinator.com