Hacker News Reader

Comment by mattlondon | Hacker News Reader

Finally, my interest in LLMs is piqued!

Seems like everyone has been getting excited around the search or code-generation use cases ... or simply trying to make it say naughty things (boring, not interested, wake up in a few more years), but this is eye opening.

The idea of this as a "universal coupler" is fascinating, and I think I agree with the author that we are probably standing at an early-90s-web moment with LLMs as a function call (the technology is kinda-there and mostly-works, and people are trying out a lot of ideas ... some work, some don't).

My mind is racing. Thanks for the epiphany moment.

View on HN

We are months away from being able to do this with images too.

All the pieces are there, and multi-modal (smallish) large language + image models are already being used in research labs; eg MS Kosmos-1[1]. Check out the visual IQ test results in the paper.

Kosmos-1 is only 1.6B parameters. When that or similar models scale out to 50B_ params they will be pretty amazing.

[1] https://arxiv.org/abs/2302.14045

Linked article / brief mentions interleaving of text and images as essential for certain kinds of learning; that particular intersection reminds me of the profoundly compelling multimodal book by Nick Sousanis, "Unflattening". Highest possible recommendation.

https://en.m.wikipedia.org/wiki/Unflattening

I had that book but lost it in LA. This is the page I remembered the most and have shown to the most people:

https://twitter.com/nsousanis/status/245176914900299776?lang...

That is the truly mind-blowing thing that's coming.

I am currently creating reference images for a game which I expect within 6-12 months I'll be able to feed into a multimodal ChatGPT to create 3D assets out of the 2D pics.

We'll be able to conjure up worlds at a whim - so start imagining them already!

Mark gets a lot of flack for metaverse, but can imagine world design where you start in a blank room and describe what you want around you, and it appears. Like the Matrix loading white room. And with voice recognition and eye tracking (and brain scans), how close are we to “you have to use your hands? It’s like a baby’s toy.”

This was always the coolest capability of Star Trek's holodecks, not the projector technology. Super cool that we might just get there within the lifetime of someone who watched the show in the 80s.

My favorite Star Trek episode has always been "Identity Crisis". Not because it's one of the good ones, it's pretty clunky. But it contains a fantastic 5-7 minute long montage featuring Geordi La Forge interacting with computers (by touch, voice and on the holodeck) to solve a murder mystery, analyzing and live-manipulating 3D "holo footage" to discover a vital clue. Whoever imagined that sequence is the hero of my childhood and perhaps the reason I became a software engineer, doing an oddball mix of HMI and systems engineering.

There's so much in that sequence. The free mixing of different input modes, the complementary collaboration between a human and an AI system, carrying state with you from room to room. Analyzing and generating. Following instructions and making suggestions. Powerful inference, precision of control.

We're getting close now!

They are calling it “text-to-space”: https://twitter.com/rowancheung/status/1631023065722245122?s...

Thought-to-space?

YES!!! That's exactly what's coming. I'm getting a head start by envisioning a rich world and creating pictorial references for it along with text descriptions. But it's a tiny head start: once the "holodeck" technology comes along, we'll all be able to create anything with just natural language and correcting anything the model got wrong. ITERATIVE CHISELING YO!!!

By the way, I am still using SD 1.5 inpainting.ckpt - thank you Runway for releasing it, it's perfect for my needs and abilities. I never even tried SD 2 and later ones - heard they worked completely differently and I'm too busy creating to be relearning.

Bit of a tangent but I also predict an explosive demand for parametric AI-first design and simulation software.

OpenSCAD works right now in the sense that the produced source is valid, but asking for even something well documented e.g. an AR-15 lower receiver does not impress.

My previous boss (we were in the civil construction industry) has been saying for years that we are on the cusp of an industry-shattering breakthrough with generative engineering.

We should be able to, by now, input terrain maps, account for local water ways & rainfall, and Autodesk's Brobdingnagian suite of tools should be able to spit out a whole new suburban package, engineered to local country/city engineering standards.

The main thing that's holding is back is the complexity of engineering standards, and its implementation in the software. But all that'll take is time & money.

I'm about to start work in this space. Some companies are getting close to full building automation, Augmenta seems to me to be closest / have the best approach. [0]

Although the complexity of building standards is a challenge, the speed and quality of various generative algorithms are still major limitations.

[0] https://www.augmenta.ai/

Thanks for linking augmenta. I was wondering if you know of other similar companies? I'm an experienced structural engineer who's just completed an MSc in Comp.Sci. I want to combine the two disciplines and so far I've only been looking for consultants who utilise parametric design (Rhino + Grasshopper).

No problem!

I don’t know of any other multidiscipline generative design companies, but for process plant design there is OptiPlant and PipeStream. There are a lot in the architecture space but these are mostly for high level conceptual design.

In my experience many structural engineers code and use simple parametric models. Finding a company using actual generative design algorithms is still rare but there are definitely engineering teams out there who value comp.sci skills.

I did not realize it could do that. I asked it to generate a cube with a hole through the center and it really struggled to succeed. Interesting concept though!

ChatGPT has a pretty good handle on G-code (the machine code to drive CNC and 3D printers). A machinist friend of mine and myself got it to write G-code to create a shape from a block of material, adding holes and arrises and so on. To top it off it wrote some decent poetry about 2 mates discussing G-code over a few beers.

Fusion 360 has a generative AI design extension already. Never used it, I’m too much a poor to use something like that.

There is a free hobby plan for Fusion 360 that I use for my personal projects.

I thought it doesn’t have the generative AI extension? When I looked at the plan comparison it was a part of the commercial.

This is where I see the Semantic Web, semantic interoperability, and ontology alignment spaces coming to life. Who needs to standardize data when you can stick an AI in the middle of two services to do the translation for you? And then suddenly the interoperability of the metaverse doesn't seem so insurmountable. I made a post last summer that speculated on this, but I didn't think this future was so near. Or even possible really.

https://en.wikipedia.org/wiki/Semantic_Web

https://en.wikipedia.org/wiki/Semantic_interoperability

https://en.wikipedia.org/wiki/Ontology_alignment

https://metaverse-research-590.web.app/posts/communication/

I remember having that epiphany in 2017, reading the original google transformer press release. I remember thinking “they taught computers to understand!, and someday soon i’ll be able to translate anything into anything.” And not like translating “languages” but translating contexts and concepts.

LLMs really shine when you try and combine two previously untouched things or map one bit of knowledge it a common pattern. Eg “summarize Mary Shelly’s Frankenstein as if it was an episode synopsis of Magic School Bus.” “Now do the same for Dracula.”

I remember my mind racing so much that day. With all the ideas of things it would unlock. And I still managed to under imagine just how many different types of knowledge, and different layers of knowledge and language, they would be able to encode into one compact dataset.

LLMs will be used to discover "not so obvious" connections in science. I think we will see first scientific discoveries done by LLMs very soon (in fact, such works are probably already happening). Expect major breakthroughs in physics, math, cosmology, biology, etc. published in Nature and authored by LLMs.

Cranking that hype level to the next stage eh?

It’s awesome for formatting and structuring.

Copy and paste a bunch of styled JS components -> get back out a single CSS sheet

Paste in a markdown document -> get out the same thing in HTML

Fun stuff like that.

Ah yes, you now too can have the wonders of pandoc - now on sale! Retail pandoc price of 20MB, now selling for only 3TB with the exclusive offer of ChatGPT!

I don't want to be mean, but this seems like the famous Dropbox/rsync comment.

The value here is how easy it is, and the fact that a generalised model can take the place of a (well-engineered) specialised tool.

Reply on news.ycombinator.com