Using GPT-3 to generate user interfaces | Hacker News Reader

Using GPT-3 to generate user interfaces

twitter.com

87 pointsSamin10019 comments

As someone working in the field I have a few thoughts:

1. The way I think about these huge models is no longer as something that makes predictions, but rather as a kind of huge knowledge base (compressed training data as representations) with smart query capabilities, which come from the fact that the model is constrained to create syntactically correct output. This could be the next generation of search. There is a fine line between search and creating novel outputs, since most novelty is just a combination of old things with constraints on correct syntax.

2. The shown results are likely a result of massive cherry-picking in the typical "pitch deck demo" fashion. You'll see a lot more failures when trying to use this and you need to get the query "just right" for it to produce good output, which a skill in itself. Also let's not forget that the model must be fed fine-tuning examples, which also need to be "just right.". That's why I believe these kinds of models aren't that useful for fully automating things - for something to be used in production it should be 100% correct - but they are very useful as a search function and alternative to e.g. StackOverflow in this case. Querying the model will give you good enough results that you can use as a starting point for your own use case.

Some relatively unstructured thoughts: I'm not sure I agree with 2 for this use case: writing the GDP table is a pain because table markup is (necessarily) verbose, and it's fiddly to copy and paste all of those fields individually. If I can get 95% of the way there with a couple of syntax errors that are picked up/fixed by my linter then I'm a happy customer.

That said the way he generates a table is in no way idiomatic react, where you take an array of objects and map the data on to the table.

In terms of code generation, is it actually any better than using an emmet expression? `table>tr*10>td` (I've probably mangled that). What's interesting with the table example is probably the combination of "knowledge" and knowing enough about a DSL to generate a coherent output.

I agree re cherry picking, though the initial fine tuning set is extraordinarily small, just 2 items. Generating a much larger dataset shouldn't actually be that tricky, though perhaps getting agreement on how to actually describe a layout in plain English might be more of a challenge. Would a plethora of descriptions for the same layout be a bad thing?

Lastly, I've been using TabNine (https://www.tabnine.com/) for the past few months, and I'm convinced that some kind of language model text generation is going to change how we write code, it's already changed it for me. If I signal my intent pretty clearly up front, I would say most of my coding is now tabbing and occasionally pressing the down arrow. Even writing comments becomes an exercise writing an initial three of four words and tabbing out the rest. Ditto tests.

Your first point is actually covered in the computerphile video on GPT-3. They discuss a paper that studies the ability of GPT-3 to do arithmetic as it's part of the learned text database

Check a look https://www.youtube.com/watch?v=_8yVOC4ciXc

That's interesting. Are there examples of these techniques being used for searchy things?

Well, I think it comes down to how you define search. People are amazed and how good GPT-3 is at generating text, but looking at it from a search lens, you could argue that rather than "generating" it searches for the most relevant pieces to the input, combines them, and then enforces correct grammatical rules on the output. A recent example I saw on Twitter was someone asking GPT-3 questions about human evolution, and the model answers with a bunch of book references and links - that's something you would have done in Google before.

The fact that the Transformer attention mechanisms works based on matching queries and keys/values is probably why the model is so good at finding relevant information to your input - that's kind of what it was trained to do.

This is a really cool demo, and congrats to the creator for making something really cool.

However, while these systems based on GPT-3, and similar techniques, can produce some amazing outputs given the right inputs, you don't have to poke them very hard before they start falling apart.

At the end of the day, this isn't really any more "clever" than a very, very advanced and long markov chain, there isn't any intelligence.

> At the end of the day, this isn't really any more "clever" than a very, very advanced and long markov chain, there isn't any intelligence.

https://en.wikipedia.org/wiki/AI_effect - at least sort of

I get what you're saying, but is this really as fragile as you claim? I've been rather impressed by (GPT) input handling.

I would never have thought this type of thing was clever. I don't want to pull expertise but I've been an AI research for 20 years, thinking this stuff isn't clever isn't new for me :)

If anything, this is "less clever" than older techniques like tree-search, as it is very easy to confuse, and will start just producing nonsense.

That's true, but who said that the goal is intelligence? This is clearly a demo from a startup that sells this kind of service [1]. All that's needed is something that can be made to work under certain business-dictated constraints. Nobody really needs these things to be actually smart.

Or, seen another way, it just has to be smarter than most peoples' web frameworks, or perhaps their IDEs, which is not a very high bar.

_________________

[1] Going by Sharif Shameem's Twitter profile (or bio, whatsitcalled):

founder at https://debuild.co – a faster way to develop web apps // @ycombinator alum // prev built @VectordashGPU // relentlessly optimistic

I can see GPT-3 being used in a tool/add-on ala Intellisense or Tabnine, but as something to create "a way for anyone to ship a functional web app just by describing what it should do in natural language" (quoted from his website) I'm positive that we have to wait for at least a GPT version that doesn't get confused after ~200-300 words.

I guess you could create some kind of UI prototyping app, but better options for that already exist, which don't require you to write everything out.

I often wonder how much (percentage-wise) the model is 'encoding' certain knowledge, the equivalent being how much does a person know about web development etc. A lot of research are using transfer learning in this fashion, which assumes most of or at least a large chunk of trained weights are relevant to a different task they are attempting (in contrast to the model was originally trained for). So if coding these UI requires very little language skill then you are really using a very small part of the model (or am I too naive)?

What I find most interesting about this is that is has just as much trouble as humans in translating requirements into code. Admittedly it makes some fairly basic mistakes as well, but always in the under specified areas

i.e.:

You didn't say what colour 'welcome to my news letter' should be, so I picked white. The numbers from 1-5 are totally a possible random permutation in the 1-10 range. I find the crying laughing emoji to be really ugly, so it's the worst one.

This is so cool. I want to feed recipes into GPT-3 to generate novel (and humorous) food items.

My actual worry about this is 'will I be obsolete as a front-end engineer'. It seems that AI will manage web optimization, and NLP conversational programming, so basically there is no need for a front end engineer. Perhaps I am wrong.

How was this made?

The GPT-3 model hasn't been published, and training it yourself would probably cost millions of dollars...

OpenAI is offering an API for it, currently in private beta.

This is so cool – looking forward to using GPT-3 on random things like song lyrics and internal company slacks to see what happens

A blue color Login Button and a green color sign up button

I was one of the people who didn't think that transformers would be powerful enough to automate software engineers out of a job.

I was wrong. It's coming for front-end engineers and it's coming soon.

Reply on news.ycombinator.com