Show HN: Generate User Interface Components with GPT-3 | Hacker News Reader

Show HN: Generate User Interface Components with GPT-3

designhat.ai

24 pointsdesignhat13 comments

I’m building an experimental tool that can generate a UI component from a high-level text description, i.e. “Design a card for viewing goals in a goal tracking application”.

Basically, stable diffusion but for user interface components instead of images.

This tool is at best a proof of concept. It currently only generates a single type of component (small cards that have basic info about an object), but the goal is to be able to design entire applications (either in one fell swoop or by piecing together components that the tool generates).

The components it generates can be exported as react code. You can also have it modify an existing design. The tool isn't useful yet but it's somewhat fun to play around with.

Having it generate a component rather than an entire screen made the problem easier but I'm not sure if there are any users that this would appeal to. I could see it appealing to developers who lack design and CSS skills and want to quickly build a decent looking frontend for their app.

Looking forward to hearing what you guys think of this project and the generative AI for UI space in general. If you're interested in collaborating on a startup or open-source project in this space, email designhat dot ai at gmail dot com

Open articleView on HN

Really interesting! I like where this is going even though i'm not sure the particular use case is that useful. Generating content for list views is marginally useful; being able to generate UI components for full web/mobile app experiences is a must have.

Suppose I'm a founder/PM who wants to design a crypto app with fiat on-ramp/off-ramp. Before the app is built, initial thought must be given to what's important to the user. Once established, a prioritization of UI elements is made and designer implements them. Now imagine if I write this as a GPT prompt and get back 20 different designs, choosing between 30+ UI elements. Almost like a Pinterest board for UX designs from a single prompt. This visual experience inspires me to tradeoff decisions quicker and thereby ship faster.

This is a great idea, but I noticed all the components I tried ended up looking very similar: a wide rectangle with (sometimes) an image on the left, and two sets of text captions, one pair left-aligned and one right-aligned.

My custom prompt,

> files in a file system

gave something almost identical to the predefined prompts:

> comedian shows in a comedy venue's booking application (which has an image too) > zany brainy stuff (has a centered label too) etc.

Is it more that it's not a UI component, but specifically only list items or cards, that it can generate? I did notice that it got the order of priority (eg filename first, size secondary) correct.

Thanks for the feedback! Yep it's only trained on examples of list items and cards, most of which follow the layout you described.

It does occasionally do different layouts, here's an example: https://designhat.ai/generations/7396ec47-ad9d-4f1e-90ce-ce9...

In the future I want to make it work for more types of components than just list items and increase the set of primitive elements it has to work with. Currently it's just arranging badges, labels, icons and images.

I love the idea but it is not working for me : even the example prompts are saying they cannot generate any design.

So how does it work? SD is not gpt right? So does it use sd or gpt or some clever mix, like gpt3 writing prompts for dall-e to make sure they are designs for web components?

Sorry, looks like a memory leak built up overnight. Should be working now.

Glad you like the idea! It's a combination of GPT-3 and traditional layout optimization similar to this paper: http://www.dgp.toronto.edu/~donovan/layout/

GPT-3 is prompted to output JSON that describes which elements should be in the design (label, icon, image, badge, etc), their copy and which elements should be grouped together. Then the layout optimization module determines how those elements should be arranged spatially.

I eventually want to experiment with fine-tuning stable diffusion to handle the layout part of the problem, by outputting colored rectangles that represent the layout, like in this paper http://interactionmining.org/rico

Very interesting project. i was wondering about the cost margins of running this. and how does the workflow from gpt3 react code update to ui

Thanks! It's using Davinci and maxing out the token limit so it's about 8 cents per generation.

Not sure if I understand your second question. GPT-3 is outputting the props of the component, the primitive elements in the component (images, icons, text, badges), the "values" (i.e. which copy should this text field have) the primitive elements should have in terms of the props, and some sample data to set the props to.

A separate layout optimization is determining how to arrange the primitive elements.

Given all that it's possible to generate code for a react component that you can plug whatever values you want into for the props.

Thanks, I have sent you an email. Let me know if you got it!

I think its broken. Just throwing up a js prompt

Yep the backend went down due to traffic. Should be fixed now!

This is cool! Nice work. How about generating figma as well?

Thanks! Generating React code seemed harder than Figma so I decided to do that first. But will most likely have Figma in the next release.

I'm not sure whether the best initial user for this kind of tool is designers or non-designers. The two startups in this space I know of (https://diagram.com/ and https://www.usegalileo.ai/) seem to think the former.

this is a very interesting concept for UX design

Reply on news.ycombinator.com