Translate screenshot into HTML using neural networks | Hacker News Reader

Translate screenshot into HTML using neural networks

github.com

71 pointsadamnemecek10 comments

First of all, this is a repost from literally 4 days ago [1] The article title also, is a bit clickbait - as one of the comments in the original submission mentions:

    A lot of people in this thread seem to think that this is a neural network that 
   takes an image and produces HTML, when that's not the case here at all.

[1] https://news.ycombinator.com/item?id=16115353

Does any one used it for random screenshot, or it only for boostrap things?

People are commenting on whether this replaces a developer or not, but I don't see the readme for this making any such claim. I believe it is just a static website (?) in HTML which is A LOT less ambitious than trying to have a machine create a full stack web application.

I agree. Also, most development happens with ambiguous requirements and wants that can never be translated/interpreted correctly by machine (yet?)

Also, most work is some form of addition to an existing code base that simply cannot be re-written every time.

That's why we need to have ML for project management first

Translating requirements into code is what a programmer does. Automating some parts of the software development lifecycle pipeline will probably not fully replace a developer, but decrease the workload enough to require fewer developers for the same tasks.

So my ideal IDE for webpages would be a canvas I can draw on and have elements tied to backend functionality automatically. This is definitely a step in the right direction. A little bit of react here and there and this could be a feasible beginning point for project generation.

Are there enough classy and elegant dynamic web pages that one could train a neural network on (GitHub repos) and end up with something similar?

At first the challenge should be to have any generated tag soup that matches the input but I sure hope it will eventually crunch on to have the absolute fewest number of html tags and style rules.

Also, making a design from a screenshot is rather easy as you don't have to worry about infinite view-port sizes. A series of screenies would fit the goal better.

Nevertheless: Fun stuff! looking forwards to the future of it.

I’d love to see a proof of concept of a network that generates the bootstrap front-end and connects a backend. Bonus, if the model applies attention to visualize the focus on the front-end markup as it builds the back-end.

Neural networks that accomplish intense categorization techniques rely on multiple layers at varying precisions. It's fully possible to seed a neural network with good names for elements (like a CSS file) and layouts. It's a neat idea, but dynamic pages would require more finesse than simply being able to generate a static sheet, if you wanted to use instantaneous transitions there'd have to be some sort of state-caching for transitions.

Reply on news.ycombinator.com