Comment by intended | Hacker News Reader

Comment by intended | Hacker News Reader

Earlier in the year I wanted to understand LLMs and GenAI so I tried to push their limits to see what broke and what didn't. One of my projects was to build a blog/site from scratch. I had 0 web dev/design background.

My experience was starkly less optimistic, and am curious if any one else has tried something similar.

-----

First off, I must state a deep respect to people who build + design websites, while dealing with Clients.

I had assumed that ChatGPT would be very useful in helping me pick up and build things. However, I had to jettison ChatGPT fairly soon. I just couldn't trust the output of the model. It would suggest things that wouldnt work, then link to sites that didnt exist.

I switched to teaching myself. I had to watch hours of videos, learn CSS, Astro, and several other things from scratch. Definitely not the LLM experience I was expecting.

Code from Figma was great - but if I wanted an actual responsive site, I had to write the CSS myself, because boilerplate CSS had all sorts of odds and ends.

Getting an image to come out as I liked from Midjourney was fun - but it was also a massive time sink.

I had hoped to be able to get complex tasks done entirely with assistance from the LLM. In the end it helped maybe 20-30%. Its greatest use was to clarify concepts instead of me having to wade through specification docs.

When I went back to the videos of people using chatgpt to build a website in under 30 minutes - its always someone who knows the domain extensively.

I did get a site up and running after ~1-2 weeks of work including the necessary ritual sacrifices.

edit: writing in a rush, grammar and text are messed up. edit: GPT 4, copilot and midjourney. AFAIK I had no half measures.

Its a bit of a game changer if you already understand things so can point it to the right tasks and also clock that it's screwed up just from glancing at the code.

I haven't tried Cursor yet, I just get it to write or refactor functions bit by bit maybe that's better for monolithic tasks?

I should probably write a post about this, but comments are easier.

0) LLMs introduce the challenge of automated semantic verification.

1) LLM work should be broken up by usecase. These use cases lie on a semantic complexity continuum.

2) Tasks like classification are simple to verify (precision recall etc.). Tasks that require semantic complexity like summarizaiton are on the other end.

3) Anything on the high semantic complexity end of the scale needs expert human review.

4) Chained or complex calls greatly complicate verification

Which is why you are getting use out of it. There is a human (you) in the loop, and the calls are not complex.

As long as a human is reviewing the output of the LLM every time, its great. This is the vast majority of "generative" output.

Its when you have things like agents, or chained calls that things go awry. ITs why proof of concepts are easy, but production is hard.

For the record, many people have called this out, including people at Open AI, and AI ops was the largest sub group in YC's fall batch.

I've been in the same boat. Earlier this year I wanted to up my game as well with an improved website and blog to go along with it, not even sure which frameworks to use.

Using ChatGPT has been a hit or miss experience, helping me maybe 30% of the time as well. But when it did help, it helped me massively, especially for quickly setting up Wordpress PHP designs and accompanying CSS. I can honestly say I couldn't have done my web and blog redesign if it weren't for ChatGPT. Not because I wouldn't have been able to get the knowledge without it– more so because I wouldn't have had the patience to figure all of this out in my spare time.

Using ChatGPT has certainly been a more fun experience than browsing documentation, but I did have to do the latter about half the time anyway.

I have generated some toy experiments for a monthish - gpt to generate code to do something - and did generate some web frameworks and pipelines - I did not have it generate a database - although that may be something to try next. I wanted to it generate circuit diagrams and molecules

My experiments https://www.qualitycontainers.io/ https://github.com/msrobot0/llmlexperiments/

> I had hoped to be able to get complex tasks done entirely with assistance from the LLM. In the end it helped maybe 20-30%.

Ok but if you would do the same task again, how much would it help you?

> When I went back to the videos of people using chatgpt to build a website in under 30 minutes - its always someone who knows the domain extensively.

You are one step closer to become this person.

>Ok but if you would do the same task again, how much would it help you?

100%, since I would never do it again.

I'm mind blown because it's already so good and I do imagine that it will get as good as you need it to become.

And yeah as you said with domain knowledge it works very well

What model did you use?

Was GPT4 out then? I self taught to build a Web app before LLMs, for the second one, I had LLMs and it helped me so much, in the area you mentioned: clarifying concepts and spotting obvious bugs due to inexeprience

Yup. I had Gpt 4, copilot, and midjourney to assist.

There is a difference when an LLM is used in limited ways, by someone who knows enough to discard bad output.

I'd put it this way - ChatGPT will help you build a ship in the desert without hesitation.

Reply on news.ycombinator.com