Hacker News Reader

Comment by LASR | Hacker News Reader

I want to use this comment to state one thing, not directly addressed to you.

Stop using GPT as a database! GPT is far more useful a reasoning engine that can accumulate fuzzy data and then provide various views or transformation of that data.

So asking GPT to parse a Wikipedia page and then asking it to teach you from it - this is a much more successful usage than what the author in the original article is doing.

It is not useful as an accurate source of information. It’s inaccurate sometimes, and it’s hard to tell when. OTOH, as a formatted, it has some actual world-changing potential.

View on HN

Is it? I couldn't even get it to figure out an NGINX config, despite feeding it the documentation and URL rewrites it would need, and the prior Apache .htaccess file. Spent days on it. Consulted with people who know more than me about prompt engineering. Nope.

Reasoning is not its strong point, IMO. It's a next-word prediction model, why would it be? It's doing what an LLM does. Frequently, nonsensically.

I've actually had great success with the automation of boiler plate DevOps nonsense using GPT-4, including: GitHub actions, Caddy scripts, Docker files, etc.

Every second that I didn't have to:

- go look up a similar problem on Google

- open three or four stack overflow tabs

- read through the stack overflow links

- copy out the answer

- change all the variables to match variable names that I want

All of this represents a huge time saver for me. I'm honestly baffled that people lack the ability to use LLMs in a productive and optimal manner.

> I'm honestly baffled that people lack the ability to use LLMs in a productive and optimal manner.

Huge +1.

I’ve got it to do a large part of my work for me by chaining some simple API calls.

The fundamental conceptual gap I see - people often ask it to do some “thinking”. Then are annoyed by the inaccurate output.

A simple example is doing word count and it getting the wrong answer very confidently.

Of course it sucks at that. It doesn’t have a counter internally. But if you ask it to number each word in the input and output a list, then ask for the word count, it gets it right every time.

Almost like how a human might count words manually.

I agree. I often use it as if it is a code generator with a decent grasp of human language.

Here is what I asked it to do the other day, and it just did it right away and got it entirely right.

> Write async tokio Rust code that takes in a Vec<u8>, writes it to a temporary file, calls /usr/bin/svc infer pathToTemporaryFile, reads and deletes temporaryFile+".out" and returns Result<Vec<u8>> containing the contents of temporaryFile+".out"

I knew what I wanted, I could have written it myself, there was no unknown but it took fewer keystrokes and honestly when asked to write small components like this with pedantic detail, it does an incredible job and the output is easy to validate quickly.

If this is a good example of using ChatGPT for writing Rust, I suspect that the problem may be with Rust async.

If "chaining some simple API calls" is "a large part of your work", I can see it being incredibly useful. I already used it in this way a few times and it saved me some time indeed. However, as a software developer I feel like it's actually a tiny minority of my work; most of it is "doing some thinking" and slowly unfolding the inner parts of systems in order to understand how they work and how to fix them or make them do what I want, for which GPT is at best useful as a rubber duck to speak to.

It's a pretty amazing rubber duck. I used it to brainstorm approaches to tracking changes to a database table using SQLite and ended up building an entirely new library:

- Article: https://simonwillison.net/2023/Apr/15/sqlite-history/

- ChatGPT transcript: https://gist.github.com/simonw/1aa4050f3f7d92b048ae414a40cdd...

Actually this is the aspect I love about GPT. The less I have to spend dealing with idiosyncrasies of arbitrary API boiler plate and can get down to the business of leveraging said things together or designing aspects of said things to achieve a new goal, the better.

As someone who is far from an expert in DevOps, this has saved me a lot of time recently, and it worked fairly well to get past duct taping infrastructure together to then leverage.

> The less I have to spend dealing with idiosyncrasies of arbitrary API boiler plate

In my experience, GPT saves me some time when I don't really have to think about idiosyncrasies anyway. In cases where I do have to think about them, GPT doesn't offload that from me. It often tries to mash unrelated frameworks together or makes up non-existing APIs that it would find useful for the given task. When it does work well, it mostly boils down to automated series of copy'n'pastes. Saves a bit of time, sure, but doesn't really transform the way I work.

Who knew the next "billion dollar security mistake" after C's memory handling would be LLM-suggested misconfigurations? I bet someone will try to sell an LLM-based service to fix thr bugs caused by other code-generating LLMs.

I've found that LLMs give good results for really basic stuff, and are increasingly fantastical as you head towards the tail of the distribution.

If you just need the beginner's gist or are completely unfamiliar with something, they're not bad. Getting specific answers to corner cases is often a waste of time.

luckily, large complex things are often made up of small simple things

Did you learn anything in the process? You'd still have to over the output and understand it, otherwise nothing has changed except the method of producing something you don't understand.

If you're not DevOps, why bother learning whatever framework is used today, but which will be replaced by something else next year? I just want to get my stuff deployed and that's it.

Saying that DevOps tooling changes every year is absolute nonsense. The tooling is fairly standardised nowadays (Docker, Kubernetes, Ansible, Terraform, Prometheus, CI/CD tool of your choice) and every part of it is over half a decade old.

ChatGPT will only help you with the simplest of tasks, and even then you're gonna fail if you don't know to correct it. If you go beyond the most basic of the tasks you can learn how to do in a few hours, good luck fitting them in a chat prompt.

I'm not sure about standardized tooling.

Docker is being replaced with Podman. Supposedly a drop-in replacement, but the tools built for Docker don't play nicely. You have Ansible, but there's also Puppet, Salt, Chef. There's Terraform and also CDKTF (from the same company too), Pulumi, plus tools from each cloud provider. There's Prometheus and Nagios, Datadog, Sentry, New Relic.

The irony of CI/CD tool of your choice is not lost on me.

I'll give you Kubernetes itself as the system.

Yes, the concepts (containers, orchestration, reproducible deployment, infrastructure as code, unified monitoring, continuous integration) are well-established. How you get there? Not at all.

Yes, the OP was hyperbolic when claimed the tools change every year. But the landscape of devop tools is changing. There are dozens of dev tool companies funded in the 2023 winter batch of Y Combinator alone. Someone sees potential to disrupt the established tools. And it will be disrupted.

There was a world before Kubernetes and Terraform. It wasn't that long ago.

https://www.ycombinator.com/companies?batch=W23&tags=Develop...

Well, you say

> Saying that DevOps tooling changes every year is absolute nonsense.

And then you list 5 technologies with an alleged lifespan of half a decade, which sort of hints at the idea that someone would need to learn/update a new tooling skillset once a year.

In that light, GP's comment doesn't sound so wild.

The comolexity of Docker I've needed over the last 5 years definitely could have been solved by ChatGPT.

The amount of Kubernetes I needed was probably more than what ChatGPT can solve, but at the same time, I spent an awful amount of time looking at various templating solutions.

over half a decade old! haha, so assuming you don't use it the day it comes out, but after a couple of years, most people have only used it a couple of years. Unless you've only worked a couple of years, you've gone trough a couple of these.

Yeah that’s true, why bother actually knowing anything about the systems they run your software /s

I have a finite amount of time. I can choose to learn and know more about the software that I'm developing and make the software better, or I can spend some amount of time to learn the DevOps part, or some mix in between. In some cases, the ratio is 90:10 in favor of software development.

Is nginx hard to understand or something ?

in surprised how few services developers I’ve encountered who don’t know hot to configure a web sever or LB.

Yeah honestly when I took a stab at it a couple years ago it was frustrating - all the material I ran into told you what to do but not why which left me high and dry when things didnt work as expected, or when I wanted to find more information about the meaning and organization of directories or other contextual background information.

Here's a thought experiment - seeing how software problems are often people problems, how knowledgeable should we expect a software engineer to be about biochemistry (the systems that "run" people)?

Nginx for a developer vs biochemistry though ? That’s a leap.

No, before I had “a problem I don’t know how to solve” and now I have “a solution I don’t know how to maintain”. Both may be suboptimal in some contexts, but one results in my application having a deploy pipeline.

Sure, but I have never seen it provide best practices solutions.

You always have to walk back and do more research to find the best practice.

And it doesn't learn. I stopped counting the times I told it it was wrong only for it to acknowledge the error and provide the same answer as a fix.

ChatGPT is nice to learn about new ways of doing things, new ways of composing things, but then a human must intervene to make sense of the mess that was generated.

> It's a next-word prediction model, why would it be?

If you want to make accurate predictions, you'll need reasoning or at least some process that approximates it.

"There a no bananas. Max is looking for bananas. Will he be successful: [yes/no]"

GPT-3 or 4? Re: reasoning, apparently it gives much better performance if you ask it to write out its thought process step by step (because it has no "internal" thought process; it must be externalized).

Yeah, that sounds like a GPT-3.5 problem to me. I wouldn't trust 3.5 to write me an Nginx config, but I'd give 4 a shot at it.

I just tried using GPT-4 to translate a random htaccess file I found to nginx and it seemed to do a good initial job: https://gist.github.com/simonw/8174f653a0c08c120830c56332054...

I tried your prompt and it works with 3.5 too, but I'm not qualified to verify the output (75% of it looks the same to me though).

> It's a next-word prediction model

So are you. https://ceoln.wordpress.com/2023/03/31/its-just-predicting-t...

And your qualifications and background are what? Why is this on a free WordPress instead of in a scientific paper if it's something to take seriously?

Counterpoint: https://the-decoder.com/why-large-ai-language-models-dont-le...

Do you think I wrote that? And what do qualifications have to do with anything?

I assumed you did, as it's a very strange post to link to on a very strange blog, and doesn't lend credibility to the idea that a human being is an LLM simply by existing, and if it was such a compelling idea then it wouldn't be languishing on a free WordPress.

Qualifications are a great way of figuring out who's worth listening to, but that's not always the case as AI/ML folks prove with regularity, but it's nice to know if someone isn't just an armchair commentator and isn't peddling baseless information.

> a human being is an LLM simply by existing,

That's not what it's claiming at all. It's taking umbrage with the word "just" in "just predicting the next word".

A gun "just moves some metal from one place to another".

"by existing" = because the blog post exists does not prove something is true.

I'd sooner listen to people who are leaders in their field when it comes to LLMs than someone rambling on their free WordPress blog.

It is just an LLM. That's an objective fact. IBM's Watson is a pipeline of different algorithms. Neither are like a human brain, but multiple processes are going to be more brain-like in concept than just an LLM.

I'm thinking you didn't read it, then, because it isn't making any claims about LLMs!

>So are you

Followed by the link. You made the claim that I'm an LLM.

What actually is your point?

Maybe you are, but I'm not :).

Interesting! I found it super helpful with gitlab-ci stuff and docker, as well as much bash and makefile stuff. I would have assumed it would handle NGINX configurations.

Tbh, it isn't surprising that it fails, its surprising that it has subjects where it is very useful.

Interesting. Curious as to what specific task you gave it and how it failed.

My team has been working on several advanced techniques using reasoning on LLMs. The stacked performance of all of these techniques combined yields is quite impressive.

I will paraphrase Geoff Hinton here - Predicting next word requires understanding context. Otherwise, you cannot predict the next word. It is a big deal.

Are you using GPT 4 ? It is far superior when compared to GPT 3.5

> GPT is far more useful a reasoning engine

Hilarious to read this advice when GPT, by it's own designers, cannot reason.

It can’t reason - as in there no internal memory or intelligence in there. But you can ask it to generate a reasoning chain as part of its output. And then extract that output and do something else with that. That’s the reasoning it can perform.

Look up the Sam Altman podcast with Lex. He specifically talks about reasoning engines.

There is an internal memory, but it's very short lived, like a register in a traditional CPU architecture. It's like a CPU disconnected even from L1 cache, let alone RAM.

I like to think of consciousness as not the substrate, but the information flowing through it. Not the H2O molecules in a river, but the current. We're like eddies and swirls in mountain streams, our thoughts are the patterns of flow, not the unmoving rocks that set them up.

In this sense, when an LLM like GPT produces output, it's like a loop that has been cut, a single pass through what would be a circular process in a human brain. It can take a fixed number of steps from a standing start, but no more. This is because there is information flowing through it, transformed step-by-step. It just doesn't recirculate, it always exits after a fixed number of steps.

Think of an unrolled for loop that always exits. It takes a few steps, which look like a loop, but can never actually iterate.

I gave it a ten page document a few weeks ago and it could not answer questions coherently from it.

I find GPT4 useful, but I still feel its wishful thinking to call it a reasoning engine. It is, in the end, a giant model for predicting the next word in a corpus which you condition with inputs. I find it most useful to think of it as exactly that.

I asked it give me specs for a well-known microphone. Over the course of several queries, it gave three different numbers for the requested spec, none of them matching what the manufacturer gave on their website.

I'm curious - how do you pass the article to GPT? Are you doing any formatting?

Reply on news.ycombinator.com