Comment by telotortium | Hacker News Reader

Comment by telotortium | Hacker News Reader

The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page is how to effectively use GPT-4 to write this script, hence why I've submitted it under this title (go to https://gwern.net/tla#effective-gpt-4-programming).

Interesting topics include:

· Writing a good GPT-4 system prompt to make GPT-4 produce less verbose output and ask more questions.

· How to iterate with GPT-4 to correct errors, generate a test suite, as well as a short design document (something you could put in the file-initial docstring in Python, for example).

· The "blind spot" - if GPT-4 makes a subtle error with quoting, regex syntax, or similar, for example, it can be very tricky to tell GPT-4 how to correct the error, because it appears that it doesn't notice such errors very well, unlike higher-level errors. Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance.

· If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script

> Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance.

By that argument, one should always make it use a language that's as hard as possible to write a compiling program. So Rust or Haskell or something? I guess at some point it's more important to have a lot of the language in the training data, too...

I think this is exactly the right conclusion.

The main complaint people have about strict, thorough type systems is that they have boilerplate.

Obviously boilerplate doesn't matter if a machine writes the code.

The type system also becomes helpful documentation of the intended behavior of the code that the LLM spits out.

Assembly has a lot of boilerplate, and every other language is an abstraction that gets a language-machine to write it for us.

So we'll just move to a new standard where we write LLM prompts describing function behavior and it will output the Rust or whatever that we end up storing in our SCM.

There's a fundamental difference though. The LLM is itself inscrutable, while all of these programs used to be written and understood by humans. The language used for programming used to be specified and have unique (hopefully) coherent syntax and abstraction boundaries. Now it's "anything goes" and nobody seems to know how this stuff ends up getting used...

Someone might accidentally find it works well and then we might all end up writing fairytales in iambic pentameter describing the use cases of software we want...

Yes, you would think so. Haskell would also be good for encouraging stateless/FP programming which makes unit-testing or property testing much easier. I can make GPT-4 write test-suites for functions which are straightforward data structure transformations, like rewriting strings, but I struggle to create tests for any of the imperative stuff. There presumably would be some way to test all of the imperative buffer editing Elisp code, but I have no idea what.

However, in my use so far, I have not noticed any striking differences in error rates between Haskell and the others.

> If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script

What an absolutely based take by GPT-4

Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…

Slow yes, but footgunny? What do you find particularly footgunny about python?

Where I grew up CQK was short for "Can't Quit the Koolaid."

<jk>

Let's make it a thing!

Does anybody have a UrbanDictionary account?

I modified the title slightly to use language from the subhead. (Submitted title was "Effective GPT-4 Programming", which does have the advantage of being a phrase from the article itself, but is more of a section heading than a description of the entire article. For the latter purpose, it's probably too generic.)

Reply on news.ycombinator.com