Comment by goodside | Hacker News Reader

Comment by goodside | Hacker News Reader

For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output. Many of these are discussed in the Twitter thread the OP links to, but I'd like to highlight a few of my favorites here too.

- GPT-3 can produce simple SVG diagrams (https://twitter.com/goodside/status/1545883023719239689) from an English description, or incrementally modify a given SVG diagram (https://twitter.com/goodside/status/1545888869207400454).

- GPT-3 can produce diagrams using Graphviz dot notation (https://twitter.com/goodside/status/1545109709035900928), including diagrams that incorporate real-world context (https://twitter.com/goodside/status/1545112039147847682) not given in the prompt.

- It can emit and understand HTML and Markdown, including table markup — e.g., "Produce an HTML table giving the members of the Beatles and their instruments"

- It can accept/emit multiple named files, with arbitrary content, simply by embedding those files as values in a larger JSON/YAML document. Here, (https://twitter.com/goodside/status/1545848754883919872) I demonstrate completing the file tree of a trivial Python package given a README.md.

- It can understand the raw JSON file contents of .ipynb files, successfully following directions given to it in markdown cells, emitting multiple code cells, etc.

While all this is fun, as many people have pointed out the model has a bad habit of "hallucinating" — producing highly specific, wrong output. Avoiding this (https://twitter.com/goodside/status/1545793388871651330) is more of an art than a science. Be highly skeptical of any output you get from GPT-3, and search for verification methods where you can.

This is very cool.

Two uses come to mind: 1. Alternative variable name suggestions. I am writing code, and I try to figure out what to name a variable that about a data frame which is my performance data which takes the maximum of random samples and batch size leaving data size and thread count, and all I can come up with is unbatched_speed or something. I can think of a lot of scenarios where a context sensitive thesaurus / autocomplete would help. If it's obvious to an AI, maybe it'll be obvious to a junior dev?

2. I expect there are a lot of cases where not technical people could understand code, but get intimidated because it looks like code. An AI to say what the code for the business logic that they care about, but using prose.

#1 is feasible now for short (<4K chars or so) blocks of code. Just paste the code in an indented block as you would in a comment, then after the block ask in English for a list of suggestions for what the variable could be renamed to.

2. Is a great idea! I bet that could be useful.

Maybe a way to do the same with formulas. I always get hung up on the odd notation and the symbols that aren’t defined anywhere.

2 sounds like auto-generated long-form documentation? That would be a major win for devs and non-devs alike.

I tried writing a tutorial with GPT-3 a few months ago with quite impressive results: https://simonwillison.net/2022/May/31/a-datasette-tutorial-w...

"For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output"

I'd like to see it get good at continuing poetry.. It's pretty awful at that, from my own experiments.

I enjoyed this one enough that I’ve committed the sonnet to memory: https://mobile.twitter.com/xkcd/status/1513922269122281473

It seems to understand a fair bit about themes and rhythm but it just cannot do rhyme. Something to do with the token encoding.

Has anybody tried code reviews? As in "GPT-3, please do a code review of this code".

For small examples, yes, it works:

Prompt:

  Perform a code review of the following module, identifying any bugs:
  ```
  #!/usr/bin/env python3
  def fib(n):
    """Return n-th Fibonacci number."""
    if n <= 2:
      return 1
    else:
      return fib(n-2) + fib(n+1)
  ```

GPT-3 completion:

  The bug in this module is that the n+1 parameter in the return statement should be n-1.

But don't expect miracles on real-world files, because GPT-3 struggles with long-term dependencies in text (e.g., remembering what happened ~4000 characters ago).

Seems to work for a factorial function also

   Perform a code review of the following module, identifying any bugs:
  ```
  #!/usr/bin/env python3

def factorial(x): """This is a recursive function to find the factorial of an integer"""

    if x == 1:
        return 1
    else:
        # recursive call to the function
        return (x * factorial(x+1))
  ```

A: The bug in this code is that the function is attempting to find the factorial of an integer by recursively calling itself with an incremented value. This will cause an infinite loop.

But if you unindent and then also perform the n+1 error it only will see the unindent issue.

  Perform a code review of the following module, identifying any bugs:
  ```
  #!/usr/bin/env python3

def factorial(x): """This is a recursive function to find the factorial of an integer"""

    if x == 1:
        return 1
    else:
        # recursive call to the function

return (x * factorial(x+1)) ``` A: The code will not run because there is a syntax error on line 6. The indentation is incorrect. The return statement should not be indented.

Reply on news.ycombinator.com