Great writeup! I made a couple of changes in your code and it seems it gets ~90% of examples correct (inside the first three categories you have) with aroung 65% having either optimal path or correctly saying that there is no path.

What I changed was:

1. Used code-davinci-002 (codex)

2. Instead of using an explanation of how these tasks work, I changed your code prompts slightly and instead gave it 3 examples (two with paths, and one to show it what to output when there is no path present).

3. Changed the output so that it has to tell me which edges, instead of which nodes, it is traversing (this helps GPT to avoid using nonexistent edges it seems).

Here is an example: https://pastebin.com/D7Hn95VC Last "Problem" is the real problem we want to solve of course, everything else is static.

I'll post my code when I finish testing :)

EDIT: I used Codex not because I think it's better suited for this, but because it's currently free. I don't have enough credit ATM to run 1000 iterations with text-davinci, so I have no idea what difference using Codex made.