Hacker News Reader

I asked how to configure something in PlatformIO and it told me it was possible to do exactly what I wanted, and gave me easy to use syntax with full examples and explanation. When I questioned it further, it pointed me to the URLs of docs covering the feature.

None of it was in any way real - the feature I wanted does not actually exist.

> the feature I wanted does not actually exist.

Did you try asking it to implement it?

Tomorrow on HN: "GitHub no longer accepting PR from ChatGPT"

Why would it need to? It knows that it already exists!

Seriously, though, sort of. I asked for advice how to implement it and it just gave me more non-existent options.

I asked it for examples of drugs to treat high blood pressure, and it gave me a few real drugs that can be used for that purpose, plus one drug that entirely doesn't exist. No results on Google for that word.

However, the name sounded very much like it could be a drug name for a real blood pressure medication.

where did the urls point to? Docs that didn't have what it was saying it did? Makes me wonder if ever in the history of that url (ie wayback machine) if that feature was ever mentioned...

> Docs that didn't have what it was saying it did?

Yes. Docs for related features.

> Makes me wonder if ever in the history of that url (ie wayback machine) if that feature was ever mentioned...

I wondered that too, but, no, a Google search for the purported feature keywords (it was supposedly an option to add to my platformio.ini file) produced literally zero results.

From the ChatGPT announcement post:

> ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.

Yes, this is indeed a known limitation.

So the other day, chatGPT was doing a great job acting as a trivia quizmaster for me when it casually threw out the 'fact' that Pope Francis was the first South American to be elected pope, and that he served in the role from 2013 until his death in 2017. (For the avoidance of doubt: Pope Francis is still alive).

I think we can all understand how that sort of thing counts as 'plausible sounding but incorrect'.

What I think is surprising is that 'plausible sounding but incorrect' can also encompass 'rich and internally consistent novel programming language semantics with fully worked examples of applications, and rich explanations of how they work'.

> 'rich and internally consistent novel programming language semantics with fully worked examples of applications, and rich explanations of how they work'.

Oh for sure, that is a really cool finding. Didn't mean to negate that, just provide a little context.

For me, it made up software packages to do what I needed to do.

I also tried to have it generate some 6502 assembly to multiply two numbers and generated an iterative add approach that failed to do any adds. So I informed it that it forgot to add and it realized it's mistake and corrected the code.

I tried to get it to optimize further and it generated some code using shifts that I really need to double check. It also generated a table-based method that was woefully incomplete. As I continued to get it to correct the table function, it was clear it didn't really understand 6502 assembly that well.

But I've pasted large block of C# code and then gave it sample data and it understood the code well enough to generate the correct results of the code.

I tried several times to play chess against it. It made reasonable if poor moves for the first 10 moves of the game. It also gave plausible-sounding if entirely incorrect explanations of why the moves were good, such as "I'm playing Nf6-g8, which is a good move because it defends my pawn on f7 [it doesn't] and controls the center, allowing me to set up an attack on your king".

At some point though, it always got stuck and insisted on making illegal moves, even when I corrected it by providing the current board position in FEN.

I'm not a chess expert but I would expect there is a lot of information available about chess openings but not so much about midgame (it sometimes appears as if you just play a bunch of standard moves before the actual game begins).

One source even mentions that a chess opening consists of 10 to 15 moves which would match your observation that the AI gets worse after that.

Maybe the AI would appear more competent again in the endgame since I assume there is more training material for that again.

A bit surprising that it provides illegal moves since it should have enough training material not "understand" the rules but apparently prefers to produce sentences which appear like chess moves from the training material without any connection to the rules.

ChatGPT has no knowledge on how to play chess. It can't even get the basics of openings or even colors right.

Prompt: What are the moves that characterize Latvian Gambit in chess?

ChatGPT response:

  The Latvian Gambit is characterized by the following opening moves:

  White moves the pawn in front of their king two squares forward, from e2 to e4. This is the pawn sacrifice that gives the opening its name.

  Black responds by moving their pawn in front of their king two squares forward as well, from e7 to e5. This move is known as the "gambit pawn" because Black is offering a pawn to White in exchange for a lead in development and control of the center of the board.

  White follows up with the development of their light-squared bishop, moving it from f1 to b5. This move attacks Black's gambit pawn and puts additional pressure on Black's position.

  Black can respond to this move in a variety of ways, depending on their playing style and preferences. Some possible options include moving their knight from g8 to f6 (defending the gambit pawn and preparing for a counter-attack), moving their knight from b8 to c6 (developing a piece and preparing to castle), or moving their f-pawn from f7 to f5 (solidifying their control of the center of the board).

  These are the moves that typically characterize the Latvian Gambit in chess. It's important to note, however, that not all games that begin with these moves will necessarily follow the same path, and the specific sequence of moves can vary depending on the choices made by the players.

PSS For the record all of the above is hilariously wrong and obvious to most chess players :) The real Latvian gambit is 1. e4 e5 2.Nf3 f5 (and only f5 is sacrificing the pawn).

I've asked ChatGPT several times about FeatureScript [0], which is a language used to create features in the cloud based 3D CAD software, Onshape. It will happily invent syntax and code that has almost nothing to do with how FeatureScript is written. When confronted with this, it will admit that it made stuff up and doesn't know much about Onshape or FeatureScript.

[0] https://cad.onshape.com/FsDoc/index.html

I tried to play rock paper scissors against it, but it kept insisting that paper beats scissors. Never did convince it otherwise.

Did you tell it to “play correctly with the standard rules”?

I did, it even volunteered what those rules were, but when I asked it to keep a running score it always scored it as if paper beat scissors.

Perhaps it's under the impression that if paper beats rock, and rock beats scissor, then by transitivity paper should beat scissor.

Yeah, I also played it a bit, and even after telling it each rule, eventually it’d screw it up again. It must have a hard time with non-transitive games?

I tried to play rock paper scissors against it, but it generated moves for me as well as it, then announced it had won at the end. When I made a move it said "sorry, the game is over".

I had a similar experience where I asked it to tell me about creating a GUI with nim, and it told me to use the 'forms' module from the standard library, and the example code it printed was showing an easy to use and nice API. Took me a while to realise it wasn't my fault that I couldn't find any documentation about it, anywhere....

There's comments elsewhere in this submission proposing using ChatGPT for designing programming languages, but what about for designing APIs?

> I would definitely question who is doing the inventing here.

> But also, potentially a fruitful line to explore would be having chatGPT collaborate on designing new programming language ideas.

It's the same in your example... It invented but you guided it with extremely deep domain knowledge. I know nothing about HyperCard and wouldn't have been able to make anything out of anything you just described.

No, the concept of a 'message stack' was something it came up with. When I asked it for more details it confidently supplied them, including what it was used for, and how it worked. Then I just asked it to show me it in a script, and... it was happy to oblige. It was making it up as it went along, all I was doing was taking it at its word and asking questions.

It is like the world's most committed improv partner. It will agree and add forever.

Could it be that that there is a proprietary extension for HyperTalk, that chatGPT somehow learned about?

Congrats, you have just made a successful logical argument against intellectual property per se!