Hacker News Reader

Hacker News

Comment by zapperdulchen | Hacker News Reader

Full thread

zapperdulchen

I played along the example of the chancellor question.

If you deviate a little from the examples in the article GPT-4 gets it all wrong:

Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel)

https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f...

Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel)

https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f...

Interestingly whether you put is or was into the question does make a difference.

Even when allowed to surf the web, ChatGPT gets it wrong:

Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder (wrong, Angela Merkel)

https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc0...

Though it's referring to the right Wikipedia page: https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germa...

View on HN

> Even when allowed to surf the web, ChatGPT gets it wrong:

> Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder

Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not.

When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)

It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…

> It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong

Schröder is the only answer it gave when using functionality that would bring some representation if a list into its context first, and 10it did it with different prompts and mechanisms for bringing a list into its context.

That LLMs are bad at counting-related tasks without doing that is well-known, and not a point I felt needed belaboring.

If you think ChatGPT is "counting" anything, you have massively missed the point.

Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while being Foreign Minister, but he was not a Chancellor. So counting him is wrong.

Imagine reading this comment thread a few years back. Just getting an answer to this question at all from a generic bot would be considered science-fiction and now we're dismissing it based on technicalities.

Not a technicality. It's like inventing a proper head of government that wasn't there.

Also, knowing German chancellors was always easy, so that fact that a machine falls behind any normal dictionary or the German Chancellor's website is poor form.

It's like a calculator getting basic addition wrong. I don't want a future full of poorly performing machines.

He was acting Chancellor. It's like saying a red apple is not an apple. It's a technicality. The fact that we have a completely reasonable explanation to why GPT-4 includes him in the list and you guys still make it sound like the AI is a complete moron is just hilarious to me.

There is no "Acting Chancellor", that is a poor translation of what the job is. There is only a Chancellor, not an "Acting Chancellor" - you have to trust that the Germans and their institutions kind of know who was Chancellor and who was not.

The machine is a complete moron for not being able to get basic data from basic sources right.

> Scheel became acting Chancellor

> Chancellor of Germany, Acting, 7 May 1974 – 16 May 1974

https://en.wikipedia.org/wiki/Walter_Scheel

I know, but the German wikipedia gets it right: https://de.wikipedia.org/wiki/Walter_Scheel

(And so does the German Chancellor: https://www.bundeskanzler.de/bk-de/kanzleramt/bundeskanzler-...)

There is zero ambiguity about who was Chancellor and who was not.

It says "geschäftsführender Bundeskanzler".

Even on https://en.wikipedia.org/wiki/Chancellor_of_Germany he's listed as "Vice Chancellor Walter Scheel served as acting Chancellor from 7 May to 16 May 1974" between 4 and 5.

There's obviously some ambiguity to it considering we are two humans discussing this with a claimed discrepancy between English Wikipedia and German Wikipedia, but your conclusion is still that the AI is spitting nonsense.

There is zero ambiguity under German as law as to who was a Chancellor and who was not. Not a matter of wikipedia etc.

The way to be Chancellor is through article 63 of the Grundgesetz, while Scheel was put into the caretaker role via article 69. This explains it a bit https://de.wikipedia.org/wiki/Vizekanzler_(Deutschland) - Scheel was only taking on the function, not the office.

This kind of giving some machine the benefit of the doubt when in fact there is zero ambiguity is really a path that makes me think we will have mostly machines designed for marketing and other non-critical things.

He's also included in this list between 4 and 5: https://de.wikipedia.org/wiki/Bundeskanzler_(Deutschland)#De...

When you need to bring up article 63 and 69 of the Grundgesetz to prove that the claim is ludicrous, maybe the reasonable thing to say instead is "I understand why you might think that".

You can say "I consider Scheel to be Chancellor", but that isn't the same as Scheel having been a Chancellor.

I honestly don't understand why in situations with a clear ground truth there is a need to debate and why we would want machines to bungle that.

I don't think it bungled it and I don't agree that there's a clear ground truth here. Quite the opposite: you've only convinced me that the semantic ambiguity is real. It's like debating whether interim CEOs should be included in a list of CEOs.

You are, of course, free to ignore German law and its definitions, but that doesn't change the fact who actually was a Chancellor and who was not. Chancellor is a very well defined role.

So the English Wikipedia author is a moron, or is there ambiguity when describing the role in English?

No. I think the problem starts with that the role Scheel had isn't given a name in the article of the basic law that creates it.

So either need to be fully descriptive (e.g., something like fulfilling the functions of the Chancellor, while not ever having the office) or it will be also open to being misunderstood.

The issue here really is that the German succession doesn't ever transfer the office, but only the function (which is different to the US, for example). So here Scheel followed Brandt, but not into the office. Only someone having the office is a Chancellor and there is a specific way to that office.

I see, so when you said the machine is a moron for getting it wrong, the English editor isn’t a moron for getting it wrong due to ambiguity in the law?

I think there’s also a functionally useful way to describe someone’s role as what it functionally is even if it’s not legally that. You’re point is super well taken, if the machine is intended to be a fact oracle, it’s awfully loose and adds a lot of interpretation in areas of ambiguity.

I would say IMO that’s specifically the power of these machines. An awful lot of human endeavor doesn’t require literalism but semantic approximation and interpretation that machines were literally incapable of. Its weird language is enough to achieve that, but I think it’s overly restrictive to assert a broad lack of utility in critical systems. An awful lot of critical systems actually need more “probabilistic” interpretation than literal fact oracling.

Maybe when read in a strictly narrow sense it is even a good translation. Practically, it is more of a caretaker role (especially in the case of Scheel here), though, and interpreting into the phrasing can be dangerous. Naming things that aren't named is always tough, and even tougher when needing to go from one language to another.

I'd agree that often things human are more loose but when there are narrow definitions ignoring them is dangerous.

> The machine is a complete moron for not being able to get basic data from basic sources right.

I'm just going to point out that you're calling an AI moronic for not getting this right when a bunch of humans are also disagreeing with you. Frankly, this really undermines your case that this is an AI failure.

What is wrong with me wanting my machines to stick to the actual definitions?

A "red apple" is an apple that is red. An "acting officeholder" is not an officeholder who is acting, but a person who is acting as if they have the office (... with institutional support - Norton was not acting Emperor).

It's a technicality, but a valid technically and it's a technical question. I agree that it's wrong in a relatively small and surprisingly human way, but it's wrong in a way that including a red apple amongst apples is not.

> Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while

I think this is really an issue of semantics & translation, because that is just what 'acting X' means? (Admittedly not always while still doing something else too, but I don't see that as significant - if it had gone on long perhaps a junior minister in the foreign office would've been named acting FM in his place too.)

That is not semantics. The law is very clear: Scheel was not a Chancellor (and indeed he is not counted as one in Germany: https://www.bundeskanzler.de/bk-en/chancellery/federal-chanc...).

Article 69 Grundgesetz has a sort of "Caretaker Chancellor" that has the function but - importantly - not the office. They way to have the office is through article 63.

It works differently in Germany to the US, for example, where the vice president could become the president, while the vice chancellor only ever gets the function, not the office (unless through a proper vote for Chancellor).

When there is actually ground truth, machines should be able to recover that, not take weird turns.

What distinction are you making between 'acting X' and 'caretaker X'?

I'm British, not American, so I'm not assuming something like the vice president automatically becomes president, we don't have that either. If anything it's even less than Germany since (at least in theory and history, modern media etc. makes it a bit different in practice) there's nothing special about the PM, it's just the governing party's leader. I suppose though you could say the heir apparent to the throne immediately becoming the monarch on the death of the previous one is like vice taking over - but I don't think that detracts from my point because nobody would call that 'acting monarch', they just are.

'Acting X' means doing the necessary duties, but not any major decisions that can be avoided/deferred, while the 'real' replacement is found. Sometimes it ends up being the same person, e.g. the head of some division is 'acting CEO' for a while as the board searches for a new CEO, ultimately ends up going with that person and title changes to just 'CEO' - or they don't, and go back to old job, or maybe quit in a huff, and the person they found is just 'CEO' taking over from the 'acting'.

The real issue here is that people stick to the word "Chancellor", not the "acting" as such.

There is no named role of "Acting Chancellor" created in the basic law, instead someone is just tasked with performing certain duties (and people might sometimes call it "acting Chancellor" or whatever). However, the position "Chancellor" is a well defined & special role (unlike perhaps PM) and whatever that other "acting" role is, it isn't a Chancellor. It is a bit like adding Oliver Cromwell to list of English Kings or Kamala Harris to the list of US Presidents - you can do it but you then move beyond the "standard" definitions.

Charles III was for years described as a 'King in waiting', nobody was confused that perhaps he was already king just because the phrase used that word.

I don't think in English usage there is any meaning attached to 'acting X' which 'caretaker X' (as you're happy to call it) doesn't also carry. Both are used interchangeably, the former you'd put on your CV, the latter might be used by the media when your employer put out the less release announcing it, but same thing: for some reason there is not currently an X, but you are fulfilling some necessary duties that that person would do in the meantime.

When used in a narrow sense "acting X" can capture it, but then sometimes people start to use it somewhat interchangeably with "interim X" etc. and it depends on the specifics what happens in successions etc.

It is exactly what “acting X” means in English.

You can clearly see it is confusing the heck out of people here by seeing "acting X" as a version of X when it is not in this case here.

Yeah, but those people are just wrong. To be fair, it's not something most people have much direct experience with.

I asked for a list of Federal Chancellors of the Federal Republic of Germany and GPT4 included Scheel ("acting"), which makes Schröder the 8th. I often troubleshoot GPT4 replies by asking it for more context. "Is" vs "was" is interesting an find!

https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265...

as others pointed out, it's important how you ask and what you think the truth is. https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaa...

>Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel)

Technically there's no right answer to this question. "Is" implies present. But Angela Merkel isn't the present chancellor. Olaf Scholz is. But he's not the eigth.

Technically, Angela Merkel never stopped being the person who served as the eighth chancellor.

Language is imprecise. LLMs should be able to deal with that.

your first two links are duplicates

Reply on news.ycombinator.com