Hacker News Reader

Here is this new thing, and here is how it is different than anything else.

We invented the Turing Test decades ago. Since it became irrelevant with ChatGPT [1], we need new tests.

[1]: We can discuss if ChatGPT passes the Turing Test or not, but I think we can now all agree that being able to have a convincing conversation is not a good test for intelligence.

[1] I disagree. I think we can agree there needs to be a refinement on the definition of intelligence, but I think LLMs passed the 1950 definition of general machine intelligence.

No, it’s not suspect in and of itself. Often you need to develop a new benchmark when solving a new problem. It’s common to see this in software engineering/CS papers too.

Of course, one should always be critical of benchmarks, and there is an obvious opportunity for bias here that should be reviewed with care. But your phrasing suggests that this is unusual or actively suspicious, which it is not.