you'd be able to give them a novel problem and have them generalize from known concepts to solve it. here's an example:

1 write a specification for a language in natural language

2 write an example program

can you feed 1 into a model and have it produce a compiler for 2 that works as reliably as a classically built one?

I think that's a low bar that hasn't been approached yet. until then I don't see evidence of language models' ability to reason.