Language Models Do Not Recognize Identifier Swaps in Python

gave this to ChatGPT4, asked it to complete the function:

  len, print = print, len
  def print_len(arg: list):
     # put code here

First try (body only):

  len(arg)  # len is now print

Second try after me telling it's wrong:

  print(arg)  # print is now len

Third try after me telling it's wrong:

  len(arg)  # len now functions as print
  print(arg)  # print now functions as len

It got the correct answer only when I asked it to first print what the function would look like without the swap, and then I asked it what it would look like with the swap.

EDIT: I tried again with:

  complete the function body:

  len, print = print, len
  def print_len(arg: list) -> None:
     """ This function prints the length of the input list   """
    # put code here

It gave the correct answer on the first try.

Twitter thread: https://twitter.com/AVMiceliBarone/status/166215065632766361...