This is not very substantial work. They used the API, gave it more challenging questions on 14 prompts, and judged whether it met expectations. I would have expected something much more intelligent from Aaronson, although this kind of light-weight analysis is par for the course with Marcus, who has been fighting his rearguard action against large parametric models for many years, and is not above motivated reasoning. It's like one of those open letters that people get roped into, only hosted on arXiv.