I'm happy both ways:
Either you say that a pocket calculator understands arithmetic, and that LLM understand language, which is something trivial. If a pocket calculator understands arithmetic, than previous substitutes to calculators, such as an abacus, do too. In this case, a word dictionary also understand language. And it is basically what Chiang's article says: the LLM don't understand language more than a word dictionary does. If you disagree with Chiang, it looks like you do only because you don't understand what he is saying, or somehow are not mature enough to realise that Chiang may use a different definition of "understanding" than yours in a fully legitimate way, like everyone is always doing when talking about plenty of subject.
Or you pretend that a pocket calculator understanding of arithmetic is somehow different than the one of an abacus or other obviously inanimate object who are obviously not thinking.
I don't think that LLMs as black boxes are fundamentally novel, I just think that their internal design is novel, and their generality and ability to give correct responses to complex topics is far beyond anything previously. For example I would argue that wolfram alpha has a poor understanding of language and a very good understanding of math. I would argue that LLMs have an excellent understanding of language and a mediocre understanding of math, but are able to temporarily increase their understanding of math through document retrieval and "thinking" (or whatever you want to call the process of iteratively generating tokens that build on each other to result in a final response).