> you could claim that a chess AI lacks understanding of chess.
No you can't, the AI has 100% of the rules of chess, there are no rules the human understands that the AI doesn't understand. Compare that to human written texts, we haven't been able to code all the rules for human written text so the AI doesn't have them. So instead of the AI being able to evaluate what it writes itself, the AI has to try to imitate what humans have written without being able to see if what it wrote is good or not. In chess on the other hand the AI can check itself if it did good moves, since it can play test games and see if the moves leads to victories or not.
If you want to try to code an AI that doesn't understand chess, then write it entirely without encoding any chess rules at all, just train a neural network based on human played games. I'd bet it be hard to even have it do legal moves reliably. That is the level of our current natural language AI.