Big fan of this write up as it presents a really easy to understand and at the same time brutally honest example of a domain in which a) you would expect LLMs to perform very well, b) they don't and c) the solution is to make the use of ML more targeted, a complement to human reasoning rather than a replacement for it.

Over and over again we see businesses sinking money into "AI" where they are effectively doing a) and then calling it a day, blithely expecting profit to roll in. The day cannot come too soon when these businesses all lose their money and the hype finally dies - and we can go back to using ML the way this write up does (ie the way it is meant to be used). Let's hope no critical systems (eg healthcare or law enforcement) make the same mistake businesses are before that time.