There's a bunch of loosely connected points here, but the author seems to take umbrage with the inability of GPT4 to break its alignment training of referring to itself as an AI language model.

If anything, the author is demonstrating that alignment is going better than the doomsdayers are admitting. The LLM is prevented from offering an opinion on the best episode of Love Boat, despite (presumably) losing the author as a future customer.

Alignment training does come at the cost of performance- the author was not able to achieve their goals of talking to an AI pretending to be a human- but as an AI language model, it also won't teach that person how to build a bomb.