> I thought summarization was a task AI was considered good at.

I found this article was a good explanation why this isn’t the case:

https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...

TLDR: it all comes back to the same principle: LLMs are predicting words based on their training. Which means they are not actually summarizing ideas from documents (or whatever other task we give them).