A GPU specializes in Vector math, which is what is used in gaming graphics. Hence the name Graphics Processing Unit and it's original use case. It just so happens that LLMs are powered by vector math much like graphics applications are.