That is an operation I never do. I guess slowness can be attributed to that the menu is application aware, customizable menu options per application. But there is also a slow animation when opening the menu.

And on top of that as soon as you hover above an item in the taskbar it wants to show a thumbnail preview of that window.

All combined the user interaction feels a bit weird.

I turned off "Animations in the taskbar" and I think the context menu got somewhat faster.

Do you have many shell extensions installed? Try disabling some items on the context menu.