Nvidia is singled out for hatred on desktop, because everybody else has cleaned up their act quite a bit. ARM is practically irrelevant on desktop, so the fact that ARM only has closed-source options is just not that big of a deal.
ARM is very relevant on mobile, and on that front so much is closed that basically everything deserves plenty of hate. The GPU is not a frontier there.