Not sure if it's because it's totally opaque, or because it adds a dependency and potentially a whole heap load of transitive dependancies. You're putting your game/app/whatever's performance in the hand of someone else (moreso than for infrequently run CPU code), not only hoping they didn't write a crappy implementation of whatever, but hoping nothing they depended on did either.
Not to mention, they might have done it differently than you would. For example, lots of times you want to do gamma correction using square/square root, and not using pow 2.2 (monitors vary so much that this is in all likelihood just as/nearly as correct, and it's vastly cheaper).
... Also, you'd think in a discussion about how to do most of these things, they'd actually show the code and not just say 'oh a library magically handles it'