First is that MSBuild hard-codes references, so building a multi-targeted project with references to a multi-targeted NuGet package is a nightmare. From what I've read in the documentation, Paket (an alternative package manager) is supposed to be designed to help with this, but it was already enough of a struggle to introduce NuGet at my work, and NuGet is "built-in" to Visual Studio.
Second is the fragmentation of the .NET frameworks, a la this article. Combined with how .NET makes native interop fairly easy (i.e. platform dependent packages), you get either bloated monolith packages or a proliferation of satellite packages.
Other than that, I've had a fairly easy and straightforward time with NuGet. But then again, I've never seen a project with more than a dozen or so dependencies, compared to using Maven in Java where you pull in dependencies like hotcakes.
I'm split. I'd love to see the churn settle down so the surrounding ecosystem can mature. On the other hand, there are definite pain points with the existing tooling. I remember seeing declarative package references as a feature request for Roslyn, which would be absolutely wonderful, but even if it's released fairly soon, it's probably going to take years for adoption to spread.