Some hypotheticals and criticisms I've heard from people outside of YC:

1) There are so many YC startups that the brand has become diluted and it doesn't get you as much PR/street cred as it used to.

2) The YC team is working with so many other startups that they don't have much time to work with your startup.

3) If you're already in a position to raise more money from angels at a better valuation, is the YC rolodex worth the equity you're trading?

Generally everything comes with pros and cons, I don't think you can say that it could never be bad or that there are no downsides. At some extreme point the above must apply, e.g. it probably wouldn't be a good idea for google to apply to YC.