The first IDE I used was Eclipse, and it kind of set my expectations for what an IDE should do. My main complaints about Eclipse were "it's slow (lol Java)" and "it's hard to find settings in all the 100 pages of options".
Here's some stuff that comes to mind (it's 10 PM here so bear with me):
* Buggy. Infuriating stuff like sometimes not being able to expand compiler errors. xcassets go weird all the time (orphaned assets). It'll lose connection to the Simulator and I just need to quit everything and re-open. Speaking of the Sim, not even sure if they ever fixed testing Today Extensions, that was super-broken for ages.
* Crashy - this was under control for most of Xcode 7, but it's back since 8 beta 6 and not fixed in the GM. How it handles crashes is also horrible. It restores the windows out of order so a tiny window I had for documentation gets the whole project opened in it and visa-versa.
* Code-completion is not very intelligent. It got a lot better last year but not good enough.
* Refactoring tools are extremely weak for Objective-C, and 100% non-existant for Swift. This bothers me a lot. In a young project I like to rename and refactor things a lot and doing it with regexps or "change it and see what breaks" feels like the stone age.
* Interface Builder is still not a pleasant experience. Xcode 8 helped a lot. I wish it had a better UI for AutoLayout stuff but I'm hesitant to complain about this since I have no concrete suggestions.
* The iTunes-inspired UI in infuriating for stuff like seeing progress when there are multiple things going on
* Source/version control integration is useless, I don't know anyone who uses it. Do it right or don't waste your time. Only thing I use is blame view but even that is excruciatingly slow.
* Project file format is version control-unfriendly.
* Doesn't have niceties that Eclipse spoiled me with such as a TODOS view that lists //TODO: comments across the project
edit to add I also have a lot of trouble with the debugger, where a regular old NSException crash will send me to UIApplicationMain with zero context (aside from the exception details that get printed to the console), but I'm not sure where to place the blame there, lldb? the runtime?