The main problems I had with it when I was using it, mainly Xcode 4/5/a bit of 6:

- super-lame extensibility support - there was basically none. You could write addins for it, but you had to use undocumented everything. To figure out the APIs, one would run classdump on Xcode's private frameworks, guess based on the method and class names, and experiment in conjunction with gdb. Then every other version, stuff would break

- inflexible window layout - yes, I know, Apple knows best, but, seriously, why do I have to have grep output share the same panel as compiler errors? And why does that panel have to be so tall and thin?

- stupid missing functionality - no M-x back-to-indentation, no M-x shell-command-on-region, no keyboard shortcut for jumping to the matching closing brace, no function for jumping to the next function, etc. (Xcode 4 shipped without even search and replace in selection! - and it took them about a year to get it back in)

- confusing rules about window layout - I used Xcode for 2 years and never figured out what was going to happen when I pressed Cmd+Shift+Y. I tried to set up a separate tab for the debugger, but it was 50-50 whether Xcode would use it. In the end I just gave up and let Xcode do what it wanted

- no memory dump window in the debugger as I recall?

- lame assembly language support - you can't step instruction by instruction using the standard keys (you have to dive into the lldb command line and do it the hard way), and the register display is shared with the locals, so you keep having to scroll (and thanks to the inflexible window layout you can't split them apart and have them separate)

- seemingly no thought put into which files are per-project (and so live in version control) and which are per-user (and so want to be gitignored or whatever). The default is for per-user build configs, which is... well, not a crazy thing to support, but it's a poor default

- when I was using it all the time, though this is going back a while now, half the time lldb never even worked properly: https://news.ycombinator.com/item?id=5125210

I've used numerous packages over the years, and I've ended up getting used to every single one... except Xcode 4.x and later.

Xcode 3.x, after a fairly ordinary period of adjustment, I had no problem with.

(I'll give Xcode 8 a quick spin this month; I used Xcode 7 a bit a couple of months ago and it suffered from many of the same problems. But is it my job to try using stuff that was awful in the past, just on the off-chance that it might have been improved, or is it Apple's to make software that doesn't suck so much that after using it every day for 2 years I basically resolve never to use it again and indeed to this day actively turn down work just on the chance I might have to use it? There are, of course, arguments for both... but I'm going to go for the latter myself.)

The first IDE I used was Eclipse, and it kind of set my expectations for what an IDE should do. My main complaints about Eclipse were "it's slow (lol Java)" and "it's hard to find settings in all the 100 pages of options".

Here's some stuff that comes to mind (it's 10 PM here so bear with me):

* Buggy. Infuriating stuff like sometimes not being able to expand compiler errors. xcassets go weird all the time (orphaned assets).　It'll lose connection to the Simulator and I just need to quit everything and re-open. Speaking of the Sim, not even sure if they ever fixed testing Today Extensions, that was super-broken for ages.

* Crashy - this was under control for most of Xcode 7, but it's back since 8 beta 6 and not fixed in the GM. How it handles crashes is also horrible. It restores the windows out of order so a tiny window I had for documentation gets the whole project opened in it and visa-versa.

* Code-completion is not very intelligent. It got a lot better last year but not good enough.

* Refactoring tools are extremely weak for Objective-C, and 100% non-existant for Swift. This bothers me a lot. In a young project I like to rename and refactor things a lot and doing it with regexps or "change it and see what breaks" feels like the stone age.

* Interface Builder is still not a pleasant experience. Xcode 8 helped a lot. I wish it had a better UI for AutoLayout stuff but I'm hesitant to complain about this since I have no concrete suggestions.

* The iTunes-inspired UI in infuriating for stuff like seeing progress when there are multiple things going on

* Source/version control integration is useless, I don't know anyone who uses it. Do it right or don't waste your time. Only thing I use is blame view but even that is excruciatingly slow.

* Project file format is version control-unfriendly.

* Doesn't have niceties that Eclipse spoiled me with such as a TODOS view that lists //TODO: comments across the project

edit to add I also have a lot of trouble with the debugger, where a regular old NSException crash will send me to UIApplicationMain with zero context (aside from the exception details that get printed to the console), but I'm not sure where to place the blame there, lldb? the runtime?