I remember loving IB when I was just starting out with iOS. If you're building a quick throwaway app using stock UI components, it's great. However, as your app grows in complexity, and the needs for UI customization increase, IB quickly becomes a crutch and a source of hard-to-find bugs.
The nail in the coffin for me was realizing that IB would frequently get out of sync with my code if I renamed anything or moved code around. I found myself frustrated clicking through menus to hunt down incorrectly set IB outlets, fix references to nonexistent classes and methods that had been changed, and rewire event targets that had gotten out of sync with my code. There's also some UI customization that just isn't possible with IB, for example much of the stuff that the UIAppearance APIs now allow isn't customizable within IB, as well as any time you make a totally custom UI widget (IB doesn't really know what to do with it, and if memory serves me just shows a blank rectangle). Trying to add additional customization to IB user interfaces (beyond what IB can do) generally involves tagging them in IB, then retrieving them by tag in code, and then making the change. Convoluted and high maintenance IMHO.
When you add to this that changes to .xib files can't be easily merged, that it's generally pretty useless to diff them (the format is complex), and the fact that you have to look at both the .xib and your code to piece together the end-to-end functionality, it was pretty clear that dumping IB was going to be a big win for me. Turns out, it was.
I now override the -(void)loadView methods of all my UIViewControllers, and create and position all of my custom UI for each controller in there. I never have to worry about what IB will or will not let me do. Additionally, in the case of customized UI elements, I either create subclasses of UIKit classes (often subclassing UIView or UIControl), or categories of existing classes if the change to their functionality is small. Doing things this way also makes it much easier in the rare case that I need to do custom drawing within -(void)drawRect. I suppose what makes this a bit easier for me is that I have no problem visualizing the UI I've written in code before I actually see it. Other people may miss seeing their UI in IB and being able to visually edit it. I suppose I got over that pretty quickly.
If you're still not convinced, consider that by coding your UI by hand you'll also have a lot more control over memory usage since you control what gets created and when, you can share resources between elements more easily (colors, images, fonts etc.), and also avoid the performance hit of your app parsing .xib files.
But what does hand-coding your UI do to code length, you might say? In my experience it adds about 25% to the length of your UIViewControllers. Perhaps a small price to pay for all these benefits? For me it was the right tradeoff. Try it - I suspect you'll come away feeling empowered and understanding a lot better how UIKit works.