I definitely see it improving in the future (I personally view Xamarin development as a better option for Android than standard Java), but it's not something I would recommend for an actual workplace to use.
I agree it's an inconvenience but I find after the initial setup it works quite well. My workflow is to have a Mac Mini under my desk, with a USB extension lead taped beside my keyboard. Then I plug my iPhone in and once I push F5 Visual Studio talks to the build host which talks to my iPhone and builds/deploys/starts debugging and everything "just works".
The biggest impediment to being productive with Xamarin, in my opinion, is that every second release manages to break something.
Storyboards are a decent solutions for certain problems, but if your app ends up as one monolithic or a group of large storyboards, you are going to be structuring and defining things in ways that will have friction with some of your better practices in your C# code.
That's not to say Xamarin "should be a cakewalk, you're doing it wrong." I found pain with dealing with the same area you describe -that back and forth with designs built heavily in storyboard. Not to mention, so many helpful posts on dealing with UI features and quirks seem handled or better handled in plain old NIBs + custom code, etc. Drag and drop layouts with D+D segues are nice, but when the client wants things "just so" I'd end up having to tear things out of the storyboard/designer to get all the things moving and performing well.
The storyboard failed me with AND without Xamarin.
I am also really loving Swift when you can use it on its own (I.e., without relying on an older Obj-C based library). The strictness of the type system eliminates many runtime bugs.