JetBrains has an IDE called AppCode, an alternative to Xcode (not open source). But it still relies on Xcode for distribution and the UI designer, and it lags far behind the update cycle of Xcode. And that is a commercial company with steady revenue dedicated to making IDE. Imagine the difficulty of an open source alternative.
Honestly, if you were willing to rely on the undocumented SourceKit API then you wouldn't have nearly this many problems.
You are not allowed to link binary not compiled by XCode to iOS/osx ecosystem.