1) Learning new concepts in computer science. For example Swift uses a lot of modern concepts around type safety, closures, parallelism, synchronization, etc. These are a big learning curve but they are not specific to Swift and you'll notice other modern languages are adopting the same concepts.
2) The UI frameworks are fundamentally different. If you're an option expert at XCode constraints, storyboards, etc, none of the applies to android and vice versa.
I can tell you this, if you do bite the bullet and learn both very well you'll be surprised how much it helps you learn other systems more quickly because you've see all the concepts before.