I really don't understand how a compiler can crash that much ( actually i've been coding in many client and server languages before and it's the first time i see a compiler crashing while i code). It makes me really nervous, because it shows that unit tests aren't good enough, and that the language is really in alpha stage, not more.
As for compilation time, you may think it's just a matter of optimisation. But the problem is that until those optimisation arrive, you can't rely too much on type inference (main source of slowdowns), which diminishes a lot the beauty of the code.
Now that was for client side. Server side probably adds its share of issues.
I used to be very optimistic for this language, but i'd say the swift team only has one more shot to make it really production grade, before the word spreads that this language is for now just a joke ( maybe apple using it for its own app would help allocate more resources).