We're not scared in the slightest. I'll reconsider when Swift has inline ASM, allocators, linear types, move semantics by default, region analysis, and a type system that guarantees freedom from data races (oh, and when the code is open-sourced, and targets both Linux and Windows as a first-class citizen).
Swift isn't intended to be a systems language: it's an application language. Anyone who's capable of choosing Swift as of today was just as capable of choosing Objective-C yesterday. And as the Rust developers themselves have discovered over the past two years, deciding to become a systems language doesn't happen overnight.
(In fact, on a personal note, I'm ecstatic that Swift has been announced. ADTs! Optional types! Pattern matching! Think of how many features are no longer alien to people who want to learn Rust! And hell, the syntax is incredibly similar as well, which further reduces friction. As a hardcore Rust contributor, I want to shake the hand of each and every one of Swift's designers.)