People are scared by some horror stories. Terrorism in the traditional sense of the word, as done by a government, is what is happening here: raid a few people and the whole community backs off. I have to admit that it makes me more hesitant too.
> I'd like to use Tor more if it was a bit speedier.
It's more than just having enough bandwidth. Tor picks random servers to relay through. Say you are in silicon valley and want to connect to Gmail via Tor. Tor might, in a really bad case, choose a server in Germany, Japan and South Africa. Now your traffic has to travel from the US to Germany to Japan to South Africa and to the US again. Limited by the speed of light (light is slow), that takes a while -- probably almost a second for a single round-trip. Connecting via HTTPS will suck and online shooter games are out of the question, no matter whether you have 56kbps or 180gbps available.
In most cases it's a lot better than this, but it remains random chance. I used Tor daily for a few months and often didn't notice any difference between normal WiFi and Tor, but sometimes it was also a bit annoying.