You don't enter a PIN? That process easily makes it take longer than cash.
Now that being said, we get an SMS on every transaction, and for online transfers, we have two factor authentication using the mobile phone. We can call the bank any time to report a stolen card and would be covered for a fair amount of money depending on the account, as long as you report it within 24 hours as I remember.
In any case, plenty of people do struggle with it- the old especially. But that's not the biggest factor. With a swipe everything is done by one person- the cashier. PIN entry requires a back and forth between customer and cashier, and I've seen plenty of cashiers not notice that the PIN entry has gone through, etc.