Assuming for the moment that by some bizarre turn of events my system were implemented...
1. Why is this a problem? Obviously legacy systems will have to stay in place until smartphone adoption is as ubiquitous as credit cards are today, and the only person you're inconveniencing by using them instead is yourself.
2. My iPhone is smaller than my wallet, and is more indispensable for travel (and would be vastly moreso if I could use it for even a small portion of payments.) I guess this one is just a culture difference, but it looks like the world is moving my way.
3. In keeping with the above, you could still carry around your debit card if you want the backup. (Your phone could even act as a passive RFID backup, so you don't have to carry around the card.) But since it wouldn't be your primary method of payment, you would be able to up the security on it such that every payment must be signed for, and signatures actually validated. A bit kludgy, but that's how it is supporting legacy technologies.
4. What happens right now if your debit card is stolen? Except without the "Enter PIN" part.
But seriously, since this technology is customer-initiated, the security of it will continue to improve as security features on phones improve. Whatever my new phone supports — fingerprint, face recognition, geolocation — I can use to authorize payments, and the merchant never has to worry about it.
5. There's a tradeoff between security and speed, here as in all things. Maybe this system would wind up being a little slower than our current one, which provides almost no security... I guess that's a fair tradeoff for me. After spending 20 minutes finding my groceries, whether it takes me five seconds or 45 to authorize the payment isn't a big deal. And as smartphones improve, it will get faster and faster.
Anyway, if you think what I'm proposing is the heavier, bulkier, or less secure option, I'm not sure if you've thought about what a debit card transaction actually entails.