It amazes me how often apps built for toddlers make fundamental mistakes: interactions that little, fat 'starfishy' fingers can't manage; instructions in text rather than voice or picture.
Flash games are the worst for this: at least with an iOS game drag and drop is easy and intuitive for a toddler - but trying to drag and drop with a laggy mouse is hard. Toddler will accidentally scroll the containing browser window or switch apps, which is very distressing for them. A simple, locked full-screen browser would be a godsend for many parents (which is what an iPad is).
I have found young children cope fine with the one button. Once they learn not to touch it until they are finished they are fine - if they accidentally push it they know what happened.
It's also great to see quality, intelligent software appearing for small children: Balloonimals is a constant favourite, Ramp Champ should be rebranded for the under-5s as it has just enough gameplay for them.
Something interesting: as my kids (6 & 8) have grown up playing iPhone games with tilt control they get very frustrated when they play games with button-only control. They naturally try to tilt the device. It shows that the next generation of gamers have no need for physical buttons - something that we were told was essential for gamers.
I've been meaning to write a blog post about this subject for years.