Smartphones have already stolen away the casual gamer. The people playing call of duty and final fantasy are not the people who will be wowed by the apple TV.
If you look at sales data for the Wii, there is a ton of people who bought a Wii, which game with Wii sport and then bought very few games. (https://en.wikipedia.org/wiki/List_of_best-selling_Wii_video...)
This will simply solidify that casual market into iPhone and Android apps. Consoles have nothing to fear IMO.