I think they have, but it is subtle. It could be argued that Instagram was a shot over Adobe's bow.
Adobe of today is like RIMM in 2007 before the original iPhone was announced.
In the low-end space there was the Aviary suite, which despite really interesting potential seems to have pivoted to mobile. Pixelmator and Paint.NET have done an extremely good job at providing basic layered image editing functionality, but I don't see anything emerging to challenge Adobe yet.
A competitor in the Pro/Prosumer space would need to develop, from scratch, extremely sophisticated, feature-rich applications to replace Adobe's big 4: Image Editing, Vector Graphics, Motion Graphics, and Video Editing. They'd also need to tightly integrate these applications. This would be HARD. I don't think that the effort/risk/reward proposition looks very attractive for startups or their investors. Also, while game programmers might have some useful experience, I think that computer vision programmers would be much more familiar with the problem domain.
Apple, or maybe Corel probably would have done this a long time ago if it was feasible.
Which brings up an interesting question...if After Effects and Premiere have strong competition, why don't Photoshop and Illustrator have equally strong competitors? I've already taken a guess, but I'm interested in other perspectives.