GPU power seems to be increasing as fast as it always was, but the things we can do with that extra power are less impressive. So if you're playing games, it's not so big a deal any more if your graphics card is five years old.
Just look at the new API features we've been getting, they aren't really the kind of things you'd rewrite your engine for, with the exception of Vulkan / D3D 12 / Metal / etc. But Vulkan / D3D 12 are really just API changes that don't reflect underlying hardware changes.
On top of that, a large percentage of games are made on top of engine middleware with good support for mediocre hardware (like Unreal and Unity, both of which run on phones if you want).
GTX 1080TI is March 2017 11.3 TFLOPs going back five years you find a GTX 680 from March 2012 is 3.1 TFLOPS. So less than 4x gains in 5 years.
Around 1997 you where seeing nearly 2x gains per year.