Largely because of improvements in graphics, game budgets have gone way up and studios are afraid to take risks. Hence you get mostly sequels, online cash cows, linear on-rails shooters/cinematic experiences that are easy to sell. Content is expensive to produce so studios are afraid of "wasting" it. Yet I was still discovering hidden things in the original Deus Ex 10 years after I first played it.
1997-2004 I think was the golden age of gaming where technology was good enough that cool things could be made, yet bad enough that small-ish creative studios could compete. Games were popular enough to make it profitable for studios to make them yet not too mainstream to dumb them down for the lowest common denominator.
System Shock 2, Deus Ex, Metal Gear Solid, Silent Hill 1-3, Resident Evil 1-3, No One Lives Forever, Age of Empires, Warzone 2100, Mafia 1, Half-Life, Syphon Filter, Parasite Eve - in no particular order; I am sure I am forgetting many many more great titles from that era.