It couldn't possibly be that those titles that you want to see are not being developed because few people want them?Given that the titles I mentioned were so popular that they spawned entire series, highly rated by fans and critics alike, and as I mention they are still being emulated today... No, I don't your theory there is even slightly plausible.
And of course there have been other games in similar genres since then, some of which have been very commercially successful. But they've tended to come with obnoxious DRM, DLC and other silliness in an attempt to keep them that way despite the pirates.
My question isn't really about those specific titles, it's about innovation. The games I mentioned either defined or greatly advanced their respective genres for a generation of gamers. Where is the next Doom, the game whose basic concept has just never been done before on that kind of level?
The music industry is figuring out that, unlike the 90's, they can't put out a cd with 1 or 2 hits and expect people to buy the whole thing. So yes, sales are down. How much are they saving by shipping fewer CDs?
Book sales may be stagnant or declining, but units are up, and a similar argument applies: how much of the falling gross revenue is offset by reduced distribution costs?
Part of the trouble with this argument is that you seem to be assuming that most of the cost of creating and distributing good music and good books came from the physical distribution cost, which obviously doesn't apply in the same way for digital distribution. But that's not really the case.
For good music and good books -- and to be very clear, I am not talking about "high art" or any such nonsense, I'm just talking about work that is well made -- a large amount of the overall cost is up-front and doesn't change much just because of modern technologies. It's the human factor: having the right team and a well-equipped studio to handle the recording and production for your song, having a skilled editor and qualified proofreaders for a textbook, and so on.
Several of my close friends have worked in these industries in various capacities, and they paint a bleak picture of cut corners and cost savings in their industries today. Certainly the books I've bought in recent years myself have been sadly lacking compared to both the editorial quality and production values of a few years ago, and knowing what is going on behind the scenes it's all too clear how that is happening.
Sure, there's no excuse for bundling music into "albums" any more. Digitial distribution of individual pieces of music and automatic playlists in every media player application make the whole concept of an album nothing but an artificial marketing tool. Likewise the old fixed page sizes and tight page limits for books have little relevance in a digital world.
But the content is what's hard. Writing/performing good material, and elevating the results to the level of professional quality, is hard work and requires real skill. This isn't going to change. No computer AI algorithm is going to spot the way a paragraph is phrased awkwardly because it relies on an idea that isn't introduced definitely until three pages later, at least not any time soon.
There's a popular anti-copyright cliche that musical performers should just give their work away as advertising and hope to make a living playing live gigs. Maybe that works for those people, but how does a composer or a lyricist give a live performance? How do editors and researchers make a living when everything they do is about information? How does a level designer or the guy who sketches out the concept art for that great end of level boss that comes to symbolise the entire game?
The creative industries are vast machines today, and someone has to pay all these people during the creation/production stage of a work if that work is going to benefit from their skills and expertise. If the realisable profits become so small on a work that it's going to be commercially risky to make it at all, what actually happens is often that either it doesn't get made or it gets made without the help of these skilled people behind the scenes and winds up a worse product for it. And as I keep trying to point out, that doesn't help anyone, including the people who enjoy the work (or would have enjoyed it, if it had been better made).