Hence we get the same thing over and over again until someone convinces people there is a better way, or simply does something different and makes more money.
In this environment there is an endless flood of salesmen - some benevolent, some charlatans - trying to sell the next fix.
The problem is as pointed out by Alan Kay among others that software engineering as a discipline seems to be a pop culture that always is uncritically on the lookout for the next methodology promoted by it's inventor. Without a hint of criticism or self reflection.
It's not so much about selling books and farming engagement. It's about commodification of labor. The "best practices" tend to cater to the lowest common denominator to get at least something done. This lowers bargaining power of labor, drives down wages and increases capital's share of the pie.
This is inevitable in capitalism. Happens to all crafts. This is why we can have cheap crap but not nice things.