This happened because a few corporations were able to capture most of the internet due to a mix of laziness on the part of users and also the death of independent blogs and fan sites.
I suspect that the independent internet died for pretty much the same reason why many people won't post anything interesting or authentic on social media anymore: Being active online can lead indirectly to real world harm to yourself or your family. Evil people use the internet to harass other people and to attack their livelihood or put the safety of their family at risk. Many employers now discriminate against employees or even just people applying for jobs who openly express opinions on the internet. And even posting pseudonymously isn't completely safe because it is usually possible to deanonymize posters if an attacker is obsessed enough.
The sad reality is that many people do inauthentic things for the sake of career advancement (think of all the college students who do extracurriculars or projects for the sole purpose of "resume padding") and refuse to do things they actually believe in for the sake of their career. When creating quality online content becomes a risk to career advancement, people are going to stop doing it. In the long term, that means a siloed internet full of corporate propaganda, SEO optimized blogspam and clearly inauthentic posts.
In short, I believe Google search is getting worse because the internet is getting worse. And the internet is getting worse because our society permits employers to have a chilling effect on the online speech of their employees and those who might be their employees someday. And because the independent internet has shrunk dramatically, the handful of corporations that control almost the entire internet can now get away with siloing it off so that's what they're doing.