I suspect that the independent internet died for pretty much the same reason why many people won't post anything interesting or authentic on social media anymore: Being active online can lead indirectly to real world harm to yourself or your family. Evil people use the internet to harass other people and to attack their livelihood or put the safety of their family at risk. Many employers now discriminate against employees or even just people applying for jobs who openly express opinions on the internet. And even posting pseudonymously isn't completely safe because it is usually possible to deanonymize posters if an attacker is obsessed enough.
The sad reality is that many people do inauthentic things for the sake of career advancement (think of all the college students who do extracurriculars or projects for the sole purpose of "resume padding") and refuse to do things they actually believe in for the sake of their career. When creating quality online content becomes a risk to career advancement, people are going to stop doing it. In the long term, that means a siloed internet full of corporate propaganda, SEO optimized blogspam and clearly inauthentic posts.
In short, I believe Google search is getting worse because the internet is getting worse. And the internet is getting worse because our society permits employers to have a chilling effect on the online speech of their employees and those who might be their employees someday. And because the independent internet has shrunk dramatically, the handful of corporations that control almost the entire internet can now get away with siloing it off so that's what they're doing.
Discussions moved from independent forums to Facebook groups and Reddit. Discord later. Facebook and Reddit are not more anonymous than independent forums.
The decision to limit the API might therefore actually be a good for the average user if it stops archives such as ceddit and unreddit to show your posts after you've edit or removed them. Most people probably don't know about them though and believe their content is gone just because they click delete.
Facebook and Discord groups have been private since as long as I can remember. Sure, someone might leak if you invite the wrong people but compare it with Twitter where you can find all posts from user X including keyword Y. Or even do it through Google cache. It takes minutes to find something to use against anyone if you really want to. Based on the last election in my country, it can be something as simple as listening to the wrong artist when you're 14.
> our society permits employers to have a chilling effect on the online speech of their employees
More generally than that. Dylan Mulvaney made one beer advert while being trans and got death threats. She would be counted as a "contractor" rather than an employee.
My suspicion is that this was the beginning of the public presentation of corporate infighting, where what the public saw from Google began to be less about good products and more about the products that particular execs, managers, and developers needed to "win" in order to bolster their careers (or what they perceive as such). This is why Google has developed a reputation for killing useful products, often replacing them with flashier and more frustrating alternatives (if at all).
I don't like putting this on external factors. I think Google is doing this to themselves. It's a more private and stretched-out version of what's happening very publicly with Twitter right now.
And surprisingly it's free.
I'm just trying to wrap my head around it but I think I'm making sense of it now and it's kinda like that race to the bottom I see sometimes mentioned here? Do I got it right?
On top of this, siloing adds the extra bonus that it breaks things like The Internet Archive. There are a lot of communities on Slack and Discord that are just prime for vaporising without a trace. A TON of knowledge will just disappear once someone decides to turn it off, for whatever reason.
The end result is companies trying to capture more of that value for themselves and not letting Google eat their lunch, which required moving content and communities behind closed doors.