I have literally been doing we development since their was a web, and the companies I developed for are openly hostile to the idea of putting their valuable, or perceived valuable, information online in a format that could be easily scraped. Information doesn't want to be free, it wants to be paid for. Unless the information shared pulls visitors to the site it doesn't need to be public.
That's a cultural and societal problem, not a technology problem. The motivations (profit) are wrong, and don't lead to true innovations, only to financialization.
So long as people need to pay to eat, then information will also want to continue to be paid for, and our motivations will continue to be misaligned with true innovations, especially if said innovations would make life easier but wouldn't result in profit.
We won't achieve post scarcity, even with widespread automation (if AI ever brings that to fruition), because we haven't yet fixed the benefits that wealth brings, so the motivation to work toward a post-scarcity society just doesn't exist.
Kind of a chicken and egg problem.