I'd rather see an open, search index whose URL's and scraping properties are shared with others. Then, small players, even researchers, can just rescrape the likely-good sites to build their own local stores. The whole Web filtered down to what is likely to be useful and safe (no malware) for a wide variety of people.

If it needs to be paid for, then a commercial product that's priced by organization size. Given to researchers for free if their outputs are non-commercial or permissive licensed. Discounted otherwise. I usually start with how Windows is priced for personal or server use to be profitable and widely accessible.

That would let people re-create data sets like RefinedWeb without violating copyright law. You'd still have to consider terms of service, contract law, etc. We have stronger, legal defenses of scraping for internal use, especially non-commercial. Knocking out copyright issues would be a huge help.