Imagine further that the "calf-cow" model of the web could be replaced with something more decentralized, e.g. a "content-centric" network instead of a "source-centric" one... such that there would be little room for "selling" traffic.
What needs to be improved is detection and prosecution of the affiliate scammers.
I respectfully, but strongly, disagree.
While you or I might be able to find what we are looking for, we cannot extrapolate this to mean that "users" (in the general sense) can do the same. Moreover, I'm not satisfied with the status quo of locating stuff on the internet. It's better than it used to be, but I believe we can do better - much better.
But since you mentioned it, it's not unfathomable to think that NSA could help the situation. The search technology they develop, e.g. Binney's work with B+ trees, is funded with tax dollars. Maybe the public should be allowed to use it (not the data, necessarily, but the software). I wonder who would argue search technology is some sort of weapons-grade "munitions" like was claimed about cryptography.
The benefit might be a dramatic reduction in internet crime without the cost of (over) regulating the internet.