The sum of all knowledge and the sorry state of the web
christianheilmann.com
christianheilmann.com
Encyclopaedia Britannica is still there, and it's great. But there's also an army of content creators are there to teach you just about anything. If one source doesn't do it, you have more to choose from.
Yesterday, I read an article about North Africa that mentioned in passing Qaddafi's underground river. A few minutes later I was watching a documentary about it, then a variety of videos about that man. My information binge extended to other African dictators, and will probably last a few days longer.
If I'm curious about something, I can go really deep. I'm not constrained to a few paragraphs in whatever book my local library carries.
You could argue that the web is in a sorry state, but if that's the cost of giving everyone, everywhere access to all this knowledge, then it's a deal worth making. This might be more obvious to someone who did not have access to a well-stocked library.
The problem is not availability, but curation. The sum of all knowledge back then was a well-curated book. Now it's literally all of it, unfiltered.
The problem is not “everything, everywhere” or a lack of filters but the extreme commercialization of all content available, closed networks, the short life of URLs…
The irony here is that Youtube videos from ten years ago are still alive and well. As Youtube makes a much better places for publishing and archiving content than the rest of the Web. With Youtube you don't have to worry about URLs changing or domain names expiring or anything like that. You just publish your video once, get a unique video-id and don't have to worry about anything else. Google's monopoly worked here in our favor for once and they have been reasonably good in not breaking old content (not perfectly, as video annotation got crippled pretty badly).
I think that's where the rest of the Web fell short. The Web has no concept of "publishing". There is no ISBN when you write a blog, no library were you could look up that ISBN. It's all just a file on a server or an entry in a database, that will get mangled and lost in the coming years. Worse yet, the article itself isn't even accessible from the Web, it's mixed together with a user-interface, ads and other stuff or spread across multiple pages. All this makes it quite tricky to keep old content readable and archived for the future.
This also leads to a weird situation that a lot of publications are still avoiding the Web after 20+ years. They publish as ePub or PDFs instead, which you can somewhat access from the Web, but really aren't well integrated into it. But it's by far the easiest way to ensure that a text document published today will remain readable a decade down the line.
IMO, they made the wrong call - it would have been better for the internet as a whole to notify all users that "We have a new 'share by link' option, and no longer consider the old links private. Please update all old links and then click this button to disable the old URL. If you don't click the button, videos shared by URL may be discovered by others in the future."
I use two methods to recover deleted video titles:
1) google videoId part of the URL
2) look up cleaned up (with removed playlist-related parameters, just "youtube.com/watch?v=[videoId]" form) video URL in web.archive.org - sometimes it even has an archived video itself!
Most of these are not information but amusing, random, short videos, and of course some of it is account deletions/disabling - copyright strikes - and people taking down their own stuff, but still, a lot!
When they delete videos, they don't even list the title of the video anymore, they remove the thumbnail and everything, so you won't even know that something you added months ago is no longer available.
They're not, though.
Because Youtube re-encodes videos every couple of years, with new "better" lossy compression algorithms. And each time the videos get successively worse.
Watching a 2008 Youtube video will not only look grainy because it's 360p, but it'll look actively *worse* than it did in 2008 because of all the lossy compression that was applied to it over the decades.
And yes the new codecs really are better. Same quality at lower bitrate.
It seems to be at least as bad as the rest of the web as far as data/link rot goes.
The problem is that Youtube is bitstarving 240p and 360p streams. That makes sense when a video is available at a higher resolution, and the 240p version is for people stuck on dial up or whatever. But in cases where 240p is the highest resolution available, Youtube should provide a high quality 240p stream!
Roughly 1/3 of my YouTube bookmarks are dead and most of them are much more recent than 10 years. They purge videos at an alarming rate.
Tons of them aren't. I've run into many linked from wikipedia footnotes that no longer exist, particularly digitized film from the 20th century. A ton of wikipedia pages still cite old films, documentary footage, linking to videos which were uploaded by Jeff Quitney, who was banned from youtube a few years ago because some of the old films he uploaded contained material that ran counter to modern values (I think the one that eventually got him banned was an old christian film warning children about homosexual predators.) When they banned him, they took down a ton of completely innocuous videos because a tiny minority were offensive.
An ISBN is a string of characters, much like a URI/URL, and offers no more "protection" for long term access and the latter. Books get mangled and lost too; their only benefit is that it is harder to mangle and lose them, and there is likely to be more than one of them.
Do videos survive account deletion ?
In particular, GDPR has provisions about deleting account info after years of inactivity, and Youtube is apparently not an exception (https://qr.ae/pv9o9j)
So except the chunk of accounts that will stay active for the years to come, a big part of youtube videos should be disappearing progressively.
One example: https://news.google.com/newspapers
I think you forget or may not have grown up with the microfiche.
The reality of the internet is that everyone has a voice and things will only be archived if someone gives a damn to archive them. And that's fine. Some information deserves to be transient. Hell, we've survived millennia without this level of information storage. Does every single YouTube video, Reddit post, and Flickr photo really need to live forever? No. Would it be nice? Sure.
Paper is great, because it doesn't just evaporate when you look away. Although it does degrade, it's a slow process—slow enough that you can notice that it's happening and think to yourself, "Gee, I maybe ought to do something about this." It's not the same for transient digital media. There are bonafide news items and other digital content that are now no longer accessible because they were digital-first but the business incentives were so misaligned and/or their legacy has been so mismanaged that, perversely, it's easier to access to the content of a 50-year old news article than it is for others that are 5–15 years old. People can always trawl through their parents' and grandparents' belongings and come across the only known surviving copy of something and donate it to a library or sell the collection in a yard sale or eBay. (Whether they know it's the only one known to exist or not isn't a precondition.) That's not just less likely with the Web, but it's drastically less likely. No one's gleaning much from the unevicted entries in someone's browser cache.
I use UK news sites. BBC Guardian and Telegraph all have their old articles online
For example: https://www.chipublib.org/resources-types/newspaper/
Much has been digitized, I hope they kept the microfiche for longevity.
For web based newspapers, good luck trusting longevity there, just because the library has a free interface to an api...
I am not sure why you replied to my comment l, with talk about not using the web, google, etc.
After all, I specifically said that trusting web bawed properties is an issue.
And you seem to have missed a key part of my comment, the state of paper based newspapers today.
They're dying.
Huge, multi-millon dollar, national newspaper corps, which used to have Saturday newspapers 2 or 3cm thick, when I was a kid, are now a mere 1/2 cm if lucky.
Many local newspapers are gone. Just gone.
The ones which remain, are barely standing on their feet.
So my comment, re archival, is that old methods of scanning phycial newspapers are not viable, and getting worse yearly.
Assuming you live in a place where public libraries are well-funded and can provide such amenities.
Isn't most of it on the wayback machine?
From August 30, 1856. Goes back even further but point mmade. Needs a payment to read but it's there.
Nowadays everyone has figured out how to package the message, and it's super cheap to do so and get it out there. (Incidentally, actual packaging is the same now, crappy products used to come in crappy packages, but no more.)
So now to pick apart an argument you have to be a bit more aware of the actual content, and it's a bit harder to get to the bottom of BS.
There's a local group here that basically specializes in devops for little public projects that have run their course. They even do a little bit of work trying to provide templates for new projects (eg, for a local hackathon), but I'd like to see them go farther.
Bigger, more national or international groups that come up with recipes for projects where they say, "If you build your project on this structure, then we will be more likely to run it for you," I think is a reasonable logical next step for the Internet. Given the task of running 10 projects that 'need' 4 servers each, it would be very good if I could do it all with 20 servers, not 44 (40 + orchestration machines).
We don't have a "PBS for the internet" but then PBS didn't always exist either. You need a beach head of some sort to even propose such a thing to government.
It also hasn't gotten easier to find information. Search results are worse now than 10 years ago because of Search Engine "Optimization". Google search needs uBlocklist[1] with a steadily growing list of manually curated domains just to poorly approximate usability.
Additionally, sites load slower every day. Why does every partial refresh of a site require 2 seconds, even though I'm on a decent computer? On phones sites are practically unusable.
The benefits you're touting for the web could be accomplished with Web 1.0 level of tech or even simpler protocols such as Gopher[2] or Gemini[3]. Everything else is an overall decrease in accessibility, usability, user experience, and the ease of finding or sharing knowledge.
[1]: https://addons.mozilla.org/en-US/firefox/addon/ublacklist/
Now you’re less likely do find that kind of thing and more likely to find a video or a SEO optimized site - which can be much more difficult to parse for verifiability.
It's also never been easier to share disinformation, and pollute the vast sea of actual knowledge that exists on the internet.
Sources of information are silo'd into proprietary closed off gardens run by large corporations who only serve their shareholders. Searching for information has been corrupted by advertisers and the sheer amount of misleading content, that finding reliable sources often feels like searching for a needle in a haystack.
The one exception is Wikipedia, though it also struggles with keeping factual information, and has its own set of issues.
Enter critical thinking. If you dig just a little (e.g. read the talk pages and the edit histories), you can soon learn that some topic has been taken over by POV-pushers and is unreliable. Anything to do with Israel/Palestine/West Bank is unreliable; the boss is a zionist, and so are a lot of the senior staff, so it's not surprising. But WP is a million times better than the web that search engines expose.
Incidentally, I usually search using DDG. But DDG seems to hate Wikipedia; WP results usually don't show up on DDG until page 2. Google surfaces WP results on page 1, if not at the top of the resultset.
Is there good evidence for this claim? Evidence is useful, especially for these claims of general bias.
On the contrary, in general Wikipedia's content is likely biased against Israel when considering its general overall political preferences / bias.
> 5 Studies Find Wikipedia Bias
> Five studies, including two from Harvard researchers, have found a left-wing bias at Wikipedia:
> * A Harvard study found Wikipedia articles are more left-wing than Encyclopedia Britannica.
> * Another paper from the same Harvard researchers found left-wing editors are more active and partisan on the site.
> * A 2018 analysis found top-cited news outlets on Wikipedia are mainly left-wing.
> * Another analysis using AllSides Media Bias Ratings™ found that pages on American politicians cite mostly left-wing news outlets.
> * American academics found conservative editors are 6 times more likely to be sanctioned in Wikipedia policy enforcement.
Neither should it surprise that a scholarly endeavor for bored young people on the internet tends to tilt leftward: there is a big education differential between political poles, and a strong demographic tilt based on age. To put it in a nutshell: if boomers spent their time writing citations instead of pounding out all-caps screeds about ivermectin in the comments sections of local newspapers, there would be a more robust wikipedia contingent.
It’s not definitive proof of anti-Israel bias - it is, however, evidence. As opposed to the unsubstantiated assertion I responded to.
Evidence of what? A leftward slant in Wikipedia, if you believe the conclusion of these studies (or the summaries of them - I haven’t read them all). And it's easy to see that such a slant would broadly coincide with somewhat less support - though far from zero support - for Israel in 2022 and the past couple decades.
I’m not sure how leftward the slant is. It could be slight. It could also not be relevant in specific areas (such as the Israeli-Palestinian conflict). I didn’t exhaustively research the topic (nor did I believe Wikipedia had any specific leftward slant before today.) If you find anything useful, please share your findings - happy to read!
“Quantifiably more leftist editors on Wikipedia” isn’t a good summary of the headlines of those 5 studies.
To your other point, number of editors matter - and the Israel-antagonists presumably outnumber the Israel-supporters - but it’s far from the only thing that matters. To name a few more things that make an editor more or less influential: the number of edits made, the prominence of their edits (for example if it is in the introduction vs elsewhere in an article), the degree to which the edits are neutral vs slanted, and the relative power they have as editors.
I'm not going to try to substantiate it; if you're not already aware that Wikimedia is owned and run by a self-confessed Zionist, you can look it up for yourself. I don't engage with the Zionists on Wikipedia; it's a waste of time, you can't win. Some Zionist admin will come along and declare that you lost the dispute. I'm not inclined to take up arms here either; it's against the rules, and anyway, in matters of this sort, you can't defeat your adversary by producing evidence or substantiation. This is an issue that people on both sides are strongly emotionally engaged in.
This is the claim that needs substantiation, not that a key Wikipedia person supports Zionism or identifies as a Zionist, or that various employees or volunteers support Israel. Those other claims don't seem significant.
The opposite is also true - that various volunteers are antagonistic towards Israel. It's also likely that some employees are antagonistic towards Israel. You can find plenty of people's experiences discussing the same pattern you describe but with different political views - some antagonistic admin coming along and declaring that you lost the dispute.
Various "bosses" at Wikipedia have different views, ideologies and identities. Given that, what on Wikipedia can be trusted, and what can't be? What's special about views towards Israel?
Evidence is useful, but when viewing people as "adversaries", and if trying to "defeat" them in general, the game is already diminished or lost - for most everyone involved. Little productive conversation can occur.
> This is an issue that people on both sides are strongly emotionally engaged in.
Yes. But that is true of many issues in the world.
I'm not particularly worried about disinformation or misinformation as long as good information is out there to improve the signal to noise ratio. But you've definitely hit a nail on the head here, good information has value so there are a lot of incentives to wall it off and charge for it. This is the real danger.
There's even an incentive there to pollute free sources of good information in order to devalue them relative to proprietary knowledgebases.
If we test people on where they went "deep" there is a good chance most will fail the test.
We don't even know what we are trying to do with these networks.
You see it all the time on HN and other forums. If you're an actual expert in a specific (usually scientific) subfield and you read comments about an article in that field, you find that a large percentage are not just factually wrong, but also written in an extremely confident tone by people who have probably studied the topic for about 10 minutes.
By having easy access to all this information people have stopped being humble about what they don't know.
Take my mom for example. She's 80 years old and doesn't use the internet that much. She is confidently wrong about a lot of things she sees on TV or hears on the radio. A recent example is COVID misinformation.
The difference is that my mom can't easily influence millions of others because she doesn't have the reach, but also because people are unlikely to take the word of an 80-year old person without any medical credentials or training seriously. It's much easier to look "legitimate" when you are hiding behind an online persona. If my mom wrote a blog or posted on HN/reddit, she could certainly come off as a doctor, or even lie about being one, and many would believe her. Doing this locally, in person, is much harder and riskier.
> If my mom wrote a blog or posted on HN/reddit, she could certainly come off as a doctor, or even lie about being one, and many would believe her.
I don't think your mom (likeable as she no doubt is) could get an audience of millions just by posting her opinion on HN/Reddit/Social media.
I think the situation is pretty much back to what it was: most of the population have a limited reach of influence. Some people have a much greater reach.
The difference is that the people with greater reach used to be trained journalists who held to a code of conduct and were given that reach by institutions. Now there is no such code of conduct, and the assignment of audience reach is more random, and totally uncontrolled by any institution.
The content producer - audience ratio was just different, a dumb line from a journalist would definitely have an outsized reach that go well beyond what the writer might have expected.
As you note, even today people with a global reach aren’t that many: I could be shouting a lot of thing on top of my soap box, I’d probably not be actually reaching more than a few dozen people, and we have enough content for the effect of a single person to be vastly diluted.
The publishing gatekeepers of a few decades ago projected an image of confidence, but I'm not sure they were actually any more accurate than random youtube videos. The media establishment of today certainly doesn't seem to be.
PS: I hope you're not one of these Wikipedia editors censoring physicists on Wikipedia because you "know" better?, or perhaps censoring virologists on Wikipedia because likewise.
PS 2: Some doctors were wrong about COVID. Some governments were very wrong. COVID lockdowns hurt the world economy worse the GFC.
Long before the web, when I was a child, my father would chide me for my bold claims unsupported by facts with the phrase "Confident; but wrong."
Strongly agree. And I'll raise you: if you can't explain in simple, plain language that a 12-year-old could understand, then you are not enlightened.
I concede that some subjects are intrinsically complex; e.g. the cosmological history of the Universe. But a large part of the reason that topic is complex is because it's not settled; we haven't got to the bottom of it, so there are loads of unanswered questions. How can you explain cosmological inflation in language a 12-year-old could understand? Well, I can't explain it to myself, so I sure as hell can't explain it to a 12-year-old.
This is still a pretty good bar, but if I just repeat what the video said, it seems like I'm explaining it, and then I can give the impression I do know something. I shouldn't get credibility for just repeating a video.
The US has devolved into somewhat of a reputation-driven expertise market, and there are plenty of ways to gain reputation without the expertise. There's still plenty of real experts, but I don't blame them if they want to focus on their work instead of fighting the endless tide of easily produced misinformation.
Map is not territory and I say this after believing it far too long.
It's a big fallacy behind the information highway roots of internet.
And that's half of it.
In the case of Reddit in particular what is it that gets people to trust anonymous strangers? It is bizarre and seems like a mind virus. If an anonymous stranger tells you what you want to hear then you are apt to ingest it uncritically.
For example on social media there is a notion that nuclear brinksmanship with Russia over Crimea is acceptable.
I totally second this, and have a recent concrete example where at the beginning of this year Sweden's (probably) most serious newspaper (what I'd consider a 'journal of record' whose articles should be a point of historical reference), published a long-form retrospective article comparing Sweden's handling of Covid with the way other countries had handled the epidemic, and included several internet 'myths' that had bandied-around on social media, citing them as facts, and even including one 'interview' with what purported to be an eyewitness of one event, which turned-out to be taken from a Facebook post.
I wrote and complained to the responsible editor with citations showing how and where the article was wrong, and a few very grudging emendations were made (effectively saying that even though the reports were still probably true, they couldn't be 'verified').
Totally horrified me that, in wanting something to fit their facts, journalists simply accepted fiction they read on Facebook and regurgitated it in their articles.
I used to hold them in greater regard than that.
That's interesting, because it possibly led to a sort of well-meaning destruction. Was the content surgically edited, or did it grow a warning that some of the info originally published was incorrect, or (more perniciously) both? Consider the affect this has on researchers trying to do studies on misinformation—they certainly want to be able to access the originals themselves.
I guess it has something to do with lack of trust in Established Sources of Information—TV, newspapers, experts and other various figures of authority. The biases of the Established Sources has become more apparent over the years, to the point where scepticism, doubt, and even defensive cynicism are fairly common default attitudes when dealing with the information they provide.
The trust in anonymous strangers is, at least in part, the result of them not being Established Sources. It's not a lying politician, or a deceitful news anchor or journalist, it's just another well-meaning regular person like you. That alone makes you more receptive to their message. If their message happens to align with your existing beliefs, even better. Of course, it can get into cultish/conspiracy territory if any of those beliefs are directly opposed to the mainstream narrative.
If the expert is right it's hardly an "opinion"; it's more a fact. But we still talk in terms of "expert opinion" because there are other experts with, sometimes, diametrically opposite views. Do we get to hear all these experts? Not a chance.
Expert positions can create a false sense of security that we know all there is to know about a subject, and that is just as damaging to society as not knowing the truth.
And yet fringe theories, from quack medicine to religions, weren’t any less widespread than now. They were just fewer of them.
EDIT: Now I think about it, they were more prevalent than now. Look at homophobia - it got global, affected everyone, even non-Christian cultures like China, for centuries.
Do you really consider that knowledge?
> homophobia - it got global, affected everyone, even non-Christian cultures like China
Do you think that aversion of homosexuality starts and ends with Christianity?
It’s not limited to homophobia of course - pretty much every single Catholic claim about human sexuality is antiscientific bull - but I think homophobia is a good enough example of a harmful, false belief that got more popular than anything post-internet.
Second, I find it hard to believe that the Chinese were turned homophobic, and remained so, by a few Christians despite millenia of tolerance, when the vast majority of the country isn't even Christian. It smells like bad historicism.
I also see issues with reading comprehension but not video comprehension. I am not saying that things are better now, just saying that observations should be balanced. A person interested and with attitude towards a specific topic can navigate the learning path with more resources and less friction.
The irony of talking about this on HN, the home of Dunning-Kruger effect.
Yes, social media gives anyone a microphone; non-experts can speak on advanced topics with zero knowledge. But Gell-Mann Amnesia predates social media. The "reputable sources" of yesteryear were anything but.
Experts ignored the media and got their info from other experts. That hasn't changed. The masses were clueless or wrong about most things, that hasn't changed. People were gullible and believed anything they read, that hasn't changed.
To me, the real negative change from the Information Era isn't a decline in knowledge, it's the shift towards centralized socialization online.
People were robbed of their ability to form independent social groups and subcultures, or to differ significantly from the mainstream. There will never be anything like the hippies, punks, or emos ever again. Music used to be the major "gateway" towards political and social countercultures. Now? Music is ultra-commercialized. Online movements are much more easily molded by commercial interests and TPTB. "Counterculture political views" get promoted on Fortune 500 brands' Twitter accounts. Remember the BLM protests in 2020? Police killings increase slightly in 2021 compared to 2020, and are on track to remain stable this year; about the only lasting "legacy" of those protests was that a few grifters bought multi-million dollar houses with donation money. There is no counterculture that doesn't get coopted.
Counterintuitively, something else reinforces the status quo: social media algorithms' bias towards empty controversy. Could Gandhi's peace message succeed today? Martin Luther King? Would they even be audible?
Project yourself 10-20 years into the future, with ubiquitous availability of GPT-3-style AI; will any of what you read online even be from real people? The "open web" movement is missing the wider problem. All the effects I describe are fundamental and unavoidable consequences of the web connecting the world together; regardless of whether it's "open", or kept in corporate walled gardens.
Centralization leads to winner-take-all effects. When every road leads to Rome, it marginally increases prosperity in the periphery, while it overwhelmingly drives traffic and economic activity away from the periphery, towards Rome.[0] The same happens when you "connect all humans" together. Remember the story from a few days ago, about 90%+ of online content being created by 1-2% of people? Any counterculture needs to be insulated from mainstream influences in order to truly thrive; but social media dynamics (karma, "viralness") inherently directs content towards the lowest common denominator, i.e. the status quo. The web itself, for all the good it does, is the problem. What's the solution? Don't know.
[0]: https://twitter.com/nntaleb/status/1184807240198426625 (for a screenshot of the quoted tweet: https://twitter.com/AidanHeronUK/status/1184856198648139777)
Yeah, that's exactly how it works today.
The web has its problems for sure, but don't forget there are still gems out there which the web has made possible. I'd love to see a comeback of blogging culture, but I guess a lot of that has been sucked in to social networks now.
For example, this blog is pertinent to what I’m doing and I didn’t find it for weeks: https://www.northernbuilt.pro/ and only found it by a link to a YouTube video from another one from another one.
We have the Internet Archive, we have Wikipedia page histories, but everything else is ephemeral. If Archive.org ever goes the way of the Library of Alexandria, we'll have lost irreplaceable knowledge of the web itself, and the cultures that existed on it. It will live on only in living memory, but this is also transient, and soon nobody will live who remembers what this time period was like. Wikipedia will not reflect it, you'd have to dig through page histories to find fragments, like historians and archaeologists sifting through ancient manuscripts and ruins for clues.
A potential dark age is forming, and avoiding it right now hinges entirely on the continued efforts of two donation-funded organizations, one of which makes it increasingly harder to view the past, the other facing legal disputes that could see it shut down. I think we need to make archiving the present important, so it does not become a mysterious, inscrutable past for our descendants.
while I think there could be a short period where this info is largely unavailable, I don't believe it will be lost forever. if the Internet goes away, once some new Internet like technology comes around to replace it, these data repos will likely get out back up very quickly. just a matter of how long that blip is
I have a better one for you. Ever wondered why it's so hard? Why web protocols have nothing related to archiving? Why web browsers are a hellscape for aggregating information over time in a meaningful way? Why this continues to be true, despite countless Microsoft and Google engineers writing all these heartfelt posts about knowledge?
If your answer is "because it's hard to implement" than you understand nothing.
You could add for example a 'Newspapers' topic, and then say 'The Springfield Times' and then have 'Articles by date', 'Articles by topic', etc. like a huge database (browsed hierarchically like "WHERE dates BETWEEN '20121211' and '20121213'", etc.). The primary datastructure could be a database, and users can add hierarchies as queries to the underlying database -- a collaborative index (in the literal sense, like a Homepage of the internet) is shown. Any unique 'object' (like a specific newspaper) gets an UUID and a row in the db. I don't know how modern dbs handle sparse data, but that'd definitely be a requirement (i.e. each object can have a handful of millions of possible properties, like publication date, location, author, colour, etc.).
As a humble suggestion, I'd divide all information in: (1) News (all news articles), (2) Publications (books, blogs, magazines, etc.) (3) Ideas and things (countries, planets, people, theories). Any object can belong to multiple categories/classes. I think it's not important that categories be perfectly devised, only that they contain all objects, and objects can be found reasonably well within them. The main point of objects would of course be a link to an actual web page that contains what you're looking for.
Please, someone do it! (I have way too many projects right now)
You could have an Article type with a link to a Person as the author, and many more types of Works besides. You could find any Work by a Person traversing backlinks to find any object linked to that person. Any work where they contributed. Any social media link they posted.
Then query that by a duration of time starting from a specific date. A specific place, if it has it, a group of sites and so on.
If my understanding of it from my time playing with it is correct. I haven't experimented with too broad a dataset yet.
I fully agree, especially when it comes to the "semantic" part of the semantic web. Reusing and publishing ontologies that define those semantics always seemed like an afterthought of the semantic web, when it should be part of the foundation that things on the semantic web are built on.
In most other parts that make up a website (JS and HTML) we figured out how to make reuse (mostly) work by replacing flimsy web references with package management. Ontologies never had something like that, and thus were stuck in an early 00s era of software/ontology development.
Where I work, we are building Plow, a package manager for ontologies (https://github.com/field33/plow) as part of our tech stack to improve that situation and allow people to build applications with large-scale stable semantics at the core.
As part of building Plow we are aiming to make the process of creating and sharing ontologies easier and with that also lowering the barrier of entry to that domain.
I want to put your post in the context of yesterday's HN front-page post lambasting the EU for trying to build a better search engine. The bulk of the comments suggested that a government effort could never be as good as a commercial effort. https://news.ycombinator.com/item?id=32915263#32916945
Your post is a strong counter-argument. All of the points you mention on the massive decline in quality of web content are due to the web being driven by commercial efforts.
Likewise, comments on this current HN front-page post "Despite faster broadband every year, web pages don't load any faster" https://news.ycombinator.com/item?id=32945858 also seem to explain the poor state of the web as being due to commercialization of everything (even the comments about the need for cookie banners-- the logic behind the cookie consent is to regulate the commercial collection of user data).
Google started as a DARPA project, and was a great engine while it stayed true to that ethos. It was the need to commercialize it, thus setting perverse business incentives, which has destroyed it.
Your post praises libraries. Which are seldom commercial ventures.
The economics are simple. I don't know why this even continues to be a debate on HN.
- A socially created entity (corporation, government, "charitable" organization, ...) needs money to function.
- Money comes from capturing a portion of the value created by the organization.
- The most efficient organizational structure depends on the relationship between value creation and value capture.
Thus: if the product/service generates immediate and focused value, the value capture can be directly linked to the product and a business is optimal. Think: a hamburger.
If the product/service generates long-term and diffuse value, the value capture also needs to be diffused, i.e. taxes. Thus a government. Think: the road network which allows the raw materials and the customers to get to the hamburger store.
I leave the case for charitable orgs as an exercise for the reader :)
disclaimer: strongly pro-business, have founded 2 personally, assisted several others.
Google started as a what now? This is an interesting thing to say during a discussion of access to information. I'd be happy to read your explanation of Google's "DARPA roots", and your citations to sources explaining how that came about and how they were "destroyed" when they "strayed" from DARPA... that should be a fascinating read.
https://qz.com/1145669/googles-true-origin-partly-lies-in-ci...
>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.
>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.
I refer you to some interesting discussion here:
Google Search is Dying https://news.ycombinator.com/item?id=30347719 1561 points
Every Google result now looks like an ad https://news.ycombinator.com/item?id=22107823 972 commments
Google no longer producing high quality search results in significant categories https://news.ycombinator.com/item?id=29772136 1275 comments
Reference: https://beta.nsf.gov/news/origins-google
And my main point is that government (funding) is the economically optimal approach for services which produce diffuse value.
See also the comment from marginella_nu in the post I linked to. Marginella is building a fantastic alternative search engine https://search.marginalia.nu/ Their view: "Arguably the biggest most unsolved problem in search is how to make a profit"
i.e., capturing the value produced.
kagi https://kagi.com/ attempts to do this with a paid tier. I hope it works for them, great product and really responsive team.
Bing tries to capture value by collecting the Microsoft tax; not exactly government-level, but on those lines.
Now everyone and their mom, as well as bots and marketers,can spam right on over.
We either need a search portal with aligned incentives, or perhaps a new internet with none of this crap.
Yes, people have gamified Google and search to get traffic and the blogosphere of old has largely been co-opted by profiteering gluttons, but there's still hope. Surf Hackernews enough and you'll find little gem posts that don't have an ulterior motive and are not 'monetizing' their content and sprinkling it with affiliate links and ADs. They just want to vent, exchange techne, and share knowledge.
Then there's Wikipedia which has remained AD-free for as long as I can remember (apart from their donation banners which I don't mind). Wikipedia is the coolest thing ever and my IQ has probably gone up a few notches over the years because of it. It is the closest thing to getting home-schooled without going through formal education, and you can verify all the claims made in its entries by going to the footer section and reading various citations usually written by esteemed scholars.
The web is in a sorry state due to the commercialization and walled garden silos, and also because of the proliferation of smartphones which are mere consumption devices IMHO and not designed for producing any meaningful or substantial content, apart from maybe uploading photos/videos to Instagram or writing tweets etc I can't write a blogpost on a phone because I have bad dexterity, and I typically have to have 100+ tabs open to verify claims, provide sources, do cross-referencing, find relevant images etc...all something done best on a workstation PC or laptop.
Some context: I have professionally blogged for more than five years but due to reasons I won't go into here, I have stopped. I'm thinking of jumping back in, only this time armed with the wisdom of my previous blogging shenanigans. Failure is an opportunity to start again more intelligently!
Is this why Microsoft products are getting progressively worse every year? How can you work on a product and then never use it natively and expect other people to enjoy using it? Or worse, like with windows 11, it leads to the product morphing into something the users never wanted. Because developers want to conform it to what they're used to using. I don't know, it kind of baffles me the way most developers view the products they make.
"As Director of the Office of Scientific Research and Development, Dr. Vannevar Bush has coordinated the activities of some six thousand leading American scientists in the application of science to warfare. In this significant article he holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge. For years inventions have extended man's physical powers rather than the powers of his mind. Trip hammers that multiply the fists, microscopes that sharpen the eye, and engines of destruction and detection are new results, but not the end results, of modern science. Now, says Dr. Bush, instruments are at hand which, if properly developed, will give man access to and command over the inherited knowledge of the ages. The perfection of these pacific instruments should be the first objective of our scientists as they emerge from their war work. Like Emerson's famous address of 1837 on "The American Scholar," this paper by Dr. Bush calls for a new relationship between thinking man and the sum of our knowledge. — THE EDITOR"
We don't have that, and it makes me sad. We should fix this inadequacy, but there are now so many interests in the entrenched model that I think they would squash something that freely allows copying like a bug.
[1] https://www.theatlantic.com/magazine/archive/1945/07/as-we-m...
[1*] https://web.archive.org/web/20220301051333/https://www.theat...
[edits - quoted editor's introduction, revised/extended]
What is missing is a beacon of light in the desert of choice. There is no lack of knowledge anymore but a lack of orientation. The author's father knew that he could buy knowledge in a bookstore. Where do we go to find orientation on the web?
People don't know. That's why they are stuck in the social net with its memes.
I avoid for-profit social networking websites because for the free flow of information because I realize that these sites only represent a small portion of what the web is about. I know the Internet is really best when I read from people that self-publish. I also publish articles on interpersonal work and the state of technology myself. As a "principal product manager in Microsoft working on tooling to enable people to do more on the web." I wonder what you think about Microsoft and Apple creating walled gardens in there respective OS's? I recently switched to Linux and [wrote](https://www.scottrlarson.com/publications/publication-transi...) about why I think the biggest threat to the free flowing information of the net has to do with how we allow our technology that connects to the net to become restricted.
Internet Archive which is a lot easier to search in than going manually through microfilm.
I'm a big fan of the Internet Archive and its mission. I'm concerned over Archive Today's severe lack of transparency (I've had a few exchanges with the site's operators, I've no idea who they are or what their motives are). I find the site useful but troubling.
I've also conducted microfilm (and -fiche) searches, as well as cataloguing of same. One affordance of a microform archive is that it is indexed, in ways that many online archives, including The Internet Archive ... aren't, usefully. The exchange is one of access-from-anywhere (yes, useful) against usable search and curation. It's ... an uncomfortable trade-off.
(The history of usefully cataloguing and indexing archives is itself a very old one. The US Librarian of Congress's annual letters to Congress, available through the otherwise almost wholly useless Hathi Trust, though, come to think of it, downloads of the entire letter rather than one single page at a time require using a different service ... are one interesting view to that process and the creation of the Library of Congress Classification and Subject Headings.)
The web is one of humanities greatest inventions, right up there with Gutenberg or even more. It's a shame that people focus on politics and miss the rest of the web. There are two webs, one is a medium of mass manipulation, just like all media before it, the other is the library of all human knowledge, ever, everywhere.
The printing press precipitated major societal changes, but the internet has yet to. We still follow the rules set 400 years ago, voting some humans to rule us every 4 years, and have not questioned the system under the current situation , where everyone can create and access information everywhere, instantly. The internet will take us to new politics , we just have to invent it.
"Most neuroses and some psychoses can be traced to the unnecessary and unhealthy habit of daily wallowing in the troubles and sins of five billion strangers."
Of course that's in a world where we still have personal computers, not apple style locked down devices.
I managed to salvage this one when my grand parents went away https://imgur.com/a/AFmTQv7
I don't think it's worth much but it's old and smells nice and feels good to handle.
The web is in a better state than it's ever been. Just between Wikipedia, Library Genesis and Sci-Hub alone, let alone things like Google Search, Google Scholar, or all the content put online from various museums/libraries/etc.
> On top of that drowning in memes, reposts and funny things you already read in newsgroups in 1998.
None of that takes away from all the good parts.
I honestly don't get any of these posts about how the web is so bad or so much worse than it used to be. The absolute amount of quality content has grown, seemingly exponentially. Even if the relative amount of quality has decreased... who even cares?! You don't have to visit the parts of the web you don't like.
And if you're complaining about reposts...? I have zero sympathy for you. Lots of people are seeing things for the first time. And it's nice to be reminded of things too.
It's simply impossible. It always irks me when people refer to the web as 'all human knowledge', because it's woefully incomplete if that is the goal.
Mind baffling that this article could end with this.
Once you read something totally unreasonable in a fact check, why should you go there as a source of truth anymore?
Wikipedia is probably a better bet - at least you have history of edits and sources.
An interesting observation, this is not the first time I've talked about this little test on HN, but this time it was not upvoted for.