Hyperlink maximalism (2022)

If there is one thing the SEO and LLM crazes have taught us, it is that small curated beats big generated, every single time.

For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.

If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)

[0] https://twitter.com/justindsmith/status/1679978286955532296

If something is important enough, the author will discuss it directly in the document. Hyperlinks can provide context when the reader may not have the context assumed of the intended audience, or they're a poor man's bibliographic citation (because the links always break eventually), but otherwise it's a signal that it's _not_ important.

Even without the contextual search—which is initeresting mind you!—I want this part, and I’ve wanted it for a long long time.

I have it for journal articles simply by the virtue of obsessively saving and filing every PDF I’ve read for the last decade, but for the Web I don’t know how I could just plain index everything I’ve read (and not just everything in Pocket/Wallabag/etc). For example, is there anything that can pull links from Firefox Sync and save/index extracted text from them in a halfway reliable fashion? (I don’t expect it to be able to bypass yucky stuff like CAPTCHAs on archive.is.)

I know there are a handful of WARC-ecosystem tools, but AFAIU these are rarely that automatic, are tied to a browser plugin as opposed to just pulling the URLs from history sync (yes, I’m unreasonably fond of Epiphany/GNOME Web), and kind of heavy. Am I wrong? Any alternatives?

That said, I've occasionally written (or adapted) materials to include heavy hyperlinking. One example would be a bibliographic note (itself already a reference document), which I extended by linking names to Wikipedia entries and texts to their Worldcat entries:

<https://web.archive.org/web/20230607050023/https://old.reddi...>

One thing that made this easier is Wikipedia's convention for article links. I could auto-generate these when composing the document, then (reasonably quickly) verify the actual links were as expected. I think I'd used some shell functions to confirm Worldcat entries, which was possible at the time. A subsequent site degredation means that Worldcat is no longer usable at all without a JS-enabled browser, which w3m (my go-to terminal-mode browser) is unfortunately not.

Assuming that a maximal hypertextual interface needs to dynamically display relationships (rather than just a huge tag cloud or making all the body text hyperlink blue), a machine-readable characterization of the nature of those relationships would be invaluable. Does my text refute the linked resource? Or support it, or provide related factual statements, or is it chronologically or conceptually derivative? Some sort of ontological palette to write with could produce something that could be traversed in all sorts of novel ways. It would give writers more agency over external systems that will otherwise apply their own semantic meaning to the text, and its links.

Also, Hypertext (HTML, anyway) is oddly asymmetrical in its normal usage. We link a small string of text within a document, one-way, to the entirety an entire separate document. I think convention is the primary obstacle, but why can't we describe a relationship from our entire document to some other resource?

That isn't a good idea. For links to be purposeful they need to be high in signal and low in noise. If machines auto-link the word 'vision' in the quoted text above, I'll probably get a long list of optometrists. There's a reason the text itself has maybe half a dozen links in it, like " Andy and Michael’s excellent essay[...]". That's a specific piece if information that's relevant to the context.

Sparseness is a feature and not a bug when linking as you otherwise very quickly end up with a network where everything points to everything else, and that's meaningless. You don't want to 'tightly bind' everything together, because many of the things you read have nothing in common. In Schizophrenics, one important symptom is Apophenia, the tendency to see patterns in unrelated things. Hyperlink Maximalism is basically doing that to your text processing.

This would be really useful when reading research papers.

1. Most people who publish their writings online are too lazy to hyperlink, even where it is extremely appropriate. Even intra-site links.

2. Link rot is real. Maintainers of web sites are too arrogant to make redirects, and happily destroy all their incoming links when playing with a new framework. So now links from your site lead to 404s, because of somebody on the other end, and you have to scan your outgoing links every few months and change them.

In a way Mac OS and iOS have already implemented the "everything is a link"-approach, with their look up feature. It's much more powerful than it used to be, and you can select any word or words anywhere and find all kinds of information.

I plan to expand this to a general purpose web browser (or a kind of alternative companion reading-oriented browser).

https://reader.manabi.io

A popup still requires interaction and hides content. If you want really zero friction, then show the main content on one side (left, right or just center) and show details on hover (or click) on the side of it without reflow. On modern sites and displays there’s plenty of room for that at both sides.

I wish it was a standard element for all ui jobs, or one of available modes. Explorer - contents aside. Text - search/details. IDE - help and examples. SERP - site preview.

So I think we already have this feature. Now we just need better search-engines. Maybe AI supported ones? Wait we already have those too.

Making every word look like a hyperlink is a bad idea because often the user will want to search for a whole phrase, not just an individual word.

That seems like common sense, but most major wikipedia articles agree with the this piece's author more than me. When I open a big article, I would personally prefer it if 1/4 as much of the text is blue-underlined as what I find. Links should be used to bring the reader's attention to key information that is useful context but only expanded on elsewhere. Not any text that could be conceivably linked to some other random article. If I needed that, I would just manually search for the topic.

In a world of scarce and distinct links, my curiosity would draw me to click on many, and I would go down wormholes in the way tvtropes pulls of. But spammed fluff links are useless.

I know I am not articulating this the best. I am a visual learner and going through pages and pages in OneNote and cherrytree has been ineffective.

Note that this is very different from a hyperlink to some "related" content that leads you on a tangent.

Imagine reading a book (about something complicated, perhaps math) and immediately when you don't understand something you can tap into discussion/QA about the exact sentence you just read.

Of course it would only work for books that are read frequently enough to build that bank of context, but I imagine for many common topics it would work.

>almost nothing in the article is hyperlinked

What did he mean by this?

With the benefit of hindsight, I can't find many examples outside topics with clear hierarchies where this is better than relying on your mind. Too often I found myself linking instead of thinking.

The friction needed to search for a definition or context for me helps in forcing me to think about what a word may mean or the intent of a writer.

There should clearly be some low hanging fruit in revisiting the hypertext concept with the benefit of a few decades of further tech development.

Its not clear though which aspect(s) would be the most beneficial. Eg when scanning text for information and connections, our context is frequently an important factor, but that context is not available to the computer.

The way I currently do it is that first, I make hyperlinks stable by automatically snapshotting & making local archives of pages (https://gwern.net/archiving#preemptive-local-archiving). There is no point in adding links if linkrot discourages anyone from using them, of course, and I found that manual linkrot fixing did not scale to the amount of writing & hyperlinking I want to do.

The next step is adding links automatically. Particularly in the STEM topics I write most about these days, AI, there are many acronyms & named systems which mean specific things but it's easy to get lost in. Fortunately, that makes them easy to write automatic link rules for: https://github.com/gwern/gwern.net/blob/master/build/Config/... These run automatically on essay bodies when compiling the site, and on annotations when created. If a URL is already present, its rule doesn't run; and if it's not, only the first instance gets linked and the rest are skipped. (This is important: there are some approaches which take the lazy approach of hyperlinking every instance. This is bad and discredits linking.) This code is very slow but fast enough for static site building, anyway.

Sometimes terms are too ambiguous or too rare or too much work to write an explicit rewrite rule for. But it will still exist on-site. In fact, you can say that the site corpus defines a set of rewrite rules: everytime I write by hand `[foo](http://bar)`, am I not implicitly saying that there ought to be a rewrite rule for the string `foo` which ought to hyperlink `http://bar`? So there is a script (https://github.com/gwern/gwern.net/blob/master/build/link-su...) which will parse the site corpus, compile all the text/link pairs, create/remove a bunch of them per whitelist/blacklists and a frequency/length threshold, and then generate a bunch of Emacs Lisp pairs. This master list of rewrites then gets read by an Elisp snippet in my Emacs and turned into several thousand interactive search-and-replace commands when I run my generic formatting command on a buffer.

The effect of this second script is that after I have linked `Foo et al 2023` to `/doc/2023-foo.pdf` a few times (perhaps I went back and hyperlinked all instances of it after realizing it's an important paper), any future instances of 'Foo et al 2023' will pop up a search-and-replace asking to hyperlink it to `/doc/2023-foo.pdf`, and so on.

Third, I exploit my link-recommendations for manually-curated 'see also' sections appended to annotations. I have a fairly standard link-recommender approach where each annotation is embedded by a neural network (OA API for now), and one does nearest-neighbor lookups to find _n_ 'similar' annotations, and shows the reader them in case any are relevant. So far so good. But I also do that after editing each annotation: embed-recommend-list, and spits out a HTML list of the top 20 or so similar-links appended to the annotation. I can look at that and delete the irrelevant entries, or the entire list. This means that they'll be included in the final embedded version of the annotation, will show up in any fulltext searche I run, are more visible to the reader, can be edited into the main body if I want to, etc.

Fourth and most lately, I've been experimenting with GPT-4 for auto-formatting & auto-linking (https://github.com/gwern/gwern.net/blob/master/build/paragra...). GPT-4 has memorized many URLs, and where it hasn't, it still makes pretty good guesses. So, as part of the standard formatting passes, I pass annotations through GPT-4, with a bit added to its prompt, 'try to add useful hyperlinks to Wikipedia and other sources'. It often does, and it's quite convenient when that works. GPT-4 still confabulates URLs more often than I link, and sometimes hyperlinks too-obvious WP links and I have to delete them. So, still some adjustments required there.

And these work well with the other site features like recursive popups, or bidirectional backlinks (https://gwern.net/design#backlink).