Hacker News Reader

Comment by andrewstuart | Hacker News Reader

Who knows what websites and pages are even on the web?

There's no index to the web that I know of apart from Google and DuckDuckGo and maybe this Kagi thing.

I want to explore the web - surely search isn't the only way to use the web?

I imagine it could be fun to explore the web, lists and graphs of interest where I can hop from here to there via list of links or graphs or nodes or something?

Does anyone know of anything like this?

View on HN

There have been some websites in the past that allowed one to browse www content by IP address, covering what seemed to be the full range of IPv4 address space. For example, a page with a list of IP address ranges where each address range is a hyperlink. One could then drill down by following hyperlinks to a specific IP address and view whatever was hosted at that address (default host in the case of virtual hosting). Not sure why these websites do not persist. Quite useful. IMHO.

DNS zone files are a decent starting point for exploring the web. Not every registered domain name has an associated website but most do. The largest zone files are available to the public for free.

Organisating a directory of websites, or information about websites, by IP address/range is still common. For example, we can see news.ycombinator.com listed here:

https://www.robtex.com/cidr/209.216.230.0-24

https://www.robtex.com/ip-lookup/209.216.230.240

The above directory has been online for at least 18 years.

Here is one I can remember that disappeared:

https://web.archive.org/web/20090907060026if_/http://onsamei...

> view whatever was hosted at that address (default host in the case of virtual hosting). Not sure why these websites do not persist.

You probably answered your own question here--I imagine at this point the amount of things on the internet that aren't using some sort of virtual hosting are quite tiny.

Even ignoring everything else, with the IP address exhaustion and the push for SSL (can't get a certificate for an IP address), websites available directly at an IP address just aren't really a thing anymore.

While not quite what you're looking for, Kagi has a "Small Web" feed of sites that are semi-curated blogs. [1][2] I don't know how often it is updated, but I like to poke around every now and then see what's going on in people's corners of the internet.

[1] - https://blog.kagi.com/small-web [2] - https://kagi.com/smallweb

Google indexes widely. DuckDuckGo and Kagi have small specialised indexes and as such rely on the larger indexes like Google, Bing and Mojeek. DuckDuckGo used to use Yandex. More information here: https://www.searchenginemap.com/

Here’s the best list of search indexes I’ve come across

https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

Webrings?

There was an attempt to push a return of Webrings I think...

But funnily enough, whilst browsing this thread, https://news.ycombinator.com/item?id=41389642

I commented to my colleagues:

Remember when people would find good websites and share them!

Reply on news.ycombinator.com