Hacker News Reader

Comment by Kwpolska | Hacker News Reader

> TEXTFILES.COM has been online for nearly 25 years with no ads or clickthroughs. If you feel like donating to its roughly $1200 yearly upkeep: Paypal or Venmo.

Performance-wise, this doesn’t seem like a $100/month site, considering it’s only simple HTML and text files.

View on HN

As people are wildly speculating in this thread (it is Hackernews, after all), I'll just say that yes, more than you can imagine, my provider has been instrumental on the site staying up through a number of challenges and controversies and that number is my number to keep the site there with the features and arrangements I have. Regardless of what people think the Internet is like in 2024, there are vanishingly small places that will host a site like TEXTFILES.COM for very long.

    $ rsync --dry-run -a --stats rsync://rsync.textfiles.com/textfiles ./textfiles.com
    …
    Number of files: 866,935 (reg: 826,831, dir: 40,083, link: 21)
    …
    Total file size: 319,889,987,473 bytes
    …

Ah ha, that explains it. Thanks.

Even on Digital Ocean, that would be $50 for the 500 GB of block storage, another $20 for keeping a backup, and ~$6 for the actual droplet for hosting. And then some extra bandwidth costs, if people try to bulk-download. Adds up.

Just as a note, you're only getting some of the data that way. The contents of textfiles.com are roughly 2 terabytes of *.textfiles.com endeavors and another 13 terabytes of mirrors and hosting I do.

That site hosts a lot of historical material that many "budget" providers would probably balk at and kick you off their network without much recourse. Having a good relationship with a provider that knows you and will go to bat for you can cost some money. That said, $100/month is not terribly expensive for dedicated hosting.

Couldn't you just host all of these files on Github under the pretense of readme files? Would they throw a fit at this?

It’s probably outgoing bandwidth costs.

It probably gets mirrored a lit too, given the target audience.

As I recall, Jason asks people not to do wholesale site copies in general, but I imagine a great number do.

You imagine correctly. I cruise open directories regularly, usually looking for books to add to my growing collection (note: books I will actually read/use), generally taking only 2 or 3 books with me when I leave the site. At least once a month, I head to an OD I like to grab a new book or two only to find the host shut the gates because it got posted on r/opendirectories and a bunch of people did pointless site rips, as though they'll read 40k pdf files or whatever. I often wonder if they do it because data caches like that are small currency on other parts of the Internet, sort of the way you had to maintain an ul/dl ratio to stay on a well-maintained BBS back in the day. Who knows.

They're certainly not rehosting the libraries in any useful way, I can say that much.

I think a lot of it is just that many people have a pretty deeply ingrained data hoarding impulse where collecting files--any files--for themselves is the end goal.

I don't know if it's always hoarding. A lot of stuff on the internet simply disappears. Especially when it's hosted by a single person, or an org that might not be around in 3 years, or they get takedowns (whether legal or not).

If you find something you'd like to keep accessing, downloading it all in advance can be a smart move.

So download specific items as the parent says (and as you suggest). But reflexive "download it all just in case" probably isn't helpful especially if it's just for you.

I'm not disagreeing with downloading stuff that you want, even maybe site sub-sections. But a lot of the time it turns into just doing a mass download.

Given it's mostly text, it's curious there's no content-encoding applied to HTTP responses. Would reduce bandwidth by something like 70-90% in most cases.

Though it's hard to say what's the best configuration without understanding the hardware context.

Above it’s noted at being north of 300gb, most cheap cloud providers will give you quite a bit of bandwidth for a 300gb server.

Might be a point to self-host something like this. The value of the project is its extreme longevity, and the half-life of external hosting services is probably something like 5 years.

Reply on news.ycombinator.com