retonato

Of course, you can just provide a code for users to run on their own computers, but don't expect that anyone will really use it (maybe just a few people here and there, I really mean it). Everyone, who is really hardcore enough to run something on their computers to obtain torrents will just use Jackett (https://github.com/Jackett/Jackett). It can search through the huge number of torrents, which no local DHT scraping/search engine can provide.

If despite all this you still decide to go forward - don't forget to import all the available torrent data on the internet (there are tens of dumps here and there), that way you will have MUCH larger database, than just by DHT scraping alone. This is a good place to start: https://archive.org/details/torrent_metadata_archive_sample

Creators of such sites (including me) tend to focus too much on the number of torrents, no matter if they are active or dead. Regular users are interested mainly in active torrents.

Plus they want to see the current number of seeders/leechers, which is very difficult to keep up-to-date for a large database.

Plus they want to see a torrent creation/upload date, which you cannot get from DHT (you can record the day you found a torrent, but it will work only for newer torrents, not for historical ones).

Don't underestimate possible legal problems. Tech people tend to simply throw them aside ("I will use a VPN/Cloudflare/Onion, no one will find me"), but here are my 5 cents:

1. Most likely - you site will not gain any significant popularity and you will decide to close it on your own. Meanwhile you will just be getting some automated DMCA emails/complaints from time to time, that's it.

2. Less likely - you site will be somewhat popular, it will become an attractive target for copyright owners and law firms (not as attractive as thepiratebay, but still). It usually takes just a month or two for some interested law enforcement agency to find out who you are (forget about all privacy protection methods you know, it is very easy to slip and eventually you will slip; you need to be lucky every time, they need to be lucky just once). Depending on what country you live, you will end up with a few thousand dollar fine (best case) or tens/hundreds of thousands + a small jail sentence + a criminal record. Not bad for a hobby torrent site with no income and a minimal community, is it?

Just to reiterate this one more time - it is VERY easy to find out who you are for any interested government agency [in your country], most of those 1-2 months will be spent waiting for replies for their letters and other similar bureaucracy, not for some CSI stuff.

Here is how it might go. First they will check the site domain and IP address and ask your hosting/domain providers to reveal you payment details (name/address on credit card). That's enough in 95% of all cases. Even if you use Cloudflare to hide your site IP and even if you use bitcoins to pay for everything - that will help only for some time, eventually Cloudflare will reveal your server IP and bitcoin transaction will be tracked back to your credit card. That's just a question of time and whether you are "wanted" enough or not.

I have been operating a BitTorrent search engine for a few years (it is closed now), here is some advice if you want to launch it as a public website:

1. There is no absence of people opening the kind of torrent indexers, which you have in mind. In 2010s there were 5-10 such sites launched each year, even nowadays there are at least a few new indexers yearly. Almost all are closed within a year or two (at most). There are reasons for that, some are less obvious than others.

2. Most likely you will close your site after some time as well, here is why:

2.1. It will be difficult to find a hosting, which will tolerate it. Forget about Linode or Digital Ocean or any similar hosting providers - they will suspend your server (first) and account (later) after receiving a few automated DCMA emails. You can use some "second-tier" providers for some time, but eventually even they will get tired of you, you will be out the moment they receive the first paper letter from some law company, which represents some movie company or alike. You can use proxy servers or some other arrangement for some time, but eventually you will decide that it isn't worth the effort.

2.2. The same thing is true for domain provider. Yes, there are a few, which are more resilient, than others, but don't expect that they will protect you and your domain indefinitely for $15 per year. If your site is popular enough - the only option may remain to register a new domain every few months and hop between them. Eventually you will get tired of that. It may take half a year or a few years, but you will.

2.3. There are no money in torrenting nowadays. Forget about ads or donations - even thepiratebay cannot earn money that way, you will not be able to do that either.

2.4. Sadly, torrents are not that popular nowadays as they were 5-10-15 years ago. Some people still use them, but in most developed countries that's more like an exception, than the rule. The rule is netflix, spotify and alike. As a result, if you imagine large and happy community of users around your site - just don't, most likely there won't be any.

2.5. Don't expect just to launch a site and rest. You will spend at least 5-10-20 hours per week on its maintenance (fixing bugs, importing/cleaning data, adding new features, etc). It will feel fine at first, but more and more tiring as the months and years go by. Eventually you will get bored and stop any maintenance. Users don't like old/unmaintained sites, so they will be less and less interested in it. Eventually you will decide, that it isn't worth the effort to run it at all.

Thanks again for all feedback. I added all recent feature suggestions on github (https://github.com/retonato/torrenthistory-public/issues). You can vote for them there (just give a thumbs up for the issue you like) and see the current priority here: https://torrenthistory.org/vote/

In my opinion even showing infohash would be illegal. Anything, which provides enough information for downloading by a generic torrent client.

Thank you, I will

In principle it is possible, but there would be some limitations:

1. Postgresql supports indexing to make regex searches faster from v9.3 (the current version is 13): https://www.postgresql.org/docs/12/pgtrgm.html . However, it makes them faster only in some (simpler) cases, in others we are back to plain old full table scan.

2. Speaking of full table scan, it is not that bad of an option, the average list of file paths for a torrent file is approximately 6 KB, that's 300 GB per 50 mln torrents, that's completely within reach of some VPS providers (like BuyVM). Still, up to a few minutes per one pass.

3. So, unless I find some efficient technology for regex search in large volumes of text, I would be able to implement only "offline" search (submit query, receive link to search results in 5-30 minutes, depending on server load).

4. Another option would be to provide listings of files for downloading as csv files (torrent_id, filepath) and let user to use command line for searching (or some text viewer/editor, though most popular ones still load all contents into ram). Compressed size would be around 1 KB per one torrent file, that's near 50 GB per 50m torrents.

What is BRE? Search by regular expressions?

At the very beginning (of January) search was done even through filenames, but then:

1. It turned out, that such search requires quite a lot of disk space for index. 2. It is not very useful without showing the actual matched filenames. This requires more code, I didn't have time to create it at the time.

I added it here for tracking, sooner or later that should be implemented: https://github.com/retonato/torrenthistory-public/issues/1

Sorry, I'm new to HN (as author). What do you mean by "pin"?

I thought about some kind of distributed storage, but the technology is not there yet (=I cannot implement that easily, users cannot use it easily). The site uses a regular PostgreSQL database. Most data, though, is published on Internet archive in a standard format (https://torrenthistory.org/docs/open-data/). There are torrents as well, users can download and seed them. I will upload new archives monthly

It would be interesting to see if that standard replaces Bittorent v1. For now it is not nearly as popular (everyone just uses v1, I don't know about any site, which has any significant number of v2 torrents).

I though about that, it is technically possible. More than that, it is relatively cheap (incoming traffic on most VPS is free). Still, I decided that searching by file size is much more cost effective. It is almost as good as searching by hash for larger files. It requires little to no code and time to implement. It is available even for torrents without seeds. It doesn't involve downloding files (which, in some people opinion, is illegal as such, even if you don't actually see the downloaded files).

That's a very good question. I think that dialog in filesharing/linking court cases has moved from "link vs torrent vs file" issue to "does this site/person make possible bypassing copyright protection, and, if so, to what extent?" issue.

Part of the torrent community still holds to the old (and comfortable, I must say) idea, that files and links (no matter in which form, http or magnet) are something fundamentally different. Files can be illegal, links cannot. Links are just text, infohashes are just numbers and so on.

The reality, as I see it, is different:

1. If something makes bypassing copyright protection as easy as making 2 mouse clicks and waiting a few minutes - that "something" is illegal, the person which operates it also commits a crime. Indirectly, by facilitation (=providing aid), but still.

2. Even if the law in this area is too old or too general to specifically mention such cases - the court practice is still little by little moving in that direction (=files and links doesn't matter, those are technical details; bypassing copyright - does matter)

Also, just to have a torrent indexer, which will be there forever. Other large indexers had almost 40-50 mln torrents, when they were closed, all that information disappeared almost overnight. In my case, the cost of running is very small, the legal threats shouldn't be the problem either (come on, it is really not possible to download anything from there). And even if something happens to me - others can continue where I left, since almost all site data is open, anyone can create a similar site in a month or so.

1. Checking if some torrent exists in principle (by searching for words in a name) 2. Searching for the same files in other torrents (when you click on the file size on some torrent page, it will show all other torrents, which contain files of the same size, it is pretty good method of searching for larger files) 3. Just as an open dataset, which contains a lot of torrent metadata and which you can use in your torrent related projects.

That's right, you can see torrent names and file names, your can search through them, but you cannot download anything. There are neither torrent files or infohashes there.

There have been quite a lot of torrent indexers already, but most of them are sooner or later closed. So, I decided to create one more, simple enough (so it can work on its own on a relatively cheap VPS) and legal (so it would be much more difficult to close it). Most of the site data can be freely downloaded. Is there something you think any decent torrent indexer should have?