Hacker News Reader

Comment by slazaro | Hacker News Reader

I keep wondering all the time why torrent search is based on websites (centralized), which can be taken down, etc., while once you have a torrent file or a magnet/hash everything is distributed.

Is there a main reason why there isn't (AFAIK, even though I haven't really researched) a distributed search that wouldn't have these problems? Is it a tech problem that literally can't be solved? Or it just hasn't been done? It seems like search is the obvious weak link, since the websites keep disappearing or taken down or blocked by governments and ISPs, etc.

View on HN

There is many search engine that use the DHT to retrieve the metadata and share torrent via the magnet. The main issue here is not the tech, it's the trust and discoverability/curating and also not spreading the seeding capabilities.

When downloading from "reputable torrent tracker XYZ" you can trust the quality of the torrent, that it is virus free, etc ... It is also usually make searching for particular torrent easier (less like searching for a nail in a hay stack) and you avoid spreading the seeding potential to hundreds of similar torrents.

As a extreme example, BakaBT (a private torrent for anime/manga related torrent) has a strict "no duplicate torrent" policy. This means that if you are searching for the OST of a specific show, you will have usually only one result and it's the most up-to-date, highest quality version. Since it is the only option, everyone seed this one. It really diminish the issue of abandoned torrents. To "replace" and existing torrent, you have to provide a strictly better version.

A decentralized torrent search engine could not do that. The real value of torrent tracker are the community.

That is also why decentralized software like eMule/eDonkey lost a lot of popularity to torrent tracker: Lots of duplication, very dodgy download, no curation, virus, ....

This makes sense but...I'm being fully serious and earnest here and revealing my own naïveté:

Couldn't some form of blockchain work here? Like couldn't some form of distributed/democratized community curation and moderation happen by using the blockchain to manage the arbitration of new torrents (and their successors, like when the community decides New Random Anime X encoding to be a superior copy)? Plus you have proof of stake or whatever the leading mechanism is to help combat and filter out fakes/illegal activity (etc)?

Then you'd have blockchain managing the trackers and torrents managing the file sharing.

How would they? Torrents are fairly trivial to build a web site around.

Blockchain sounds lile experimental engineering

I asked because the original question was around centralization (a tracker website) being the weak link in the chain. I suppose you could build a very simple decentralized voting system / index of sorts... but it wouldn't be a centralized hosted online "website" (..right?)

Tracker websites are hardly centralized. There are thousands (millions?) to choose from.

Bitcoin is going to be 15 years old soon and aside from this original usage, no real usage has really taken off for "blockchain" technologies (and no, I am not counting the occasional pump and dump / virtual scarcity scheme as a real usage of the tech). We really have to stop asking this question. And I say this having had some involvement with some web3 projects/company.

Even if it is theoretically possible, it create a huge barrier for entry, a lot of user friction, issue with governance and distribution of power, exploit, etc. And it is extremely hard to put in place for something that can be replaced by a generic phpBB forum in an afternoon. It is like trying to make a ICBM to kill a fly.

> no real usage has really taken off for "blockchain" technologies (and no, I am not counting the occasional pump and dump / virtual scarcity scheme as a real usage of the tech)

In the Bitcoin world, Ordinals have started to make waves as they are a creative use of unintended consequences. This will likely spur other innovations.

In the EVM world, DeFi is still used all the time. It lost a lot of steam from the last bull run, but trading, be your own bank, interest arbitrage, options, perps, all of that is still going on quite a bit. New sites are popping up all the time. LybraFinance LUSD/eUSD is one of the more recent ones.

Wow. DeFi? Be your own bank? Ordinals? No dear, we are here to hate and give our opinion on something we don't understand!

There are plenty of real uses for cryptocurrency that exist in the real world, but most of it is boring stuff: people holding their assets in stablecoins in countries that have capital controls.

This is an example of a real, yet illegal use of cryptocurrency (in the sense that it is used to circumvent the local laws). Also said cryptocurrency has to remain hidden like a treasure in a chest inside a secret cave, or it can be seized, and if the owner does not comply with the seizure, he can be jailed.

Yes, illegal with respect to the laws of the country, but there are many countries where most economic activity is underground/casual, not tracked by the state/outside of capital controls, and crypto is absolutely muscling into those spaces because it allows for complex financial infrastructure & also much better security compared to holding it.

Cryptocurrency is the only actual use-case for blockchain, all the rest can be done better with other technology.

And cryptocurrency is usually used for illegal stuff, so it is not necessarily desirable.

> Cryptocurrency is the only actual use-case for blockchain, all the rest can be done better with other technology.

How I can take out a collateralized loan, anywhere in the world, without having to ask for approval?

What is a use-case for a "collateralized loan"? Can you finance anything with it?

On the 'person supplying crypto for stablecoin' side, my opinion is it is not very useful. It allows you to keep crypto exposure while taking your cash out so you can make leveraged bets on crypto going up.

On the person 'supplying trust & owned property and getting a stablecoin loan', the other side of some Dai loans, there is real value being created there as they can use cheaper credit to invest in their business.

Most of the value goes to people in places like Argentina & Nigeria facing massive inflation who are now able to hold an asset that is backed to the dollar and overcollaterized by crypto, western bonds, & real properties held in the West, which is a better option than pretty much anything they can get in their home country.

Sorry, but something doesn't add up. How is "cheaper credit" created by a collateralized loan exactly?

If you put up your home or property as collateral, you can still own the property will getting lower rates. Similar thing with the Dai loans I am describing there.

Yes, but it's not quite the same, because both the collateral and the asset being borrowed belong in the same asset class. It's as if you're renting a car but you have to put up the same car that you're renting (only in a different colour) as collateral. It seems a little pointless. Usually, people want to rent a car because they don't have a car. Likewise, they want to borrow money, because they don't have money.

> It's as if you're renting a car but you have to put up the same car that you're renting (only in a different colour) as collateral.

Huh? That confused the heck out of me. How would you get the same car that you're renting in a different color?

The same model. Or a different model, but one that is worth the same in the market.

Wrong for two reasons:

1. There is no restriction that the asset being borrowed and the collateral are in the same asset class - loans can be extended based on whatever criteria the DAO in charge of the Dai votes on. There are a number of loans backed by real property investments in the US, see [0] under RWA. There are a number of loans also backed by large holdings in US bonds.

2. Dai is pegged to the dollar. It can be instantly exchanged with the dollar at a number of onramps. It is not equivalent to holding volatile crypto like ethereum.

[0]: daistats.com

Okay, these loans that are backed with real world assets are just traditional finance right? We were discussing loans that are granted automatically to anyone without requiring an approval, remember:

> How I can take out a collateralized loan, anywhere in the world, without having to ask for approval?

Is it traditional finance if it is a backing loan for dollar-pegged assets that can be held by anyone in the developed world (ie. rather than rely on your rapidly decreasing Naira - buy some Dai from a local dealer and you have a stable store of value backed by a global basket of assets)? And if the entities granting the loans are anonymous decentralized collectives?

I agree that the overcollateralized loans backed by crypto are less useful unless you want to be investing crypto, which is what I mentioned at the beginning of my comment. I am not the commentator who said the thing about anywhere in the world without approval.

A loan agreement that is enforced by courts of justice is traditional finance certainly.

Loans in decentralised finance use smart contracts, instead of relying on the legal system.

> Loans in decentralised finance use smart contracts, instead of relying on the legal system.

Code over law.

And what I was pointing out is that there is a use-case for crypto. DeFi.

A complaint about crypto is that it is too volatile to hold. If you can hold AND put at least some of it (up to your comfortable collateralization ratio) to work through lending it out (and borrowing against it), then that can offset the volatility.

What you do with the borrowed funds is up to you and there are an endless stream of possibilities. One example is to re-lend out again for additional interest. Interest arbitrage is a way to make your existing capital, work for you.

Lend BTC +1%, Borrow ETH -3%, Stake ETH +5%. 2% profit on the ETH and 1% on the BTC. Of course, you could then sell the profits for say, USDC, which you can then lend out again (~4%) or use to pay your bills or drug habit or whatever...

At the end of the day, this is what banks do with your funds. But instead of giving them the vast majority of the interest, why not take the interest yourself?

Larger picture, I personally see this as the opposite of credit cards, which are a blight on society [0]. People go into debt that they may or may not be able to pay off in the future. Why not reverse that and teach people to not spend more than they can pay off (collateralized loans)? You're punished with liquidations instead of stupid high interest rates.

[0] https://www.cnet.com/personal-finance/credit-cards/americans...

If the amount that you borrow is the same amount that you lend, both amounts cancel out. Your net investment position is zero. As I see it, you can't do anything you the borrowed funds because you haven't borrowed any funds. You have the same funds that you had before taking the "loan".

No. Here is an example:

I have 1 BTC, worth $27,000 today.

I lend that 1 BTC. I'm getting 1% on it. This is better than the pet rock that it otherwise is.

I can then borrow safely 50% of the value of that BTC, so $13,500 worth of goods. Now, I've magically given myself an additional $13,500, that I didn't have before.

The price of BTC would have to drop 50% before I'd be liquidated (some of my BTC would be automatically sold to cove the loan). Certainly, BTC can drop 50% in value... but if you are paying attention, you should have more than enough time to pay back that loan or add more BTC. In practice, BTC drops that much slowly over time (like a month or two), not in one go. It also works the other way, if BTC gains in value, you have less to worry about.

Let's say I pick to borrow ETH ($1800). I can now borrow 7.5 ETH at 3%. I can then lend that 7.5 ETH at 5%. I can then take those 5% earnings and sell those or even restake them.

Money legos. None of this requires permission or credit reports, just transactions on a blockchain. If you use a chain that has far lower fees (like Polygon, Arbitrium, Avalanche), then the fees are a rounding error in the cost of operations.

I don't think this is correct. Let's say you have one bitcoin. Your funds available for spending are $27,000 (we're assuming bitcoin has perfect liquidity, for the sake of the argument). If you borrow $13,500 against your bitcoin, your funds available for spending are now $13,500. So, less than what you started with. This isn't a loan. A loan increases your funds available for spending, not decreases them.

If you have $13,500 that you didn't have before, isn't that an increase?

I do see the distinction you're trying to make though. That's why I qualified it with the word "collaterlized".

Let me also correct you: "Let's say you have one bitcoin. Your funds available for spending are $27,000".

No, we're not talking about selling the bitcoin. There is no 'spending' here. What you're doing is borrowing against the value of the bitcoin that you're holding. You don't need to sell it, you're using it as collateral. The system can liquidate you.

You started with more than $13,500. You started with $27,000. Then you used those $27,000 as collateral to borrow $13,500. Once you pay back the $13,500, you get your $27,000 back. So, you never have more than $27,000 at your disposal, which is what you started with.

No, I started with 1 BTC.

Are you sure? Did it fall from the sky? No, it didn't. You saved up $27,000 and then you bought the bitcoin. So first you had to have $27,000, otherwise none of the subsequent steps would have been possible.

I think I spent $200 in 2013 on it. Now that pet rock is worth more and I'd like to capitalize on it, without spending it.

Just like people invest in a house and take equity out on it. Or they buy gold because they think it'll go up in value over time.

Regardless, you're being pedantic at this point. Back to the original discussion, there is utility to crypto now, and I've proven a way.

Yeah, I understand... these type of loans exist in finance, they serve a very specific purpose.

The point is that, unlike consumer loans, such loans require that the borrower has savings before they can take the loan. For example, most people who take a mortgage do so because they don't want to save money for 30 years before they can buy a house. They want to buy a house now, and save later. A mortgage loan allows them to do precisely that.

These other loans don't, and so they're less useful. They aren't completely useless. It's just that in most circumstances they aren't useful.

> The point is that, unlike consumer loans, such loans require that the borrower has savings before they can take the loan.

EXACTLY my point. If we actually turned lending and borrowing around so that you had to have collateral first (1), that could actually transform an entire economy, as well as people's minds away from spending what they don't have.

Credit card debt is at all time highs and they are raping people with high APY's on those funds.

(1) In SE Asia, they don't have the concept of credit reporting agencies and thus their 'credit cards' are all collateralized (what we refer to as debt cards). Living there is what turned me onto this concept and how it is a better way to do things. You can't spend more than what you've already deposited and you actually earn interest on it too!

That would be really interesting usage of blockchain!

You could probably accomplish as much just indexing torrents and combining it with an HN style voting system + user profiles. There’s always risk of bad actors but the main thing is having humans in the loop to provide reputation signals.

Cryptocurrency could be used here to both decentralize and incentivize such a system. I think a major hurdle would be the fact torrents have been "free" since their inception. PoS is irrelevant here - it's a mechanism that lives "higher up" to protect the entire cryptocurrency ecosystem. Then there's also the social issues and useability issues that plague the space.

My best guess is most bittorrent enthusiasts (and myself) would like to see a more natural solution to the decentralization problem. "Natural" being incentives which arise around sharing / not sharing and similar, vs straight up money incentives.

A reputation system could work but again, Sybil attacks could happen. So some how the network needs to figure out a way to make certain actions more expensive in the large.

Freenet works this way. I would replace all torrents with Freenet(s) but the technology seems to be too obscure to be comfortable for large audiences.

It's also not capable of handling this kind of volume.

IPFS might be a more modern alternative but it has similar issues. It's used for fairly small files such as books (libgen) but not really for movies.

Edit: Or do you man the torrent files only? That might work but they really are just links, it might be a bit overkill for them.

Not just the technology, if Freenet is still like it was when I tried it years ago (?), the "front page" is full of sites clearly being for pedophiles and fascists. Kind of tends to put you off, you know ?

This happened in the dutch usenet piracy community.

Downloading happened over usenet, but curation and discussion on a centralised website. The site got seized and the community moved to a new forum that runs on usenet itself.

Blockchain is overkill here - don't need a coin or stake or whatever

> Downloading happened over usenet, but curation and discussion on a centralised website. The site got seized and the community moved to a new forum that runs on usenet itself.

True, it worked really well for a while but lately it's been pretty terrible due to local copyright representatives suing major uploaders and others stopping because they just got old.

Also the notice & takedown has really killed usenet a bit. Private trackers are hard to get into just like with torrents.

It's therapeutic, people use whatever is available to them. Having a struggle at the edge allows people to do the busy work they bill for while not accomplishing anything meaningful. Technical folk will find new ways to keep their thing going. If to many websites are gone people will move to a new formula (new to them)

The network of sharing software and movies is much older than the internet. Eventually you just purchase a preloaded data carrier from your local pot dealer. The drives are so large, the formula would go dramatically faster than BitTorrent. Shipping [say] 10 kg worth of data carriers is amazingly cheap.

Looking at some random portable drive 10 kg box / 0.265 kg = 37 drives and 37 * 5 TB = 185 TB ? Something like 100 000 to 400 000 hours of film. Good for a maximum fine in "lost revenue" of $ 1 000 000 000 000 for the single box.

There is a lengthy therapeutic treatment program between that stage and the torrent websites.

Eventually some bean counter will discover crowd sourcing Police Academy 8 and people will just give them money provided they desire to see it. Star citizen raised over $569 million. On the most profitable movies list nr 191 is Fifty Shades of Grey with $569 million from a budget of 40 million. I can see the problem, with crowd sourcing it would be like normal work. That extra 500 million would be unlikely. They would have to make 10 movies.

What you propose sounds broadly like it could easily be built on IPFS and Filecoin. I think you find that there isn’t enough financial incentive to go decentralized here; groups get together and host/curate for community and kudos more than for profit.

> reputable torrent tracker

I once visited RarBG with uBlock Origin turned off by accident. The intensity of the shitstorm of fake links, transparent GIFs on top of the content and other stuff like that was overwhelming. I believe that it is universally stupid to trust torrent sites with anything.

The problem is not search, it's curation. Any system that accepts everything will be overrun with things you do not want (fakes, *very* illegal things), and so there needs to be some authority that determines what is allowed. At that point, you are centralized, and hosting a website doesn't significantly hurt you.

This is also the problem with all distributed social networks. In the end, your options are formal centralization, and informal centralization, because absolutely nobody wants to live in true decentralization.

> because absolutely nobody wants to live in true decentralization.

oh, some ppl do indeed want to.

thanks to encryption they can. but i agree that the likes of kazaa, edk and freenet are not for most ppl.

I seem to remember kazaa being extremely popular (especially after napster was gone) and moreover centralized. The client was local. gnutella sounds like what you're describing?

Most of the time when I search for something, I sort by more seeders first. Some type of signature can be used too.

> I sort by more seeders first.

But this couldn't be easier to game. All you have to do to be a seed is to report that you are a seed. You don't even have to send any data.

Sorting by number of seeders will probably get you a file that is fast to download because of the number of seeders, but it will rarely get you quality. Let's go take a stroll over to a popular public torrent site and search for, say, a recent superhero movie.

I see 100 (!) results for that movie's name. 49 of them have zero seeders at all. I don't know what even is the point. 29 of the results have one seeder. So already, 78% of the results are pure crap.

Let's look at the top result with 338 seeders: File is 3GB, H.264 video, 1080p, but with a crappy stereo AAC audio encoding... arrggh why??

Number 2 result with 84 seeders: 1.43GB, H.264 video, 720p, no word on the quality of the audio encoding. Even more worthless.

Number 3 result with 17 seeders: HEVC format, 2160p, audio streams include TrueHD Atmos 7.1, DTS-HD, Dolby Digital 5.1, stereo, and three non-English language streams. But, with an eye-watering download size of 61GB. Holy shit! Nice, but wow, what a download.

You have to go a few more down the list to find a good balance of high quality video and audio encoding, but with a reasonable file size. By that point you're in the single digit number of seeders.

Don't get me wrong, it's great to have a few choices and quality trade-offs. I guess there's someone out there who doesn't care about the stereo audio because they watch their movies with laptop speakers. But 100 results, with the vast majority of them either unseeded, poorly-seeded, or flawed in some way. I agree with OP: You definitely want some curation, not just search!

> arrggh why??

Because you are vastly overestimating how much most people actually notice or care about this.

I often pick the 0 seeders results if there is even a slight chance of better quality, and quite often there is.

0 seeders now doesn't mean forever. Some seeders only hook to the network once in a while, long enough for a few leechers to fetch it all. If the content is in demand enough then a few seeders may be left and suddenly a 0 seeder search result becomes an attractive one. Even curating is complicated for the human eye, so good luck automating that.

Yes it sometimes works but usually for smaller stuff like ebooks or music. You dont want to download gigabytes if not more from 1 spotty seeder, the chances it wont take days of 24h online and source being obline too are tiny.

Sometimes disconnect from site seeder count is bigger, and say 1 seeder mentioned is actually 15. But that was never rarbg's case, it was reliable and dependable like no other similar service. It also had IMBD rating so I could quickly weed out not so great stuff and focus on well rated ones, it generated maybe 60-80 movies a day with my filter applied, so quite a stream. After 2 week vacation, catching up took some time.

I used to, including today, to just go there every day or two and check whats new with my predefined filter. Often great movies that I never heard about before, old and new alike, took 2 mins to get 1080p x265 variant. Glad I've still managed to download that highly rated turkish movie this morning.

It is sad they didn't take the care to open source their stuff with whatever doc may be floating in their chats, for a resurrect

Great example. Some private trackers have strict rules about quality, e.g. at most one 720p and at most one 1080p (probably 1080p and 4k nowadays), bitrate must be above a certain threshold, whichever was published first stays, unless there is an "official" version published by the same people maintaining the website. The exact opposite of decentralized torrent sites, and works really well. Takes a lot of work, obviously.

I always assume that the movie studios themselves hired some proxies to put out those 61gb files so as to waste everyone's time and bandwidth.

This is a proven fact, do not be fooled. For decades Big Movie have been propagating disks out into the public with giant movie files on them.

Just like how Big Music decided to fight their pirates by only releasing unlistenable music for the last 20 years.

They are probably just ripped straight from a bluray without re-encoding.

> doesn't care about the stereo audio because they watch their movies with laptop speakers.

Stereo doesn't imply low fidelity. Just as 7.1 Dolby Unobtainium doesn't imply high fidelity speakers.

It's pretty much impossible to support Anonymous + Distributed + Free from poison/injection attacks. Not to mention, you still largely need at least some known points of entry into such a system.

Has anyone tried to decentralize curation?

Here is straw man proposal, similar to cert chains and webs of trust: Say I'm a "curator". I say on HN/Reddit/Discord "here is my key hash 'p2pcuration:185da2bc59167692f596404fd83235f9bcb4e107b041f2e6e8d972da6dba00b7'". Any user that clicks the link or copies it into the search app adds the key to the trusted user list. With my private key I can sign torrents after I download them myself, which would mark the torrent as "good". When anyone who has added my key searches, the system searches for a corresponding signature from me as well. If a signature is found, the UI can chose to elevate that result.

The system could be extended so that signers could also sign other keys, expanding the trust network.

This system doesn't need to be run or maintained by each user. It could be served through a webui that can be run locally or shared with a small community. Migrating the interface to a new host would just require moving the config and keys.

What happens when you do this is that a few or one people in the system become trusted above the others, network effects do their thing, and the whole thing becomes informally centralized around them.

Some examples trying to deal with this involving Mastodon (years before the Twitter Musk debacle) :

Mastodon is big in Japan, and the reason why is uncomfortable ["child porn"]

https://news.ycombinator.com/item?id=15053064

Mastodon and the challenges of abuse in a federated system

https://news.ycombinator.com/item?id=17894684

How the biggest decentralized social network is dealing with its ["]Nazi["] problem

https://news.ycombinator.com/item?id=20429465

Trump’s new social media platform found using Mastodon code

https://news.ycombinator.com/item?id=29079404

You can't build a decentralized search because there's no way to trust whatever results you get until you actually build it. If you don't want to rely on a specific community, your best bet is to crawl yourself and search locally: you can do that with magnetico (https://github.com/boramalper/magnetico/). Don't be frightened by the fact that it is archived, it works.

The problem then will be, how do you make sure your content is legit? There's no magic way here, the best thing you can do is compare the number of seeders and aim for the highest. If a torrent is fake, people will delete it and it won't be seeded. I have a thingy for that: https://sr.ht/~rakoo/magneticos/

The problem then becomes, number of seeders naturally selects towards popular content. It doesn't ensure viability of content. But I don't think there's a technical answer to that.

Very interesting, thanks for sharing your project. Magnetico is also integrated into Jackett I believe, which might be helpful to some people who use that

Decentralized search is a complex problem and there exist different approaches to it with different degree of centralization and resource requirements.

DC++ is a bit more decentralized than BitTorrent. There still are central servers ("hubs"), but they don't even host any metadata. Search works by the hub broadcasting all search queries to all online peers and them replying with results if they have any. The file transfers themselves are p2p.

I have an idea that's kind of more decentralized. Initially envisioned as a missing global search feature for the fediverse, but can be adapter for anything that has a similar network structure. A server has a number of peers already established because of the ActivityPub federation. Each server would send to its peers some kind of bloom filter that determines the tags or keywords that this server has results for. Then, when searching, your server would find the peers who are likely to have what you want, and only send your search query to them. If there aren't any, then it would send your query to the peers that have most users (with some random bias for load balancing purposes) because they're likely to have more connectivity, and they would point you where to look based on their own peers and their bloom filters. There would also need to be some kind of reputation system (centralized server lists? p2p exchange of scores/reports?) so that servers that return spam or intentionally wrong results would get punished.

This could probably be made to work in a fully-decentralized p2p network, but I imagine it would be too easy to abuse. Getting a new domain costs money, yet getting a new IP or public key is free and easy.

boredcaveman (at] tutanota d0t com

Mining the DHT for content isn’t particularly difficult, just time consuming and noisy.

For example, here is a personal DHT monitor where you can view what’s being announced on the DHT: https://github.com/retrohacker/taboo

That's a good question, but the answers are somewhat sad. I see the other commenters saying it's not about the search, but about the curation. Curation or identification (as in, "who's the author?") is essential, but decentralized search is non-existent too. Yes, there is unmaintained, resource-hungry, locally-run, unscalable software out there that you can use, but there really isn't a public search engine for this. Which is a shame. I really hope someone will tell me I missed something, but I'm not holding my breath :(

Edit: found mentions of https://btdig.com and https://bt4g.org. I wasn't aware of the latter. A problem with the former is that it doesn't track number of peers.

r/DataHoarder are sharing their backups of Magnet links and torrent files they backed up for years and are trying to bring the site back from the dead. For torrent sites, nothing stops you really from scraping the site for torrents. As long as the file itself is being seeded, it'll be impossible to take down. It IS decentralized.

How is it distributed? Don’t you need a torrent tracker? How do you know which peer has which files to share?

Here is a good resource https://wiki.theory.org/BitTorrentSpecification

You can easily decentralize search, but not fetching seed/leech counts essentially

Reply on news.ycombinator.com