BitTorrent v2
blog.libtorrent.org
blog.libtorrent.org
It works not in the sense that there is a white paper that should work. Not in the sense that there are a few company-made swarms hosted on industrial servers that keep everyone up and alive, so that the thing gives the impression the 'P2P' network does work. Not in the sense that there is a very-well oiled marketing machinery talking about Web 3.0 that allows its founder to go on TechCrunch and talk about the upcoming distributed paradise.
Bittorrent works in the way you install it into your computer and it does something for you. And for that alone, it has my immense respect and attention.
It's a tool that doesn't pitch that it's a P2P tool - it doesn't try to convince you with sob stories about how using P2P helps fight against the big bad evil web. Instead, you use it because it's genuinely the best at what it does: it being P2P is not a selling point, it's just how it happens to work, and that is exactly what it should be.
That is something all P2P developers should aspire towards.
[0] Aether P2P: https://getaether.net
What about the trackers?
BT only worked for new popular stuff.
This should be something that every creator who markets or sells products should learn. Consumers don't care how it works, it just has to be better than the solution before it. Appealing to how things ought to be moralistically doesn't work (for the vast majority of consumers). They just don't care, they want a solution to their problem.
Most developers, and therefore most projects, get stuck somewhere in the middle area, where there is enough skill to create features, but no direction or vision for those features, other than a basic template based on other software.
Often, the project will be called "minimal" to excuse having few features, but only a few projects adhere to a genuine restriction like SLOC limits or eliminating dependencies.
Appeals to commerce as the goal often have a further narrowing effect of making one anxious with thoughts like "project must moonshot on day 1", but then you look at Bittorrent, an all-time software success, and it's like, no, this probably wasn't ever going to create the next Microsoft. Funds were raised to have a Bittorrent company, but the market success it had was always modest at best. And yet it did and still does represent an idea that people can believe in.
The actual thing that I see every great project do, whether it's defined artistically or not, is to define up-front some themes and principles that you believe cohere well, and then direct the specific elements around deeply studying and exploring them. This can manifest as a corporate mission statement, or as an artist's manifesto, or an academic research subject. In final form, it generally manifests as "do one thing exceptionally well", since if you have a very clear idea of the goal, you can direct all your energy upon it. But in the intermediate stages it is still trial and error to learn what fits the theme best, what the actual success metrics are.
The thing is, if you have coherence in the abstract, it's way easier to explain the reason for being: "This is just a manifestation of the principles". Being easy to explain in turn makes it marketable without resorting to sales tricks. And because it deals with general concepts it taps into a breadth of appeal that can't be found just by looking at any one feature. So adhering to principle sets you up for success in a general sense.
I distinctly remember being a teenager and learning how bittorrent worked, thinking it's the coolest thing I'd heard of and laboriously explaining how it worked to a friend that sounded interested. At the end of the explanation he asked "So how do you download music with it?"
And I realized all he cared about was using it to pirate music.
It's really easy for programmers to get lost in how cool some code or technology is and lose sight of how the rest of the world views it.
Story of my fucking life.
I’ve given my friends the spiel about how cool it is that I can use the EM spectrum to send my voice through the air to the other side of town. And how other people have put up repeaters which mean my voice can actually go a lot further than that —- maybe across the country!
And their response? “Sure, but I can already do that on my phone”
After losing a lot of my curiosity the past few years, it’s coming back and my confidence is growing and I want to explore various fields and technologies deeper [and for the first time].
Do you think people don’t know about this beautiful stuff because inventions and products have become overly corporatized and use intellectual property and patents, which stifles curiosity?
I can’t help but dream of a fully open source world where we don’t dominate each other by keeping information about the inner workings of things artificially scarce like we do today - as evidenced by Aaron Swartz, Library Genesis, corporate fight back against right to repair etc.
I share it occasionally with friends to remind them of this. Computers are just tools. The really interesting question is what those tools can do for us.
I can’t remember who aid it originally, but there’s a great test you can give to any statement or idea. Just immediately ask “who cares?” The best product demos I’ve ever seen answer who cares in each part of the pitch. The worst ones just rattle off mumbo jumbo forever.
Who cares? Anyone who feels like their purchasing power isn't where it should be is either underpaid, or a victim of central banks diluting their dollars with printing.
So, nobody in the EU or North America, then.
https://www.cbc.ca/news/business/cyprus-bank-account-tax-put...
(Also, the grandparent post perhaps should have given some context to the statement "Lightning made scaling possible", such as the fact that Bitcoin originally supported 32 MB blocks of transactions, before being "temporarily" limited to 1 MB. Alternative blockchains have managed without this artificial restriction, and without relying on complicated second-layer "solutions" like Lightning.)
European leaders wanted to limit the size of the rescue loans — which are backed by European taxpayers — to €10 billion. Leaders were also reluctant to bail out Russian depositors whose funds may be the result of tax evasion, crime or money laundering.
Additionally, there are sanctions in place since the 2014 Crimean crisis.
https://www.themoscowtimes.com/2019/01/10/cyprus-no-longer-m...
In the US, stocks are 187.8% market cap to GDP.
But please, tell us more about how there's no inflation.
As I understand I need to do something that's pretty much exactly like opening a bank account, but transactions take forever, they cost money, and the currency is not accepted by any business I use, unlike my free credit card.
I have 55€ in my wallet and everyone on this continent knows what to do with it.
They: What is this?
Me: Money.
They: No, bring money.
Sometimes you have to transfer money in and out of the country at war with it’s currency in free fall and capital controls in place. At times like that “normal” ways take about 30% of the sum as transactional overhead, while bitcoin doesn’t.
Sometimes “normal“ ways just don’t work normally. Sometimes your government is actively working against your ability to transfer your money for whatever reason — be it drug laws or capital controls or what not. Sometimes you enter account number into the “normal“ system and field just turns red for no reason at all. Sometimes you have your access to “normal“ system severely restricted because of your legal status and residency rights.
So yeah transferring money between Canada and US being resident of either of them and not doing anything funny — is not a use case for Bitcoin.
Seriously in reality the specifics of any sort of wild situation like the one you describe matter. Which countries? Which currency? Which kind of illegal behavior?
And (I ask out of ignorance) in this scenario you describe why would whatever group managing whatever bitcoin exchange or whatever is being used remain virtuous and not charge a 30% markup themselves?
It beggars belief that someone would so callously call the situation in these countries 'hypothetical'. Please, inform yourself about the war and strife that is occurring outside your own bubble. Its very important that we in the West stop ignoring the plight of the people our military industrial complex is targeting with their weapons.
Bitcoin has held things together in many such places.
Also: Russia.
User adamsea is informing himself, that's why he's asking a question here.
Just because people are starving behind economic blockades and sanctions and mafia hell, doesn't mean they don't RTFM.
(Oh, and you can use BTC on the street, all over the world.)
Shouldn't we amend that statement to be "You can use bitcoin to buy anything which is being sold by some other bitcoin user?" ;)
Which sure, is potentially anything/anyone, but will obviously vary on circumstances.
Bitcoin users are a tiny, tiny percentage of "anyone".
WikiLeaks may have amassed more than $46 million in Bitcoin based on the number of coins held by its known wallet address. https://bitcoinist.com/wikileaks-has-received-more-than-46-m...
That indeed is a bunch of real situations that have happened to me in 2014-2015 and some of that is still a thing. Check your privileges, sweet summer child.
Like seriously — for the first 30 years of my life it was outright illegal to have a bank account in a foreign country in my name. Not enforceable in practice, except I can’t make wires between two.
So 2014, fucking Russia is being Russia again, USD wired from foreign clients goes in... and is sold for UAH at government mandated rate. I sit in a third country, withdraw it through ATM. Guess what? It is converted back to USD at a market rate. Of course you can still bring cash or btc in and sell it at market rate plus some margin, but not 30 damn percent.
Sometimes it’s less dramatic and web interface of your bank doesn’t have regexp for IBAN format of destination country. Ridiculous but have happened as well, so I have proxied IBAN that starts with GB instead of UA and that always works.
Please don't do this here. It is needlessly condescending and insulting..
"Does this hypothetical situation you describe exist anywhere outside of the hypothetical situation posited by a character in a Neal Stephenson novel"
But, like, the classic "society could collapse" fear-and-doubt, doom-and-gloom, grab-the-gold-and-the-burner-laptop-and-the-ammo argument can be used for just about anything.
Heck, even when society actually is collapsing, the argument isn't actually that great an argument!
I'm not even arguing that cryptocurrency couldn't have value in the scenario you describe. Just that genuine, specific, examples are more compelling than generalized scenarios. Generally speaking ; )
For example, all of the more specific examples folks talk about below (Argentina, etc).
Plus the parent poster made fair points, based on my limited knowledge, about accountability issues with cryptocurrency. And trust. Especially for nontechnical folk, and even for technical folk. Even (especially!) when society is collapsing, there's always someone out to scam you ...
For arguments sake, if someone is in the scenario you describe, what are good initial resources for a nontechnical person? For instance, should I use something like a coinbase wallet?
https://www.reddit.com/r/BitcoinBeginners/comments/fdykz3/is...
Then again people who I talked to in 2014, were still a bit in denial till the start of school year and only then left the area directly affected by war.
My point is simple — you take the ability to access banking system for granted, when it’s far from universal. It’s also not as unrestricted and interconnected as bitcoin or internet itself. Sometimes you have a bank and a card, but your ability to transact is restricted. It doesn’t always mean that you sit in the basement hiding from daily artillery strike or wandering post-nuclear desert on a cool bike.
I guess I was more pointing out that right now it seems highly scenario-specific whether or not cryptocurrency can help in the way you described.
Doesn't it depend on the amount of money you're dealing with, what you may or may not need to do with said money in the near-future, your own technical competence/facility, and perhaps even the current state of cryptocurrency itself?
You just need to be in a situation where the law prohibits, or somehow restricts or severely taxes, the money transfer itself, or the use of that money to procure something you need.
Sometimes crypto is the only way of sending/receiving money from other countries reliably and cheaply.
Other countries are much worse, Venezuela for instance has destroyed the value of their money, so bad that people use it wallpaper.
I heard there was a similar problem in Egypt, no idea if and how it was solved.
https://en.wikipedia.org/wiki/2012%E2%80%932013_Cypriot_fina...
Bitcoin was the backstop that saved my ass when I went to SE Asia and none of my first-class Australian or US bank cards would let me get cash.
In the US perhaps. In at least one other country however...
https://www.bbc.com/news/business-47553048
https://www.somagnews.com/bitcoin-becomes-the-dominant-curre...
So in my view, it's pretty nice as an alternative to holding gold. Gold can be nice as well of course, but it's difficult to transport. If you keep gold in your home, at some point someone might break into your home and steal your gold. You could pay some company to store your gold, but perhaps in times of crises a government might step in and claim some of your gold.
Crypto currencies can be easily held in a digital wallet or people can even just memorise the keys to their wallet in their brain. It's hard for governments to control crypto currencies and I'm sure some of the wealthiest people in the world now keep a certain small percentage of their wealth in cryptos as well as a hedge against fiat currencies, stock market crashes, etc...
---
[0]: https://news.bitcoin.com/venezuela-bitcoin-use-hyperinflatio...
[1]: https://www.financemagnates.com/cryptocurrency/the-impact-of...
Not really. If I want a bank account I need to fill out a page of paperwork requiring my name, address, social security number, citizenship information, income, occupation, date of birth, scans of various pieces of ID and/or other documents. Compare this to bitcoin where you only need to install an app, and new identities can be generated on-demand. It's true that trying to buy/sell bitcoins via an exchange will subject you to AML requirements, but in-person transactions won't.
>transactions take forever, they cost money [...] unlike my free credit card.
The cost and time it takes is mostly a function of how much demand is there for blockchain space. During periods of high transaction volume, you'll either have to pay high fees or wait a long time. You can see in this chart https://jochen-hoenicke.de/queue/#0,30d that there are periods of very high transaction fees (the yellow peaks correspond to a $3 fee for a typical transaction), but also periods with low transaction fees (the parts with blue troughs correspond to a $0.12 fee for a typical transaction). People in a hurry pay more and people who can wait pay less. It's a classic microeconomics.
Credit cards aren't free either. The cost (around 2-3% in the US) is borne by the merchant. Cash has fees for handling, transportation, and storage. "Free" bank transfers (eg. vemo, SEPA) are typically restricted to consumers, are often part of a bigger banking package, which have fees or minimum balance requirements.
Nobody does bitcoin transactions in person, at least not in America and not anyone I know. I have several bitcoins and the only way to use them is through an exchange like coinbase, who require the same identity documents that a bank does.
You could argue “but you just have to find vendors who accept bitcoin!” But with 3 bitcoin I won’t find enough vendors to spend it on.
>You could argue “but you just have to find vendors who accept bitcoin!” But with 3 bitcoin I won’t find enough vendors to spend it on.
There are also bitcoin-for-gift-card stores, which vastly expands your vendor options.
In EU, regulation caps interchange fees at 0.2% of the transaction value for debit cards and at 0.3% for credit cards.
> "Free" bank transfers (eg. vemo, SEPA) are typically restricted to consumers, are often part of a bigger banking package, which have fees or minimum balance requirement
A bank that i use has free SEPA transfers, no montly fees and negligible minimum balance reqirements, both for consumer and business accounts.
Same for Bitcoin, you install a wallet app and use it. It's actually much easier than opening a bank account.
> Meanwhile normal people without a serious networking/distributed systems background will not really grasp what they are doing
They won't grasp what they're doing on their bank account without a serious economic/financial/law background either.
You can already purchase Reddit MOONS from the r/cryptocurrency subreddit here: http://xmoon.exchange/
So do stamps, coins, Star Wars figurines in original packaging, baseball cards, yada yada.
At least you can look at those other things while you possess them.
I saw Bitcoin project on GitHub's artic code vault[1] and advertised heavily. But I couldn't find libbitorrent on it besides it being hosted on GitHub as well. Is this because the project owner hasn't enabled it or is there some other reason behind it?
but I think you do. You find a seller who is willing to accept the currency you're using, and you use whatever interface to the bitcoin network you've chosen to do "send x money to y identifier". No different to a regular EFT.
You identified the exact problems. It's not that bitcoin transfers are some weird foreign concept, it's that they take forever, have high fees and aren't accepted universally.
The major users of bitcoin as an actual currency rather than a speculative vehicle are places that can't use the more convenient forms of currency. Guaranteed they'd rather use USD if they could.
Well that's a non-starter. If I wanted to buy a beer with bitcoin right now, I'd probably end up drinking tap water.
However if you wanted to purchase "illegal" drugs online you'd be hard pressed to find anyone who will accept your USD.
That doesn't make it useless for someone else, who might for example live in Argentina.
The day you want to hire someone who wants their payment in Bitcoin for some reason, that's when you'll have use for it.
You don't have to go anywhere, you don't have to provide any kind of ID or documentation, you don't have to wait around for a free salesperson.
> but transactions take forever
Bitcoin transactions are instantaneous. Look up "settlement cycle" if you want to make a direct comparison. Bitcoin final settlement usually takes about an hour, compared to anywhere from several days to several months for traditional payment. Credit card settlement is usually 60-90 days.
> they cost money
It costs less than a dollar to move ~any amount of money anywhere in the world with ~1hr settlement. A credit card usually costs something like $0.25 + 3%.
> the currency is not accepted by any business I use... I have 55€ in my wallet and everyone on this continent knows what to do with it.
I have AU$80 in my wallet and I'm pretty sure no one on your continent would take it.
In fact, last time I traveled to SE Asia none of my debit cards worked for cash withdrawal and the only way I was able to get cash was with a Bitcoin ATM. So it worked very well for me!
That said, unfortunately, short of a large nation state, or collection of banks also allowing exchange/transactions, I don't think it will ever take off. Even then, transaction times as you mention are not fast, and often are controlled by a limited number of organizations.
What the technology does offer, is a means for multiple parties that don't trust each other to proceed with a transaction. This is often where "smart contracts" come into play, and also why banks are considering a cryptocurrency system(s) for inter-bank exchanges.
In the end, those that win will largely be the incumbents imho.
I setup bitcoin on a few machines really early on... and had a couple coins... but I had no way to "use" them and it was just costing electricity at the time... I deleted it all and didn't look back until they cleared $20k a coin. Kind of wish I at least saved a zip file of the wallets somewhere. Though if I'd kept mining, may have been worth something.
The largely obviates the need for any domain knowledge.
There's great progress being made, but we'll not get there during 2020.
https://www.reuters.com/article/us-crypto-currencies-africa-...
Adoption will establish itself in Africa, S.America, Asia first.. and then it will compete head on with traditional western banking services.
In the case of Bittorrent they should know a little bit about how it works before they start pirating software, music or movies using it, however.
Back when people left things downloading for days, there were severe shortages of people willing to upload, and tit-for-tat encouraged uploading, solving the issue.
Bittorrent only became that level of magical for me when the DHT got factored in some time later and magnet links became workable.
The other time was Bitcoin.
I’m a big fan of lack of centralized/coordination/tracker nodes.
Crypto currencies owe a lot of their successes to the pioneers behind technologies that also power bit torrent.
- Patricia-Merkle Trees proven in DC++
- DHT proven in eMule Kademlia
The block header has the merkle root of all transactions that are a part of that block. The witness merkle root is stored in the coinbase transaction (if the miner is segwit enabled).
And proof of work is done for the block header, which includes all these merkle roots.
There are some alternative cryptosystem designs that do take blocks in "losing chains" into consideration using a DAG structure, like GHOST and its successor SPECTRE (by Aviv Zohar et. al). Ethereum also has a concept of "uncle blocks", which are rewarded and contribute to chain selection.
That's pretty much the definition of "rebase" and again, that's a functionality of the algorithm on top of the data structure (Proof of Work) not the data structure. The raw data structure is still a merkle tree even if in practice the algorithm suggests to people there is only one rebased trunk. But even that isn't entirely true in practice because there are still multiple rebased "branches" among the Bitcoin forks such as Bitcoin Classic, Bitcoin Gold, etc. All of those are branches that share the same conceptual merkle tree. Even if they aren't "Bitcoin" that's more of an algorithmic and political distinction at that point, not a technical one by means of data structure. It's not the data structure that makes it a chain, it's the algorithm and the politics, hence why I think blockchain is a misnomer for the data structure itself.
A merkle tree is a very specific type of hash tree, which Bitcoin only uses for transactions and not for blocks. Merkle proofs are used to prove that a txid exists within the root hash committed in the header block. What would be the reason to organize blocks into a merkle tree? What would that let you prove?
See this SE question for more information on how Bitcoin uses merkle trees: https://bitcoin.stackexchange.com/questions/69018/merkle-roo...
> here are still multiple rebased "branches" among the Bitcoin forks such as Bitcoin Classic, Bitcoin Gold, etc. All of those are branches that share...
Bitcoin, BCash and BGold each have incompatible rule sets; A full node will only accept chains that are valid according to its own local set of rules (embedded in it software), so chains of different coins will not even be considered for chain selection, regardless of the proof-of-work backing them. They just don't exists from the full node's PoV. Validity of blocks/transactions comes first, everything else is second.
If light SPV clients weren't a consideration, we could just concatenate all txids together and use the hash of that in the block header instead of a merkle root, and get the same effect.
What merkle trees give you is an efficient way to prove that a certain txid is committed to within a block, without the verifier having to fetch the full list of txids. Instead, he just needs a valid merkle path from the txid to the root, which is much smaller to communicate and to store.
For a full node that has the full list of txids regardless, this is basically meaningless. Full nodes don't (ever) verify merkle inclusion proofs, only that the merkle root in the header matches the full list of block txids.
I would still consider Satoshi's invention to be an incredible breakthrough even if he didn't consider light SPV clients since day one and only described the full node operation mode, therefore I don't consider SPV to be a core component of the Bitcoin breakthrough.
(And also, we know today that SPV is not as great as it was once hoped to be. It puts users at the whims of the miners, with XT/Classic/Unlimited/S2X/BCash being marvelous examples of how that can go terribly wrong. The fraud proof concept that Satoshi described in the whitepaper as part of the SPV model (under the name "alerts") was discovered to not actually be workable due to the data withhold problem, giving this model much weaker security guarantees. And privacy is totally and utterly broken in traditional SPV -- though Neutrino is making good progress on that front.)
<shesek> does bitcoin core ever verify merkle inclusion proofs? (I assume not, it only verifies that the merkle root matches the set of txids. but maybe I'm missing some other ways its being used?)
<sipa> i don't think anything verifies them
<sipa> shesek: they don't even ever receive any
<sipa> though they were an essential part of BIP37 [related to light SPV clients]
<phantomcircuit> shesek, for a full node theres no real difference between receiving a merkle tree and a hash of a list
<sipa> yeah, for a full-blocks-only bitcoin like protocol, the "merkle root" stored in the block header could just be a flat hash of all txids
It is kind of a selling point with regards to what most people actually use it for. If not P2P the movie / record industries would shut down the servers.
It is the software I download for free that is the most reliable IME. Spending more money does not make today's consumer computers any more valuable. And that's how it should be. The price of hardware (and software) should continue to fall.
The other big thing was seeding the already downloaded bits while downloading the rest and making this behavior very hard to disable. This combined with ratio systems of the various trackers ensured that people played nice and gave back as much as they took.
The downside was the elimination of the long tail very niche content that you could find on DC++ while browsing individual people's messy but unique collections.
Setting the precedent that media companies can seize domain names and force ISPs to block access to search engines if they don't like some of the search results (while ruining the lives of the people running those search engines) is arguably a high price to pay for this improved search service.
The main reward for torrent uploaders is fame, would that translate well in IPFS?
A few years ago I was very interested in IPFS but once I learned about the limitations it was pretty much not suitable for me. If you really insist on using IPFS then your best bet would be to have a central tracker host an IPNS based site and publish its database index as a series of json files. The javascript client will then have to query the index on its own in the browser.
The alternative would be to download the entire database and create a local index. In theory that's not a bad idea but over time your database will grow to several gigabytes. That's not comparable to just going to whatever site exist today and submitting a search term but it could be highly resilient.
The fundamental problem is that untrusted nodes can't provide search services. It's possible that they end up redirecting you to a fake listing. Blockchain style consensus doesn't work for something that requires responses immediately like a search engine.
Also bittorrent trackers are slightly more complicated than just a file store
Uploading became a privilege, something that normal average people don't do. Earlier people ripped their own CDs, DVDs, magazine scans, digitized their VHS, recorded shows with their TV card etc. and shared these files. I don't know anybody who has ever created a torrent themselves. Everybody just downloads and seeds the pre-packaged stuff from the centralized sites, much like the walled garden philosophy of app stores.
But your experiences sound like what goes on in the public trackers. It's reasonable that folks who go to Pirate Bay to get their TV shows are not going to be as engaged as those who go to TVV, MS, or MTV. There's also the security risks of letting untrusted users upload files that may get consumed by thousands of people in the first hour.
In private trackers, I have often found that uploading is encouraged, going so far as to provide very detailed guides in their wikis showing how to rip the media, prepare the torrent, and upload it to the tracker. If a better quality version can be uploaded to trump the previous version, that is also encouraged. Learning, quality, and appreciation are cornerstones of the private tracker ecosystem, in my experience.
For that, you want Direct Connect [0]. As a one sentence pitch: IRC for file sharing ;)
I like how the data expiration is set as an intentional feature.
I wanted to try Aether, but I avoid Snap as much as possible
A few years ago, that was precisely their marketing pitch.
Bitcoin is kinda like San Francisco. They decided to stick with the current block size and that made any use case other than buying bitcoin to speculate very expensive.
Quite a few projects are using P2P, machine learning, Kubernetes, No-Sql or block-chain not because the project actually requires scalability, decentralization, advanced data analysis or responsive big data, but because somebody got excited and refused to accept that it was a waste of time and money to do so.
https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
> Your typical architecture astronaut will take a fact like “Napster is a peer-to-peer service for downloading music” and ignore everything but the architecture, thinking it’s interesting because it’s peer to peer, completely missing the point that it’s interesting because you can type the name of a song and listen to it right away.
> [...]
> If Napster wasn’t peer-to-peer but it did let you type the name of a song and then listen to it, it would have been just as popular.
> Files that are identical can also more easily be identified across different swarms, since their root hash only depends on the content of the file.
Wait, content addressed blobs across swarms... does that mean torrents made by completely different people at different times that happen to contain one or more identical files can benefit from each other's peers? If so this feels like a significant feature that would boost the long term health of a lot of torrents and connect more peers that could be helping each other.
Content addressing is one of the big advantages of p2p applications, and IPFS has been pushing it for a long time.
Just imagine if this was done all the way down to the piece level. You have two different torrents (say, Linux ISOs) where 20% of the pieces overlap due to similarity (maybe a point upgrade or something and you want both versions). Rather than 100% of both, you only need to download the shared pieces once. Not only that, but say that the latest version has many more seeds/peers, you could download the pieces from that swarm instead, saving the bandwidth of the older torrent's swarm for the remaining 80% you need to download.
This could probably be done client side somehow, but it would be good to see actual protocol support for it so that bittorent can move in that direction.
I don't know about that. With a reasonably large number of users, wouldn't you start seeing hash collisions via birthday paradox? Might be better off keeping it at the file level. (Though this does incentivize malicious users to find hash collisions of particular files they want to defend, and seeding the swarm with garbage files)
Does "get a collision" here mean finding a particular collision with a known hash, or finding any collision between any two of the generated hashes?
https://github.blog/2017-03-20-sha-1-collision-detection-on-...
sha-256 was chosen because it provides a sufficiently high number of different hashes that this won't be an issue for the forseeable future.
more details here: https://stackoverflow.com/questions/4014090/is-it-safe-to-ig...
also need to watch out for compression
> Identical files will always have the same hash and can more easily be moved from one torrent to another (when creating torrents) without having to re-hash anything.
The idea is to make blocks (slightly) variable in size. Block boundaries are determined based on a limited window of preceding bytes. This way a change in one location will only have a limited impact on the following blocks.
There isn't a way to advertise some "rolling hash value" in a way that allows other people with a differently-aligned copy to notice that you and them have some duplicated byte ranges.
Rolling hashes only work when one person (or two people engaged in a conversation, like rsync) already has both copies.
The rolling hash is used to find the chunk boundary: Hash a window before every byte (which is cheap with a rolling hash) and compare it against a defined bit mask. For example: Check if the first 20 bytes are zero. If so, you'd get chunks with about 2^20 bytes (1 MiB) average length.
As a good explanation, I'd encourage you to look at borgbackup's internals documentation: https://borgbackup.readthedocs.io/en/stable/internals.html
If I discover that the file I want to publish shares a range with an existing file, that does very little because the existing file has already chosen its chunk boundaries and I can’t influence those. That ship has sailed.
I can only benefit if the a priori chunks are small enough that some subset of the identified match is still addressable. And then I may only get half of a two thirds of the improvement I was after.
If they both used the same rolling hash function on the same or similar data, regardless of the initial and final boundary and regardless of when they chose the boundaries, they will share many chunks with high probability. That’s just how splitting with rolling hashes work. They produce variable-length chunks.
Yes, this won't be useful for speed.
But it will be useful for health and appearance of health.
So a torrent that has no seeds because it's missing "RARBG_DO_NOT_MIRROR.exe" can be filled in quickly and have proper seed info.
Also large seasons can be filled in. They get corrupted at later episodes because people will prioritise the first episodes then the seeds can jump.
So anyone filling in the later episodes through single episodes torrents might repair the season torrent.
It'll allow over lap between torrents only differentiated be a missing RARBG.txt if they are both fragmented.
There's always been talk of allowing a torrent to have files added to it. It's be interesting to see if this might be useful client side to do this in a Claytons way.
Personally I don't think it's useful for Linux isos. Not sure on this. Do Linux isos get unhealthy?
but it would take some handcrafting to make sure the new files are at the end of the ISO so everything else stays in place.
technically this should be possible since ISO should allow for adding files given that it was designed for write-once media.
Yes. It also makes your files more discoverable for copyright trolls. They will be able to fully automate the process of sending takedown notices for each copy of zlib's README.txt, that was "stolen" from their software by evil pirates.
Except for so many that are a single archive file. It seems logical that you'd want to compress what you're sending as much as possible before uploading... but in today's bandwidth environment, it actually makes sense to leave certain things uncompressed so you can benefit from swarm overlaps?!? Wild!
It's not uncommon to see multiple .torrent files with the same content FYI.
[0] https://github.com/picotorrent/picotorrent/releases/tag/v0.2...
I suggest you test with Narrator rather than NVDA. Disclosure: I work on the Narrator team at Microsoft. But that's not why I say this. The reason is that NVDA and JAWS have some ugly hacks that they can use to make some GUI implementations (particularly using Win32 with GDI for graphics) accessible even if they're not accessible by design. For details, do some searching on the term "off-screen model". Narrator doesn't have this, so if your product works with Narrator, you know it's really accessible.
I want to ask, were Rasterbar-Libtorrent and Libtorrent always the same thing? I thought they were different implementation? Did they merge or did memory serve me wrong.
I couldn't Google anything useful so I just ask. In the era of streaming It has been far too long since I look at anything BT.
Personally I understand the economics around it, but still don't like the idea of paying per GB. I would end up skipping some Netflix and would get mad at kids for playing Netflix to an empty room (much like I currently get mad when they leave the lights on in an empty room). It's nice on a personal level to avoid that.
To use the power analogy, when I was young the price of power was relatively high but in recent times it has now dropped to 0.6 NOK / kWh so even charging the car from 10-80% (~61 kWh) cost 40 NOK or $4. I do not get mad at the kids leaving the light on like my mother did to me.
I (or rather my job) currently pays about 1000 NOK/ month for unlimited 500 MBit synchronous fiber internet. I transfer maybe 500 GB/month so for it to be cheaper for me it would be under 2 NOK/GB or $.2 which sounds really high but that is what I currently pay with my usage.
Is it a normal connection or a business connection perhaps?
Your peering will only scale to your (combined) interface speed regardless of your SLA. Our 2x 40G peer with level 3 serve far more than 40 1G devices (I personally have 600 in one building alone, and that sets aside the rest of the users), but it’s rare it’s more than 30% utilised.
Clearly that’s not going to be a domestic isp architecture, so a reasonable question is what does uncontended actually mean.
I very much doubt an isp with 10,000 customers has 10TB of peering physically available. Linx public peering is less than half that for the entire UK, and while private peering will likely increase that, the suggest is it’s welll under tenfold, so the 50 million plus internet users in the UK only use at peak times 1M each, and that ignores all the non-domestic use.
Even a 100:1 contention ratio seems enough at a core level, so any isp spending money on improving on 10:1 ratios seems that they are just burning cash.
Clearly as you go the the edge contention ratio needs to drop - but 40G, or maybe even 20G uplink for 48x1G users would be reasonable to me.
Wow really interesting to see such high (assume Mbps) bandwidth guarantee for consumer service. Is the guarantee really work?
the price was competitive too. other ISPs charged $70 per month, and had a limit of 50gb (that was 15 years ago) my ISP charged $20 as base fee and $1 per gb of usage. if i used less than 50gb, i saved money, if i used more then i could...
But I think it would considerably change the Internet landscape. No more listening to the same exact song multiple times on Youtube or Spotify or whatever. No more downloading then deleting the same stuff over and over again. I think about it often, what a massive waste of bandwidth, I don't even know why. It's a lot of electricity used, I guess?
Even though I've got unlimited fiber, I'm looking for some sort of local "Internet cache" solution that would store everything so it would be re-downloaded from my home instead of across the ocean. Would be great for outages, too.
Is it though? How much electricity is used to serve a 1080p Netflix movie several times vs. playing the same movie from a local storage medium?
If you asked me, I'd rather burn Bitcoin mining rigs if I wanted to get rid of electricity waste.
Pretty sure I use more than that in a day.
Seriously, why would you use that much data per day for? Even getting full blu-rays you wouldn't be able to watch that many.
95/8 = 11.875 (Mbit to MByte)
11.875 * 3600 = 42750 (MByte/s to MByte/h)
42750 / 1000 = 42.750 (MByte/g to GByte/h)
42.750 * 24 = 1026 (GByte/h to GByte/d)
1026 / 1000 = 1.026 (GByte/d to TByte/d)
In a two month period I uploaded about 50TB according to the tracker so my calculations seem about right.
It's easy to reach these numbers in a country where net neutrality is a thing.
Luckily it was over fiber and through a corporate VPN and I barely noticed it. But I was shocked when I saw the usage on my router later that month.
that's because you benefit from it. You're not currently paying the cost of that 1080p stream - netflix (and the ISP/peering networks) are paying. Netflix recoups the cost from your subscription, and the ISP/peering arrangement is mostly cost neutral to them (save for a small amount i presume).
But it's still not "free".
Unless that's what you meant...
Just open another tab; problem solved. :|
I would bet this is already the most efficient way of it working.
You can setup a proxy service to cache internet requests too, but they're getting less useful due to greater use of HTTPS.
Because they can get away with it.
Why wouldn’t you want to pool bandwidth with your neighbors so you could all get faster speeds when you were using it instead of rate limiting everyone?
It's gigabit speeds until all your neighbours suddenly want to download something too.
So, the defense of using a misnomer to name your service is because the service warranted is actually impossible to supply given the margins?
Call me a fool, but that still seems like a company that is getting away with lying to the vast majority of people that don't bother reading the asterisk ( like T-mobile style "Unlimited" plan that gives X amount unthrottled, and some arbitrarily low rate after).
Criminal issue? Of course not, that's why the companies present such things this way.
Dishonest? You bet.
I can't sell you a pony made out of diamonds because that's impossible. Consequently, there is no legitimate reason for me to be advertising the sale of a diamond pony!
- Does this guarantee that you get the speed that you pay for? If not, then should the price during heavy congestion cost less per GB? I’ve had the unfortunate luck of having lived in areas where the internet is unusable for anything other than email and light browsing during peak hours. Bonus: Do you know why this happens? If not, I encourage you to research this.
- How will this affect the advertising model of the entire internet economy? If you, like me, have lived your whole life on a mobile data plan that is priced based on data usage, you’ll realize just how much bandwidth is consumed by advertisements (worst are the video/audio types that auto play).
- How will this affect “future” technology such at smart homes? Most devices send data back (Amazon Echo devices) and the growing security camera ecosystem that sends recordings to the cloud would suddenly cost users a ton more.
- Speaking of cloud, what will happen to the gaming industry that is heavily cloud based? Many games don’t even ship in a completed state anymore. Instead, part of the install requires downloading a ton of additional updates. Not to mention online play, DLC, etc.
There a many many more everyday examples like this that would be heavily affected by pricing per GB. Your question sounds like a simple solution but like many things in life, that is rarely the case.
Furthermore, even in places where you do have some sort of choice for internet service, it is almost certainly limited to choosing between a cable monopoly and a phone monopoly as your ISP.
When Xtra came out at $2.50 per hour, it was a complete game changer.
It changed our internet browsing habits so much. With hourly charges, you would try to plan out what you would do before connecting and instead of reading webpages, you would save them to disk for reading later.
With unlimited, you could just sit there and browse. Or leave it on overnight to download files.
At some point we got a second phoneline and I was downloading torrents on dialup all day and night for years before we finally moved somewhere with ADSL in 2006.
Looking at pricing for Spark (one of the largerproviders). [1]
For each of the fibre speed plans you can get 60GB/month, for another $10 you get 120GB/month and for a further $10 you get unlimited traffic.
BTW: "Mb" = Megabits, "MB" = MegaBytes
[1] https://www.spark.co.nz/shop/internet/plans-and-pricing/
I've used 15 TB in the last 5 weeks with my torrent client alone (and considering my Backblaze backup size, that's not nearly all of my traffic), but how many people like me are there in the general public?
In addition, it gives people a solid measure over which to sue you. If you aren't delivering, you're headed to court.
The ISPs like all the vagueness.
The word 'responsibility' that you used is of course too strong a word, however the comment 'it would be nice' to have is definitely true assuming we want downloads to be faster.
The hash it relies on is broken, so their hand has been forced.
Second preimage attacks are MUCH harder to pull off, even md5 is still safe from them, many years after they were found to be broken in other contexts.
The only thing that sha1's weakness would let you do in a bittorrent context is create a torrent, then let you send fake data for a chunk, which really does not seem very useful, because you could just make the data malicious when you created the torrent.
Certainly not a trivial attack, but not benign either.
You'd want to look like some other protocol and you want that protocol be encrypted by default. Otherwise yours will get fingerprinted via the deep packet inspection.
The most obvious choice is to run your protocol over TLS. But then they can just throttle long-lived bulky TLS connections where neither side is on 443.
You can then require the responding side be on 443 (which is already a big hassle), but they will then throttle down TLS connections towards residential IPs or throttle them down cumulatively, as a group.
Other choices here are OpenVPN, WireGuard and, possibly, IPsec. But, again, it will come down to defeating the throttling of multiple inbound bulky connections of the same type, which is doubly hard to bypass if you are on a residential IP and your ISP is really bent on throttling.
Any successful obfuscation technique is short-lived, so it has no place in the protocol itself. Instead it should be delegated to a transport layer and the clients should be coded to support BT tunneling over X or over Y.
Fundamentally, nothing has the same characteristics as BitTorrent - almost nothing has the same high uplink requirements, which is an absolute signature of BitTorrent traffic.
Oh, also some of the shaping is implicit, not explicit. Most home internet connections are asymmetric - they dedicate more channels to downlink than uplink.
Streaming video, whether that's a Skype conversation or showing your gameplay to Twitch, can be a pretty bulky upbound stream.
I wonder how fingerprintable those are, and if they have diverse enough endpoints to be able to disguise other traffic as them.
or, get a better ISP? not all technical problems need a technical solution.
A better solution would be to have rules to make that kind of throttling illegal, but we've seen how that played out.
Which is mainly because we have privatized roads here in the USA.
Yes, you read that right. The "gold standard" is underground fiber, which lies in the public right-of-way yet is 100% privately owned with no "duty to serve" like the electric utility has. Oftentimes the fiber owner doesn't even have to dig up the dirt -- if they lay fiber along a newly built highway the government does all the digging for them (to create the roadside drainage ditch). The phone company just unreels a spool of armored OS2 into the ditch and hey, it's Miller Time.
Until privatized right-of-way stops we will continue to get screwed.
[*] Transcontinental railroad right-of-ways are the exception to the above, but there's only four of them. And, frankly, that land was privatized through outright fraud. Read the book _Railroaded_ sometime, it's shocking.
This allows ISPs to throttle uploads with very low risk of pissing off the bulk of their customer base when their detection algorithm gives a false positive.
The real question is whether they care enough about torrents to do anything about it.
https://iknowwhatyoudownload.com/en/stat/US/daily
Given that 0.31% of internet users torrent, maybe?
Do you have a source on this? I believe you I just want to know more. It explains a lot.
I have a gigabit link. I can download torrents at almost line speed. But I can barely get uploads past 5K/s. I spent hours at one point trying to tweak every possible setting and eliminate every bottleneck, and still couldn't get past 5K/s.
There is little to know, it's largely a commercial preference (that has, over the years, driven technological research; see ADSL for example, the first A is for Asymmetric).
Average consumers are precisely that, consumers: they rarely upload anything, but they download tons of content (from webpages to streamed media). So it makes sense for residential ISPs to maximize downstream bandwidth, since it's what consumers will evaluate them on. One way to do that is to simply throttle upstream, to ensure resources stay available for downstream. (This has the side benefit of reducing headaches, i.e. less people distributing questionable material on your network...).
If you need good upload speeds, you simply have to talk to your ISP.
Something else is the router AT&T requires will not completely open ports for you. What it does is it is closed for incoming connections that do not hit it multiple times then it opens up, so I appear like my ports are not open on AT&T. But even when I get around that (you can rip the certificates off of the router) AT&T itself limits the connection. If you transfer too much on one port on the backend they will remotely block the port sometimes. (Transferring too much being very little here, around 50MB, maybe even 10MB.)
Good news is Comcast does not pull this crap and has fiber internet. Bad news is they start at 2gbps and start at $210 a month in most areas. I'm paying $60 for AT&T gigabit.
I just got it last week, coming from the slowest ADSL option available, it blows my mind.
In other words, how would you authenticate peers? How would you prevent ISPs from mitming your encrypted p2p connections if you have no authentication for them, and you in principle can't have any?
If you want QUIC's congestion controller benefits then you can also get that with TCP if you're on linux. Set BBR as congestion controller and set the TCP_NOTSENT_LOWAT socket option and bittorrent should work well over long fat pipes (i.e. international peers).
I'm not saying QUIC is terrible, but it's mostly designed to improve web traffic. Bulk transfers à la bittorrent benefit only marginally if you have a modern TCP stack and use it properly.
Yes, it'd be a minor suggestion at best. The biggest benefit would be to the internet as whole, motivating middle box makers into not breaking it and so on.
The low-latency idea seems interesting, I suspect WebTorrent people would appreciate it. But live streams over P2P sound too fancy for now.
> but bittorrent-over-TLS would work equally well (not standardized, but libtorrent supports it).
Which is where the suggestion to use QUIC would have the biggest effect and getting the swarms more encrypted.
> Bulk transfers à la bittorrent benefit only marginally if you have a modern TCP stack and use it properly.
Unfortunately very little software actually goes into the effort at figuring each such detail out and the end result, defaults are often rather poor.
I guess this more of a questions of "what are the defaults" not "what's possible."
The article states that hybrid torrents that support v1 and v2 can be created. But what does this mean for end-users (clients)?
Will most torrent software be upgraded to support both v1 and v2? And will a client be forced to choose from the v1 or v2 swarm, or will it be able to download from and seed to both?
I mean it seems like clients would participate in both swarms -- I'd just like to know if that's confirmed.
Also, is is possible to add v2 to existing torrents "retroactively"? Who would do that? Or would this solely be for new torrents moving forwards?
The main issue that I see is the existence of millions of torrents in private trackers that would have to be manually updated for v2.
Are people going to bother? I think not. So for me, v2 is practically a new-torrents-only affair.
Private trackers tend to gamify quite a few aspects of their sites (obvious example being the seed ratio itself).
You would need the file data to do that, so torrent indexing sites could not do it. And since you need the full data you could only do it after downloading at which point it doesn't add that much value.
> Or would this solely be for new torrents moving forwards?
Indeed. v2 offers a few nice improvements but not world-changing ones. So there's no pressure to upgrade existing torrents. They'll keep working as-is.
Both are not based on Libtorrent.
I'm not even convinced they couldn't have done it in a backwards compatible way. Why not stick with SHA-1 but also add SHA-256 for verification for clients that support it?
Apparently you can do that with 'hybrid' magnet links.
The problem is the two swarms are different so as more people move over to v2 the v1 swarms will become smaller and smaller -- thus giving people an incentive to upgrade, I suppose.
...and they seem to have solved more than one minor issue since they were breaking things anyway...
[edit]
That doesn't imply that BTv2 is the right successor protocol.
Also, this seems quite bad: https://en.wikipedia.org/wiki/Collision_attack#Chosen-prefix...
Huh, why?
That is a much more serious weakness called a "second pre-image attack"
This class of attacks is MUCH harder to construct against a cryptographic hash.
Creating a SHA-1 collision is doable, but it's still hard. If you want to serve someone a malicious piece of data, that's already one hash of the two colliding hashes that you've used up. Now you have to create harmless or "benevolent" data that collides with the hash of your malicious data so that you can create a positive reputation for your file from users who aren't your targets. That way, when your target inevitably goes to download the file, you wrestle into the protocol with a lot of speed and/or nodes, and you serve the malicious data to your target instead of the data you've been serving to everyone else.
If you don't need the positive reputation, and someone will just download and run whatever you put in the torrent, you don't need the collision in the first place.
Most Linux distros offer an installation iso via torrent, large files with many blocks. If you can change just a small part of those files, you’ve got compromised machines before the install even begins.
(That is, we've got practical collision attacks emerging for SHA1, not pre-image attacks).
1. The user trusts the source of the .torrent file.
2. A malicious peer makes a preimage attack in some block in an executable file with contents containing some malicious executable payload.
3. The executable wasn't signed, or the targeted block must include executable headers.
4. Some peers get the malicious exe, some don't.
The (2) step is still hard — preimage attacks on SHA1 are still expensive.
And it is probably much easier to bypass SHA1/SHA256 entirely by just uploading a malicious torrent directly and hoping (1) still applies.
Even MD5, for which you can generate colliding blocks in seconds on an average PC, is still quite resistant to preimage attacks.
In other words, even after spending the resources to find a colliding block, you'd also need to create both files with the same hash, and can't simply collide existing torrents' files and replace them with malicious ones.
Since there's some control over the original hash value, executing step (4) is not exactly a preimage attack, it should be a bit easier.
So your scenario becomes an issue if you're downloading executable data that you deem :trusted: without any additional verification besides the hash itself. If that's the case, you have bigger worries than the hash.
So if you get your hash from whoever made the original binary, you can know that any binary with a matching hash is fine. But it could be a problem if someone creates a new pair of binaries and uploads the hash to TPB. You might see lots of good reviews from people who got the innocent version, but then the peers you connect to send you a malicious version with a matching hash.
No, "they" don't. SHA1 collisions had been "in the wind" for a while, they had been in sight ever since MD5 started showing signs of clear weakness in the early '00s. Wikipedia has a Rivest quote about it from 2005. There is nothing like that for SHA2, although attacks are improving.
> What are the chances of a collision if we simultaneously use multiple hashes
Define "simultaneous". Shipping twice the hashes for each piece seems a big waste of space. If you mean re-hashing hashes, it's just a waste of cpu power, since an attacker only has to break one or the other to get in a position to poison data.
This is an understandable reaction, but the security margin on new crypto is way higher than old crypto. Roughly speaking, we went from "I guess if a state-level actor dedicated all their resources to this for a few decades, they could probably brute-force it" to "Even if you broke 9 out of 10 rounds in this algorithm, you'd still need to harness the energy of every star in the universe for 10 billion years to brute-force it."
Most algorithms today have been "attacked" in the sense that there are tricks we can do that allow us to recover the key faster than a simple brute-force attack. But "faster" usually means doing something like 2^100 operations instead of 2^128 -- still far beyond the realm of practicality.
It's telling that cryptographers are now seriously discussing reducing the security of various algorithms: https://eprint.iacr.org/2019/1492
Very unlikely that this is going to happen any time soon.
Most modern symmetric cryptographic primitives with sizes >=256 bits are considered safe even against quantum computers. SHA256 turned out to be even stronger than expected. SHA-3 adoption is delayed in many protocols because there is no much need for it and hw implementations for SHA256 are commonplace.
Without the ability to support multiple versions, it would be impossible to upgrade anything at all. That would be a whole load of other problems.
> Downgrade attacks have been a consistent problem with the SSL/TLS family of protocols; examples of such attacks include the POODLE attack.
On Ice Lake, where BLAKE3 benefits from AVX-512 and SHA-256 benefits from the SHA extensions, BLAKE3 seems to do better on both long and short messages. But maybe surprisingly, SHA-256 does better in a medium-length regime, where SHA-256's poorer startup time* has been mostly amortized out, but BLAKE3's chunk parallelism hasn't yet kicked in. See for example the 1536-byte results here: https://bench.cr.yp.to/results-hash.html#amd64-icelake . Using multithreading would exaggerate BLAKE3's advantage for very long messages (usually about 1 MiB and above), but it wouldn't improve the results for any of the message lengths measured there.
* I don't actually know where SHA-256's startup overhead comes from. Maybe someone who knows more could jump in here?
On ARM chips, the performance benefits of NEON are less dramatic than AVX-512, and the performance advantage of SHA-256 hardware acceleration is comparatively larger. I think it's rare for BLAKE3 to beat accelerated SHA-256 on ARM without at least some multithreading, but I've only personally benchmarked a few Raspberry Pis, and I want to be careful not to overgeneralize.
... but for consistency (like their narrowing of valid bencode), they’ve presumably chosen one main hash for now, so that every client and server doesn’t have to handle all of these cases as people provide a million variants of the same torrent.
This is huge. Now the protocol supports downloading identical files from multiple torrents simultaneously.
In the wild west of the internet "update" really only means "add" because you don't want the source you barely trust to provide some data to issue an update that deletes all the previous download from that source. But you also want to avoid wasting storage so some size caps and rehashing old data to see if it's an incremental update will also be needed.
Bram Cohen developed BitTorrent and released a (Python) reference implementation in the public domain (later under MIT and then GPL licenses); he later founded BitTorrent Inc. and assigned this implementation to the company to maintain. Eventually BitTorrent Inc dropped this codebase altogether and became closed-source with a completely separate project.
Libtorrent is just one of many independent BT libraries that have been developed since Bram published the first BT client.
??
So there are no relationship between the two?
>Eventually BitTorrent Inc dropped this codebase altogether and became closed-source with a completely separate project.
That was from the acquisition of utorrent.
Rather ironically, you get a much more readable view if you select "Desktop site" in the Chrome mobile menu and then double-tap the main text column.
Most of the time it's not web sites but browsers, who are unfriendly.
I'm asking, because this would allow for sharing of big files across swarms, even though the files might be slightly different.
This seems like a really cool approach that could be applied lots of other places where versioning can be tricky. But probably also hard to come up with new, good names.
The amount of effort into backward compat and keeping the general UX the same (ie: no overly-large new Magnet urls) is really appreciated.
Hopefully this will find quick adoption in both clients and submissions.
As a question for someone who knows better - does this mean that you can download single files within torrents themselves? I imagine if each file is independently hashed, this should be possible.
Most torrent clients can do this already, but depending on how the torrent was created and the size the files involved you might download a some of the files on either side.
1. the new hash function is going to be broken eventually - what happens then? 2. support for "remixes". It would be nice to reference pieces from another torrent. Example use case: adding subtitles for a movie. Right now it requires either downloading the "main" version of the movie and getting the subtitles externally, or sharing the file from scratch.
SHA-2 will not be broken as easily as SHA-1. This seems to be a common misconception in this thread.
Wikipedia: "Since 2005, SHA-1 has not been considered secure against well-funded opponents".
This was 10 years after its introduction (1995) and 15 years ago. We had 15 friggin' years and we're finally switching git and bittorrent around. It took 2017-2005=12 years to get from first serious signs of issues to https://shattered.io.
SHA-2 is now 19 years old and the "uh oh, better switch before it's too late" recommendation has not come yet. It's withstanding the test of time better and there haven't been 12 years to mature any weaknesses. The theoretically known attacks for SHA-2 are fairly insignificant.
Since SHA-2 had a similar construction to SHA-1 (Merkle-Dåmgard), NIST figured they better launch the SHA-3 competition at the first sign of trouble and picked something with a very different operating principle. For now, however, it's still fine. Thomas Pornin put this a bit better than I can: https://security.stackexchange.com/a/21116/10863
As for "what happens if/when it will be broken": we could make BitTorrentv2 another multi-crypto soup like with TLS, but then you open up a can of downgrade attacks, potential null ciphers or other such tricks simply due to increased complexity, and you still can't switch that quickly because everyone needs to take manual action in changing configuration files. Much better if we can instead do apt upgrade and let the software take care of making security decisions rather than those who install the software. (Remember that SSL was designed in 1994, when LiveScript/JavaScript didn't even exist yet, DES was state of the art, and we wrote books with algorithms because cryptography was ammunition. Having multiple options for strong/weak ciphers was not yet a crazy idea.)
Not trying to be rude – I found the blog post rather informative and concise!
I believe it uses libtorrent under the hood, so this might be integrated soon.
https://github.com/qbittorrent/qBittorrent/issues/4965
The only difference between qbit's pause and uTorrent's stop is the check if files still exist.
Users of open source software are free to fork the software if the original developer isn't responsive to their particular needs. They're not entitled to demand that the original developers respond to their feedback. They're even less entitled to complain when the develop does respond with a specific reason why they won't act on their feedback.
https://bugs.chromium.org/p/project-zero/issues/detail?id=15...
uTorrent seems to have some efficiency problems, and in my experience it can't even do 1 Gbps with a cross-over cable between two peers, let alone across the Internet...
- BiglyBT sometimes, because of Swarm Merging
As for how this works with private trackers is a different story. I guess since DHT isn't used for private torrents, this shouldn't be an issue.
It works exceedingly well. I've never had it falter or slow, despite the number of torrents. My preferred client for desktop and server :)
https://twitter.com/markopolojarvi/status/130324298806421094...
- new hash function
- more efficient and less error prone .torrent files
- each file in the torrent gets its own hash meaning deduplication across torrents is feasible
- backwards compatible with v1
What is your opinion of existing P2PTV apps, is something missing?
https://en.wikipedia.org/wiki/P2PTV
(I don't know much about the space.)
At some point, I suppose things get very fuzzy, e.g. you're just uploading a single second of an album, but I don't know who practically useful that would be.
The Whonix wiki has an incredible amount of information on topics such as this: https://www.whonix.org/wiki/Documentation
Basically, it's very hard to protect the security of users when a government seeks to prosecute dissidents. There's a lot to take into account. A simple "we saw an upload from IP address x.y.z.w containing <illegal file>" will be enough to hang whoever it is.
On second thought, I guess it would be a nice addition to let users control which files they're seeding, rather than the whole torrent. I'm just not sure it's enough to get them off the hook in the event of legal troubles.
Or you could make the block size smaller (e.g. 1 bit) and tell the lawyers to piss off because 0 and 1 are public domain.
Sadly, new domains of this sort were discontinued in 2010. A cool relic of the internet past and massive geek cred. I remember these URLs from my childhood and they make me very nostalgic.
[0] - https://en.wikipedia.org/wiki/.ca#Third-level_(provincial)_a...
One implementation of this is the "Owner-Free File System"[0], but it is no longer being maintained.
> If you create a new 1GB torrent, you'll need 1GB of new (never seen before) blocks
If you XOR your 1GB file (X) with 1GB of blocks that already exist on the system (Y), you get a new set of 1GB blocks (Z), but it will be hard for an observer to prove that your Z blocks don't actually pre-date Y.
Your defence would be that someone generated Y based on your Z in order to frame you as having created X, when actually your Z, when combined with another set of random-looking blocks Y2 produce a different 1GB file, X2, which is completely legitimately.
I'm not sure if "scraping the network" is possible if chunks have unguessable names. Also, it's possible that someone could have a file shared among friends (over TLS), for years, without it being publicly announced.
> it will be mighty suspicious when the Y blocks are scattered randomly across the network, and the Z blocks are conveniently hosted in one place.
I suppose it depends on how blocks are distributed in the system. If the blocks of Y are all served from y.com and you host your Z blocks on z.com, then both hosts will look equally suspicious is Y XOR Z produces an infringing file X. The owner of z.com just needs to be able to credibly claim that y2.com was already hosting the Y2 that XORs with Z to produce a legitimate file.
Alternatively, a single node could host Z, Y, and Y2, all created by users, with no logs kept of when each block was created or requested (and no search/listing function). Such wilful ignorance may not impress a court, but is roughly equivalent to running a non-logging VPN, or a chat service that doesn't retain metadata, or a Tor node, or an online encrypted backup service. The service could even offer to follow a DMCA takedown procedure, in case non-XORed non-encrypted blocks were stored on it.
Cynically I'd assume they'd just use it to add a conspiracy to commit a crime charge to the list since you technically need to coordinate with other people.
The key weakness in every smartass technical technicality to circumvent the law is forgetting that intent is core concept when the law is interpreted in the court.
Did the defendant have intent to do X? Did the defendant use some purpose build technique to avoid doing the most literal interpretation X ? .. Yeah, defendant is guilty of doing X.
The real difference is between criminal law and contract law. In contract law. In contract law the intention is fixed by the language of the contract document. It matters less if matter if one party didn't intent to violate the contract. It is what it is.
I think we are ok as long as there are no precedents? Isnt that how smartass laws work?
EDIT: while I don't (yet) consider myself delirious, responses to this comment have illuminated how little I know about the influence and application of BitTorrent. In other words, I retract the above opinion.
[1]- https://www.statista.com/chart/17321/global-downstream-mobil...
From the original comment.
Also, just because it was 50% at some point, it doesn't really matter today. Paraphrasing, a technology is only as good as it's latest match result.
Perl was once powering the web, nowadays you have to look at the web with an electronic microscope to find it...
My point is that it was a fad, it's barely used anymore and it was 50% in 2009, now its usage is much, much lower. I've seen a lot of statistics of streaming services being 20-30% of the internet each (Netflix, Youtube), so I'd be amazed if BitTorrent is more than 10% 2020 (and that's a very generous estimate), as I said previously. Anyway, it doesn't matter, there are a lot of techs that were super hyped and in the end, faded away. SOAP, CORBA, XML databases, semantic web techs (RDF, etc.), RSS, ...
> And that Perl didn't live up to its hype because it's only common today, and not as universal as it once was?
Perl is not common, it's uncommon. Let's say you're a developer in one of 100 big cities around the world, how likely it is for you to find a job developing Perl web apps? I'd say that in 80-90 of those, you can't even find that job. That's not a common tech in 2020 ;-)
Perl was not a 'fad', it was what powered the internet's dynamic sites in the early days. Foundational technologies that eventually are hidden from the average person is not what 'fad' refers to.
Just because you don't use something doesn't mean it isn't used. You keep digging in to nonsense. (Also total internet bandwidth usage has increased significantly over time).
* defend myself
* rob you
* murder you very effectively
* enjoy target shooting
* hunt wild game
* provide peace of mind in troubled times when law and order break down
The fewer guns there are per capita, the more useful they become for criminals. They are so useful, in fact, that people will spend a lot of money and subject themselves to costly and invasive regulations just to buy one gun. If no one found them useful, no one would find it necessary to ban them.
Perhaps you meant (and people should have charitably interpreted you as meaning) "I would consider it to have no legitimate uses" or "... no uses worth supporting".
BitTorrent has been influential in lots of applications. If you really need proof that BitTorrent can move milestones, it help perpetuate sites like thepiratebay.org and assisted in the advent of modern day streaming services that both the music and movie industry neglected for years. It is also a central influence for p2p networking and file sharing.
I guess mobile internet was its downfall as few people want to seed from a mobile connection.
Finding URLs of private trackers is trivial in google. Getting an invite usually requires knowing another user, unless they temporarily open signups for everyone.
Here's one of many lists of private trackers
CGNAT has a lot to answer for, too
At least BitTorrent is environment friendly :-)
Setting aside the fact that BitTorrent came about long before BitCoin was a glint in Satoshi's eye, it's still massively used for content distribution, on top of being a foundational technology to several industrial applications: Apache Spark, for example, uses BitTorrent to shuffle/broadcast data around cluster nodes.
Even when BitTorrent alternatives are used in its place (such as IPFS, Dat, or Kademlia), BT was still a massive influence in all of their designs, and to the design of any other DHT that came about after its release.
All of these, plus other systems that are (in abstract) similar constructions, have several industrial applications for asset distribution (Netflix uses IPFS to distribute container images internally [0], Uber and Alibaba do similar things) and network management (DNS for example, is really just a DHT, as are several other network protocols)
Ever since BitTorrent came about, it has unequivocally been a resounding success. It never for a moment had to look for applications, there were already plenty since day one.
[0] https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...
Not BitTorrent exactly, but close enough that if the blockchain spinoff was bought for that much the blockchain folks would certainly consider (rightfully IMO) a victory in the blockchain column.
Is it, though? Youtube doesn't use it, neither do Netflix, Hulu, Amazon Prime Video, Disney+, Spotify, etc. Does anybody (except for Blizzard) use it? I don't think Steam or the Epic store or the App Store or the Play Store use it, either.
I'd be extremely glad to be proven wrong.
I'm pretty sure the BBC iPlayer used it too, at some point (no idea if it's still the case).
Most Linux projects use it.
Any company offering a "download manager" for 1GB+ files, is likely to be implementing something similar to BT behind the scenes.
The thing is: if bandwidth costs are a real problem for you, BT is a killer solution to offload some of those costs. If they are little more than a footnote (like for the mega-services you list), then BT is unnecessary and it will probably slow down overall performance too.
Still waiting for the day such a feature arrives on Linux, it'd be an instant hit everywhere where uplink is slow and expensive, especially in developing countries.