Amazon S3 quietly deprecates BitTorrent support | Hacker News Reader

Amazon S3 quietly deprecates BitTorrent support

github.com

175 pointsgreyface-55 comments

Open articleView on HN

It already says:

> Amazon S3 does not support the BitTorrent protocol in AWS Regions launched after May 30, 2016\.

So BitTorrent support has been sort of deprecated for a long time.

Commenting for visibility. Its much ado about old news. Upvote the parent.

> Commenting for visibility

Can you elaborate? Do you believe that HN's algorithm prioritizes "comments that have replies" more highly?

human eyeballs sorta do

Human nature is going to give those comments more upvotes, which automatically moves them to "top comment" position, so at the risk of being facetious: yes, obviously?

If you're claiming that upvotes on replies count against the ranking of the parent comment - I didn't know that, thank you!

If you're claiming that replies can be "top comment" - I don't think that's true, but I've learned that HN has a lot of secret UI and features that are only unlocked at certain reputation levels, so maybe you're a higher-level Laser Lotus than I am.

If you are claiming that "comments with replies" are more likely to garner upvotes _because of_ those replies - I don't believe that's true. I can't remember ever having read a reply and then going back to upvote the parent.

If you're claiming that "comments with replies" are more likely to garner upvotes due to some shared underlying cause ("good comments get replies, good comments get upvotes") - while that's probably true, it doesn't affect my question of "why would you leave a comment that _just_ says 'upvoting for visibility'?", if that doesn't affect the underlying comment's ranking?

I don’t think the HN comment ranking algorithm is public. I’ve seen people speculate that it’s a function of comment age (newest ranked higher), average comment score of author, current upvotes and current comments.

That said, it doesn’t make sense to add a comment to game the ranking.

Sorry, where are you pulling this quote from? It's certainly not in the PR, and if you google for that quote, nothing official shows up, so... where in the official AWS S3 docs is this supposed to come from? Because without first party (not third party) sources, this quote is a made up bit of text, with zero validity.

If you expand to see the whole file named “doc_source/S3Torrent.md” that contains the changes mentioning the depreciation, that quote used to be on line 10 and is now on line 13

cheers

I actually had a use case that would have made sense to use this, in theory, but the implementation left a lot to be desired. The tracker Amazon ran was really trigger happy to ban peers (I'm sure it saw tons of bad actors, but still, it made testing really painful especially if you were churning infohashes as you tested new objects). And the seed was somehow painfully slow. Never once did any of my test scenarios see faster downloads than just downloading from S3.

Maybe if I got some critical mass of users seeding, but since I couldn't get stats out of the tracker, I had no way to encourage users with gold stars or whatever.

And, on top of all of this, since you couldn't create a multiple-file torrent with this, you had to really work around silly limitations.

This is somewhat of a non-issue, S3 can still be used in a a .torrent file by entering a raw HTTP URL as a Web Seed.

This probably means that no-one was actually using this feature to an extent that it was not even worth putting an announcement out.

Probably, I had no idea there was a tracker support, never seen any option to control it.

I evaluated it for possible use ~5 years ago, and found it too unreliable. Their trackers would shut down after some period of inactivity and didn't often seem to come back online for later peers. AWS neglected this feature.

Seems out of character for AWS to deprecate something without at least notifying customers of a schedule.

Does this commit even reflect that BitTorrent support is being dropped?

yah, just looks like 58 markdown files were edited. Maybe remove from documentation first then begin to remove functionality. But if it was in the documentation at any point you know you can only stay in such and such version of software.

> Does this commit even reflect that BitTorrent support is being dropped?

Yes; more direct link: https://github.com/awsdocs/amazon-s3-userguide/commit/0d1759...

>> As of April 29, 2021 Amazon S3 is discontinuing the S3 BitTorrent feature and it will no longer be available to enable\. AWS will support customers currently using the S3 BitTorrent feature for 12 months\. After April 29, 2022, BitTorrent clients will no longer connect to Amazon S3\.

> As of April 29, 2021 Amazon S3 is discontinuing the S3 BitTorrent feature and it will no longer be available to enable. AWS will support customers currently using the S3 BitTorrent feature for 12 months. After April 29, 2022, BitTorrent clients will no longer connect to Amazon S3.

Seems like this is the notification.

Sorry for the stupid question, but how was this used previously? You could just set all your torrents to download to Amazon's cloud?

the other way - S3 could seed directly to a torrent, I think

For example Linux distro could provide a DVD image. When your users would chose to use torrent you could get huge savings in bandwidth if file is popular, also the users would enjoy higher throughput.

I'm not knowledgeable on bittorrent but I feel like it should still be 'compatible' (with a shim layer) with S3, just maybe not natively.

You should still be able to effectively use S3 via Webseeds - the Bittorrent protocol specifies including http sources in your torrent file. The only big issue is that not all torrent clients support it.

If doing this stops someone from their precious, they will update their clients right away.

I don't know of the internals, but on the surface there were three things this did:

A torrent creator. This is a simple extension to the REST endpoint of S3, and a specific S3 API, that'd scan a S3 object to create the torrent file, and register the resulting infohash with a tracker.

A tracker. Technically not necessary, but with the implementation AWS went with, they needed to run their own tracker to allow peers to discover each other.

A seed peer. This was probably a shim on S3 (if I recall correctly, the IP was within the S3 range of IPs). This was responsible for speaking BitTorrent and always having the objects available for clients to download.

There are other ways to do this, but I'm sure this pile of tech was a source of abuse and problems for basically no one to use, so I'm not surprised to see it go. I also seem to recall there was a single tracker, and AWS really wants to get rid of single entry points to S3.

Genuinely curious as I could be totally wrong on this - didn't p2p video streaming sites like Peertube use Webtorrent with S3? Also I think many Linux distro rely on it? Other than self hosting it, what other data services offer this support going forward?

disks are cheap, residential bandwidth is cheap, AWS and s3 are crazy expensive to use for torrenting

I think the idea was that you could host a file as a torrent but use S3 as an always online source for the file so you can distribute the .torrent without worrying about all peers being offline at some point since s3 would always be online. In theory it would be cheaper than the alternative of just giving a link on s3

I thought HN swore to me that AWS never deprecates anything ever? What’s next, SimpleDB?

Seems neat, but I'm sure this form of hosting attracts fraud and is probably difficult to maintain.

I’m not sure how you arrived at that conclusion. I can’t find a harder way to spam than this.

Spam isn't the only kind of fraud?

Pardon. I meant to say abuse, not spam.

if they're going to deprecate the thing, i feel like they have an obligation to at least state why. letting random joe's from the internet chime in with sympathies explaining why they agree with the decision to remove good well used technology doesn't feel quite up the the bar i'd like to see for technologies being removed.

If AWS is deprecating something, then it's likely that literally no one was using it and the few that might have been were given complimentary consulting to move them off of it.

Yes, exactly. I get a billion messages a year about obscure lambda runtimes expiring for junk I provisioned half a decade ago and haven't ever invoked.

There's no way this had any real users if it is being quietly turned off.

It sounds like there are customers using it and they will have a year to migrate off of the feature.

There are undoubtedly a couple personal accounts with buckets configured in 2014 which have been idling at $.04/mo ever since, and they're allowing the grace period just in case. But likely no clear active users.

I think Bittorrent's history makes handwaving deprecation away slightly less easy.

I mean, you're probably right - that none of their $100m contracts were using the feature. But otherwise, why remove it? What was it costing them in maintenance and support? Were there actually any security issues?

I would assume that no one paying them anything was using it.

An obligation to who? For every interesting explanation there are likely a hundred banal ones that rationally lead to the same conclusion.

Terribly unlikely. AWS outbound bandwidth costs are outrageous.

Nobody in their right mind would provision a BitTorrent that could suddenly give you a couple thousand dollar bill if your torrent suddenly got popular.

The possibly ridiculous bandwidth costs probably meant that absolutely nobody was using this.

What would an economical alternative be?

Grab a digital ocean VM for $5/month and call it a day. Should be enough for all your (legal) torrenting needs.

Plain ole bit torrent. Run some seed nodes on dedicated hardware with unmetered bandwidth. It'll get you pretty far.

How about: where the cost is amortized among sufficient uploading peers that it can be easily hosted on consumer home connections, despite very small upload rates available on consumer ISP connections?

> AWS outbound bandwidth costs are outrageous.

The torrent would have been a way to get the data from peers instead of serving it purely from S3 outbound.

If this was the only active seed for a torrent and there was nobody else, that would really be a problem.

But otherwise, this should technically result in less outbound bandwidth usage than serving the whole file out of S3.

> If this was the only active seed for a torrent and there was nobody else, that would really be a problem.

An AWS connection is going to have WAY more bandwidth than everybody else in the system.

The bittorrent clients would prioritize the AWS seed and suck way more download off of it relative to everybody else in the swarm.

> An AWS connection is going to have WAY more bandwidth than everybody else in the system.

Depends on how popular the torrent is on a local level. Two people on the same LAN probably have better bandwidth to eachother than from AWS. Two people on the same ISP in the same metro might have better bandwidth to each other than AWS if their connections are symmetric or if their ISP has poor transit and peering. Maybe a handful of other scenarios, but even if most of the clients get most of the download from AWS, that's still better for your wallet than all of the clients getting all of the download from AWS.

Apparently torrenting from S3 did not work very well:

https://news.ycombinator.com/item?id=27524963

> The tracker Amazon ran was really trigger happy to ban peers (I'm sure it saw tons of bad actors, but still, it made testing really painful especially if you were churning infohashes as you tested new objects). And the seed was somehow painfully slow. Never once did any of my test scenarios see faster downloads than just downloading from S3.

Fraudulent billing is a problem for public cloud providers. In particular with hosting pirated or illegal content, so BitTorrent seems like a natural issue.

Pirate content on a stolen AWS account is less likely than crypto miners.

It seems to me that the worst case scenario for using BitTorrent is that S3 is the only seed and everyone downloads from that, however, I don't see how that is worse than the alternative which is to offer direct downloads from S3.

In other words, from a costs perspective, the worst case scenario for BitTorrent is the best case scenario for direct download.

Reply on news.ycombinator.com