An Uber-like CDN
medium.com
medium.com
- They are paying the third party less than what it costs to rent the server, in which case the third party is losing money and shouldn't be doing this.
- They are paying the third party at least what it costs to rent the server, in which case why aren't they just renting the server themselves? They could probably negotiate a bulk discount.
- They are trying to hide this activity from the cloud server provider by involving third parties. While this may not be a ToS violation as such, I'd guess that if they're hiding it, it is something the cloud providers will want to crack down on if it becomes significant.
- This entire scheme is actually a plan to get access to countries with weird regulatory requirements, e.g. foreign companies are not usually allowed to own servers in China. This one's pretty legit actually, but it's a really complex version of the scheme.
I think at least one of those must be true, and none of them would be a good sign for the future.
Edit: I guess "paying the third party less than their costs" and/or "not making a profit themselves" might be what they mean by "Uber-like", so it would just be an elaborate joke?
Agreed, I'm mildly annoyed that farba.js repo has not been updated since 2021 (no further info) + grammar issues [1]
[1] https://github.com/FarbaCDN/farba-public/blob/master/src/far...
Bare-metal servers are much better is our case. Please note, bandwidth is not free. You can follow the links and see the actual price for it. If LeaseWeb tells you: 1 Gbps dedicated/unlimited per $143/month, it means you can use it 24x7.
>- They are paying the third party less than what it costs to rent the server
we don't know prices in all datacenters all over the world. That's why the _Peer_ sets the price for his service.
While that is true in theory, if you in practice saturate your bandwidth 24x7 by reselling it your contract will probably get cancelled (or at least that is the case for most "unlimited" contracts).
I don't think you actually answered the questions of the parent.
Yes 0.0005$ per GB egress and a provider in Europe still makes money.
That being said I could still see this being against TOS if you end up chewing up SSDs, hosting dodgy content or running at full power constantly.
But that breaks the economics of this, no? If you're in the US and charge $6/TB and I'm in Germany and charge $100/TB, why would traffic ever go to my server? If your US server gets overloaded, am I now making $100/TB? How does the network ensure that this is both fair and competitive? Moreover, how can a customer know that they're not going to get a surprise bill for an order of magnitude more than they planned for?
There's many variables, but the important ones for choosing a peer are cost and latency/distance. What conditions do you have for choosing far-but-cheap versus near-but-pricey? You can't simply choose peers randomly or round robin because I could just set my prices through the roof and pick up a massive paycheck.
In markets, generally, providers are not chosen randomly or round-robin; they are chosen based on the balance of cost and what is offered.
So, if this becomes popular, flat rate servers with unmetered bandwidth at a fixed speed would become more expensive, or come with a more specific ToS to prohibit that.
Similar to how data hoarders ended the "really cheap personal cloud storage" bubble.
The more this trick will be used, the higher the cost of the dedicated fast internet connections with cloud hosts will become. Most people buying dedicated, unlimited uplinks don't upload the amounts mentioned in this article, which is why bandwidth is relatively affordable. If everyone buys into this network to make a profit, the prices would probably rise to the point that profit drops. This isn't against any kind of ToS I know of.
The "make 2.4x your investment" promise also assumes your server will actually serve 100TB of content. The more people join, the fewer files your server will serve and the less profit you'll make.
In the end, the system is profitable until the network has reached the size where everyone breaks even. If people leave the network, profits go up again, if people join the network, profits drop and people back out. People renting servers game the system trying to one-up each other and in return Farba gets a worldwide CDN for cheap.
The best trick may perhaps be that server providers may directly buy into the system, as they can get bandwidth and servers for lower margins than their customers. This gets Farba high-quality service for cheap while the service providers add a relatively easy income stream on the side.
All of this comes at an additional cost to the customers, because now your battery-starved mobile phone is validating cryptographic signatures of images and other content.
This also leaves legal questions: who's responsible for illegal content hosted on servers hooked into this network? If your hosting provider of choice gets a takedown notice for child porn or malware, will you still be able to use their services?
>now your battery-starved mobile phone is validating cryptographic signatures
with a native app, it costs nothing. Actually, the main power consumer is the phone screen.
>who's responsible for illegal content hosted on servers
this is a very important question. We are going to run AI models against all content we host on Peers. Right now, we don't accept third-parties and "fake" all Peers ourself. Want to ask lawyers what is the best way to handle this case.
1. Performance. In terms of latency, this is adding an extra round-trip to each page request (to get the metadata). It's also likely going to load different resources from different servers (hostname, IP) preventing connection reuse. I guess they can't even start fetching data until the full page has loaded? And it feels like this scheme would make it much harder to reliably cache resources, since the browser will cache by URL while they'd struggle to make sure the URLs are stable across calls.
2. Privacy. Their threat model only addresses malicious peers changing content, not malicious peers trying to track users.
3. Economics. If running peer servers really was profitable, the company would do it themselves rather than outsource it. Given the "where can you get a server" section, it's not even that they're expecting this to just be running on spare capacity.
The problem with the setup is that incentives of the peers are badly misaligned with the network, the clients, and the end-user. A peer wants to maximize traffic; the clients and end-users want to minimize it. So a peer would be incentivized to set cache-control headers to prevent caching, to increase traffic. Likewise a peer would be incentivized to only keep copies of the most accessed resources, to be able to serve as high a proportion of the traffic as possible with a given disk budget, while all the other stakeholders would like each file to be available just N+1 times in each region.
4. UX. URLs will be really crappy compared to a proper CDN. Right-click copy a link to an image and send it to somebody? It'll initially be to some super-dodgy URL, and stop working in a day or two as the CDN node starts caching different content. And AMP showed that people do actually care about URLs.
It's pretty hard to take this seriously.
fair point. We use same technique as lazy-load (you have to wait the full page has loaded).
>3. Economics. If running peer servers really was profitable
Uber doesn't drive taxis himself, airBnB doesn't own property. This is how Uber-economy works. We believe it works in the CDN industry as well. We just don't want to be another CDN provider.
1. It started out burning huge amounts of investor capital, basically transferring all of that money to drivers and in the form of subsidized rides, to riders.
2. It works today by a combination of the above and underpaying drivers, who generally make less than minimum wage when they manage to cover their completely costs at all. Despite this, prices have gone up dramatically, so that taxis are once again competitive on a price basis.
3. Uber was profitable in 2018, and not before nor since[0]. Deeply, profoundly unprofitable. Why would anyone want to be the Uber of anything?
0. https://www.macrotrends.net/stocks/charts/UBER/uber-technolo...
> We just don't want to be another CDN provider.
Don't do something differently for the sake of being different. Do something differently because it unlocks some new capability.
Uber was special for a few reasons. Primarily, it was a better UX to use an app for a taxi, and apps couldn't exist when taxis were first popularized. Secondarily, Uber not driving taxis themselves was "different" in labor law handling. Uber didn't own cars/employ drivers not because "they didn't want to be another taxi company" but because they wanted to unlock the capability to pay people differently. It's the same story for AirBnB - what's legal for rental units is not the same as what's legal for hotel rooms.
Any company can rent or own thousands of servers across the planet, we know this because CDNs do it today, and because if an individual can do it, so can a company.
also please note:
if you have 1 image on the page - we have to do 2 requests (1 metadata, 1 download)
if you have 50 images on the page - we have to do 51 requests only (1 metadata for all images, 50 downloads)
But unless you return the same host for 50 objects, I'm going to have to do a bunch of DNS lookups too, right? And a bunch of SSL setups.
Hopefully you at least make sure to try and return a single host for all the objects in a request.
our goal is to saturate the _User's_ network. Now it's hard to say how many Peers we need to meet this requirement. It looks like 2-3 Peers are enough.
Also, when you have 50 Peers, you have to do 50 DNS/SSL requests in _parallel_. With current hardware, it's not a big deal.
You should make this your marketing slogan.
"Try our CDN, with current hardware the gross inefficiencies other CDNs spend a lot of time and effort eliminating are not a big deal."
How do you know the nearest peer has all the images at hand?
- we divide web cache in buckets
- these buckets have different sizes
- we know a list of files in every bucket
- we know what buckets every Peer has
2 is definitely an issue.
3 is a maybe. There's definitely a model where people with access to cheap servers and bandwidth can get on board and not have to worry about the application-level details, finding customers, billing etc.
4 isn't an issue as long as you are selling the CDN specifically for website assets that don't need to have pretty URLs.
Not exactly. For a normal CDN, your webpage HTML directs the client directly to the CDN server, which in theory already has the object. Only if it doesn't have the object does it need to go get it, but there isn't an extra lookup. The CDN does it's direction magic via DNS or IP routing, not with double lookups.
3. Why would the people running a peer have cheaper access to servers and bandwidth than a company specializing in that business? Everything about this business should have huge economies of scale (buying hardware, system administration, transit/peering).
The only situation where I can see a distributed model winning out is if the entity running the peer does not pay for resources but gets them with fraud or theft, e.g. paying for cloud servers with stolen credit cards.
I'm narrowing the scope around 'user tracking' specifically to 3rd parties, because 1st party user tracking is a requirement in order to perform some necessary functions (Providing accounts & user sessions, for example).
In the current centralized CDN model (Cloudflare, AWS, Akamai, etc.), the 3rd party is made clear (those aforementioned companies), and their need to track users is documented on the provider's ToS. Likewise, reputational damage is accrued to the providers in question.
In their proposed CDN model, there is no clear 3rd party, and so it must be defensively assumed that all of the data sent for requests is recorded at all times (The FBI Tor exit node problem). As a consequence, the minimal amount of request data can only be sent to these providers (IP address, content hash). Tor-like routing could be implemented to further disperse the requests from one IP address to multiple addresses, but now the monitoring problem's increased to the routers as well, on top of the additional latency introduced for said privacy. Mandating said routing into the protocol would make such tracks useless, as the IP address collected will roughly appear as random noise in the optimistic case.
Personal opinion below
However, it is likely that no technical solution provided would resolutely satisfy the question being asked.
Cache management (eviction, invalidation, etc) is done where? By the look of that diagram, you are actually doubling your origin traffic just to push to the cache after serving the original client request. Perhaps subsequent requests could get served from their 'peer' but the hit rate will be abysmal.
Maybe there is more to it than they describe here, but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
The disk performance - is the weakest part of the Peer. It is a part you can't guarantee any QoS on a cheap server. The idea is to keep popular cache on Peers in memory. Everything else we will host from a limited number of regular servers (belong to Farba), with high-performance disk subsystem.
>Cache management (eviction, invalidation, etc) is done where?
yes. You can invalidate either a single URL or a folder with a wildcard
>you are actually doubling your origin traffic just to push to the cache after serving the original client request
this is something we can proxy and do only one request per file. Current implementation was done for simplicity and reliability.
>but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
you can easily check this up. Please open the network console on our demo-page and see how it works:
- we have two different certificates (one for the main site, for for the Peer)
- everything works flawlessly
we just need to add more Peers and evenly distribute the workload.
> what kind of cache hit rate are you expecting at the peer?
I don't know. We have too many moving parts right now and can't predict anything.
That doesn't make sense! If you add more peers just to keep more files in memory, each peer is serving less and less data. There's a fixed, finite amount of usage that the system will receive, and the more peers you add, the more that usage gets subdivided among the peers. Which means they make less money, which makes the system unprofitable. Moreover, you're spending more money to distribute the file to more peers (well, you're not, you're offloading that to the customer right now), which means the more peers, the more expensive for you or the customer.
The problem you have is that handling more files means cutting peer profits without adding any benefit for yourself or the peers. Storage isn't factored into pricing, which means that any accommodation you make to increase availability (an important part of being a CDN!), the worse the economics end up being.
I use a CDN to serve about 50GB of files stored on my origin, which turn over every week or so (a new 50GB every week). What you're saying means that _at best_ I need at least 3-4 peers in order to keep one copy of all of my files available. That's your using your numbers for a LeaseWeb or OneProvider peer. And if I'm competing with other customers, my content is getting evicted, resulting in more hits to my origin. If I'm actually using your system as a CDN where peers should be physically close to my users, that means that I'm filling up RAM of lots and lots of peers just so that enough peers that are close to my users can serve the files.
Akamai acquired RedSwoosh precisely on the same economics promised and if this were economical/viable they would already have deployed it, no?
I did hack with Travis on RedSwoosh code base optimizing his TCP Zipper algorithm which was pretty cool BTW
I don't know how it works in details, but it looks like you need to install a browser plugin to enable the p2p network here.
Our solution works transparently for the end user.
You could send a hash of the file to the client from the coordination server and check it. If the hash isn't valid, you can re-request the file from another peer. But I don't think you can trust the client to report abuse: a malicious client could report innocent peers (e.g., a vindictive Chrome extension).
Or, the site owner could manipulate the coordination JavaScript to report the downloaded file as invalid/corrupt/tampered with, and then just not re-request the file (and there's nothing you can do to stop them). So in that case, the site owner is managing to not pay for the service (assuming you don't bill for downloads that were reported as invalid).
But on top of all of that, this is _slow_. It's slow on my (quite fast) internet. If you're using this, it's because you're trying to get cheap bandwidth, not because you value performance. The whole point of a proper CDN is to improve performance by spreading out load and moving the edge server physically closer to the user. This might be putting the edge server physically closer to the user, but the mechanism to do that seems to take almost as long as the request/download of the asset itself.
This fraud is possible. We collect logs from three sources:
- the Balancer (belong to Farba)
- the Peer
- the User
We can analyze every single case of suspicious behavior.
However, a lot of people use CDNs because they are fast. They want the latency to be ultra low and they want it to be reliable. This is where the author's solution falls short: CDNs can get premium quality bandwidth because they have more scale than their customers. Sure you can find servers from various hosts in lots of different data centers, but you might not get the latency or quality of bandwidth you were looking for. This can be critical for some applications like streaming or serving images.
If your constraint is speed, you almost have to go with a CDN (even if you are huge). If it's volume, you are in a better position. If your volume is small, you might as well go with something off the shelf like Cloudfront or Cloudflare. If you are ultra huge like YouTube or Netflix, then you have to peer directly with ISPs.
The reason businesses like this work in general is because they break a big, complicated business into smaller businesses where there is specialization. There may not be be meaningful potential for specialization here; but that is the argument to have. For example, it may be that Farba can be a leaner business if they don't have to think about how to get the best pricing in dozens of different hosting markets.
Here is one of their papers: https://www.usenix.org/legacy/event/nsdi10/tech/full_papers/...
Was the issues you were seeing in this general area?
1. Round trip to Farba to get metadata.
2. Round trip to your "driver" to get the resource.
3. Computing the hash of the resource to ensure the "driver" didn't cheat.
It could still be worth it as a way of saving money on bandwidth, but I'd be quite surprised if you got the speed benefits of a CDN.
Do you imagine seeing peers in the form of smaller players with beefy home setups?
The network scales very well. The network is divided into multiple GeoZones. We can add remove these zones easily. GeoZones are not connected to each other, so there is no single point of failure. Actually the major cloud envs are the most expensive possible variants for Peers. Bare metal servers are much cheaper in our case.
>Do you imagine seeing peers in the form of smaller players with beefy home setups?
Home internet has a couple of disadvantages: - traffic is not "unlimited". Xfinity doesn't allow you more than 1 Tb in and out traffic per month - home internet is asymmetrical: you could have 1 Gbps in and only 5 Mbps out.
Anyways, other than that it’s a fun concept. I like the P2P nature of it.
Zero-cost CDN for 2 USD/TB! Sounds like a great deal to me.
On a more serious note, is price the problem with other CDNs?
as a web developer, no thanks, I do not want to trust my app’s speed to your profit incentive or margin.