Double paid traffic
wiki.hetzner.de
wiki.hetzner.de
What do (presumably) Comcast and these other companies say when you communicate with them about peering, and you ask them why they won't? Anyone with insight into these issues care to share the DLS/cable provider perspective? (Is there some kind of reasonable argument there we're not hearing - "Infrastructure costs are rising, you see, and we built the infrastructure we have now with a government subsidy that's no longer available. So when we expand, it's going to cost 10x as much ... " just to make something up. Is there another side to the story, or is it really just supply/demand where the providers exploit their market power? "Our customers are not likely to drop our Internet service for a competitor just because Netflix is slow. In fact, our data show that they start watching cable TV more, which is good for us - we want cable TV to be sticky. So we'll charge a fee since we're nearly a monopoly and the market will sustain it. Further, since Netflix is where all the traffic is going, we'll just charge them directly rather than expand pipes all around".) [edits]
> For this reason, interfaces are being operated at their highest capacity -- to make paid access to the DSL and cable providers' networks attractive for content providers. The end consumer is then forced to pay double for unlimited access to the Internet.
I don't entirely follow this part. Could someone clarify? Is "operating at their highest capacity" a bad thing? Is the implication "operating at their highest capacity [without expanding capacity when they should]"?
Yes. This allows providers to say "guys look, we can not increase the traffic going over the lines, so we have to priorize things. Thats 3.50€ a month per uplink"
Yes. Basically existing links (thru transit providers) is saturated, if you(content provider) want good connectivity with these end-user networks you gotta pay more for direct connection.
* Not upgrading their network as they should
* Discriminating against certain traffic, see Comcast and Netflix speeds
* or by not allowing content providers to connect their CDNs free of charge, which should be a win-win situation if the providers acted in the customers best interests.
I honestly believe that these bussiness practices are the providers digging their own grave. Best case scenario, they make a few quick bucks before laws get passed against this sort of thing, worst case scenario: the big boys get tired of dealing with them and start competing directly (like Google fiber).
Yes, it's a bad thing. Internet traffic is bursty so usually you have to overprovision a lot to deal with the spikes. Operating at full capacity means that they get money for a service that they cannot fullfil. This is their intended goal such that later they can ask money from whoever wants to have priority.
The article is about T-Systems (AS-DTAG), which is the largest ISP in Germany and basically the company we mean when we talk about net neutrality violation here.
Look at the speeds of their exchange points: https://beta.peeringdb.com/net/196 - 34 Gbps total: 1x 20G, 1x 10G, 4x 1G. Seriously, 1Gbps!
Now compare with another ISP with only 25% the revenue: https://beta.peeringdb.com/net/997 - 890 GBps total.
Whereas DTAG _does_ way better.
I dont like it, but as others have said: Relying on transit peering will not work well once you reach certain traffic levels.
Also you can get Servers from Strato (DTAG Sub) for about the same price as from Hetzner and pump your Traffic directly into a Tier1.
The cable service slowdowns are technology inherent. But that's a last-mile problem and not a backbone problem. Also, these providers aren't trying to use their customer base to extort money from hosters.
yet i doubt that buying from a (DSL-)Reseller is improving service quality and prices are roughly the same.
We even needed to contact Github to change our peering since the default route from Github to DTag just sucks.
Our overall bandwidth with Telekom simply sucks. Most networks are really really slow, compared to Kabel Deutschland.
and they were like "sure cosider it done"?
no. they tried to find a way with us. but after that it didn't worked they changed the routing and it worked.
https://en.wikipedia.org/wiki/List_of_countries_by_Internet_...
But also, people who work for exchange point operators are the only ones left flying business class in the telecom industry. At the same time public exchanges are a single point of failure in the network that one should(!) design around anyway. There are much cleaner solutions you can design into a network at a fraction of that cost.
[1] https://en.wikipedia.org/wiki/Talk:List_of_countries_by_Inte...
[2] http://www.huffingtonpost.com/bruce-kushnick/americas-ugly-t...
Marginal capacity changes are pretty cheap. There's no technical reason to treat network links as constrained resources. So as a network provider, you only create constraint to make money.
Think of it as food & drink in a movie theatre. A bag of popcorn at the Target snack counter is $1.50. A bag of popcorn at the movies is $9. Same product, both high-margin scenarios -- the difference is you're captive to the movie theatre.
I have compared speeds directly to OVH's servers, and indirectly through a reverse proxy, and AT&T throttles direct connections to OVH. When I prompted OVH about it they confirmed to me that AT&T requested that they pay them for priority bandwidth and they refused (as is their right) and now my customers get throttled to 200 kb/s when accessing my servers on OVH.
PS. OVH has arguably the largest data center in North America so this means a lot of services are being affected by these ISP's.
Since we cannot change market dynamics, we are offering our customers a way to offer their customers faster service during peak periods with this "pay per peering" option. ===
My understanding (which may be incorrect) is that Hetnzer is telling existing customers that instead of having to install servers within the ISP network, they can instead pay a marginal fee per month to bypass the congested peering links to these providers.
I really like that paying up to DTAG is not the default and is more bureaucratic. This might already be enough to kill the ISPs' profiteering schemes, similiar to what happened with the Leistungsschutzrecht (see https://en.wikipedia.org/wiki/Ancillary_copyright_for_press_...) in Germany.
I think the name "double paid traffic" is a very clear statement. They are not trying to sugarcoat this or anything. They just don't want to loose customers over this.
Of course, this only works if there is competition between ISPs.
https://blog.cloudflare.com/net-neutrality/
Their article about the relative cost of bandwidth around the world made the rounds on Hacker News when it came out:
https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...
> Australia is the most expensive region in which we operate, but for an interesting reason. We peer with virtually every ISP in the region except one: Telstra. Telstra, which controls approximately 50% of the market, and was traditionally the monopoly telecom provider, charges some of the highest transit pricing in the world — 20x the benchmark ($200/Mbps). Given that we are able to peer approximately half of our traffic, the effective bandwidth benchmark price is $100/Mbps.
Cloudflare supposedly buys transit from transit providers. However, since it's serving cached data locally from each POP, presumably that means that connections from a site visitor to Cloudflare are served locally, for cached assets, rather than being passed on across transit to Hetzner.
Isn't what they are advocating for the exact opposite of net neutrality? Everyone's connection is saturated, but they argue that because they are a large content provider they should get special treatment and not need to pay for direct access. Smaller companies are equally affected during peak hours, but they would need to pay for that same privilege? If not, what is too small?
There seems to be a trend of adding "free peering" into the net neutrality debate. Unless you're carrying equal bit-miles, free shouldn't even enter into the equation. If data flows unequally, money should flow the other direction.
I think a lot of companies are leveraging the current hate for cable companies to get out of properly engineering their infrastructure. People have leased lines, built out additional datacenters and paid for colos around the world for years and it was never "extortion" until a popular company twisted the definition of net neutrality a bit so they wouldn't need to build their own datacenters.
Ok, data flows (mostly) this way:
Hetzner (or any other DC) --> ISP --> customer
DSL companies want money to flow this way:
Hetzner --> ISP <-- customer
The end user is not paying for access to content, so the fact that Hetzner is providing something requested is irrelevant because that's not what the internet ever was. In your diagram, Hetzner is an end user and whether it's up or down data you still need to pay.
The end users are not Hetzner's customers - the people paying them for hosting are. If they want to lease a line and offer access to that leased line at a premium to their customers for better connections during heavy times they are free to do that. It's not the cable companies job to make sure Hetzner's customers are happy though. There is already a framework in place to achieve what they want.
Your ISP is not supposed to both ask you, the other ISP and also the client of the other ISP, your subscription service, for money, independent of actual load.
The ISPs customers are paying for a connection to the ISPs network, nothing more. Every connection to the network is a node - in a network neutral world the fact they are a content provider is meaningless. If they need to push more data, they need to pay.
I really don't see the difference between this, and a large company attempting to send their backups via the web. We could easily set it up so the remote servers are "requesting" that data -- but it's not the cable subscriber's job to subsidize my network infrastructure so we lease a line to ensure the speed we need. We have over 100mbps connection on both sides, but that is only a 100mbps connection to the ISP. If we need to move data at that speed all the way from point A to B, then we need to pay for the infrastructure to do that.
There's no such thing as "push" data, it's like trying to push a rope. Any reasonably sized data stream has the consent of both parties on the network.
VoIP: generally negotiated somehow before the UDP "pushing" happens
scp: just because in your mind it's a "push" doesn't make it so. In reality the other machine is sending you acks, without which your machine wouldn't continue "push"ing and thus, again consent is required. finally you're logged in to that other machine which nearly universally equates to consent
In no circumstance should an ISP be directly charging the customer of another ISP.
In this situation, Core-Backbone pays for peering with DTAG and Hetzner pays them to transport their data. I'm not seeing how that is an ISP billing another ISPs customer? Hetzner is paying Core-Backbone, Core-Backbone is paying DTAG. Same as Netflix paying Cogent to transport their data.
They are also (usually) paying for a specific connection speed. If a customer is paying for a 100mbps connection and Hertzner has 100mbps of spare bandwidth, but the ISP consistently only has capacity to relay 20mbps per second from Hertzner to my computer, then I would argue that it is the ISP's responsibility to upgrade their hardware so that they can fulfil their obligations to customers that they have sold 100mbps connections to.
I request my Amazon package, but amazon still pays for delivery--even if amazon charges me for the shipping, UPS or Fed Ex get paid by Amazon.
The internet works sort of like that, except small ISPs will still pay to receive since their network doesn't interconnect with everyone else. ISPs have always charged CDNs for access to their network.
As a practical matter, not all traffic is requested. Which is one reason that the receiver shouldn't pay. Also, big data centers sending massive amounts of data are the in the best position to negotiate good rates and be efficient. It makes sense to lay the cost at their feet.
I think the difference here with CDN provided traffic is the bit miles are basically zero so I don't think that argument really holds up.
I would think rules enforcing net neutrality would explicitly not allow this. If I am a small time host with 20 clients, a host with 5000 clients shouldn't get a direct connection for free while I would need to pay. Both my clients and their clients are equally affected.
All ISP will only carry traffic that's either from their customers or to their customers.
Heztner isn't arguing that they are large enough to be worth getting a free connection but that their providers Level 3 and NTT are worth it to get a connection to Comcast's customers (not every ISP connected to Comcast) at a reasonable expense.
The issue is that Comcast / insert other residential ISPs have a monopoly on access to their own customers and the large ones can charge for that.
This is not about moving the data to the telco's network, but about the telco delivering the packets to their customers. Cost-neutral peering does not mean free peering, those "content providers" complaining about net neutrality violations do not complain because they have to pay for a line from their datacenter to some location where the DSL telco's network is, so that they can interconnect (they generally are more than willing to pay for that, at least if it's not some location where the respective telco is the only one who could provide the line and would bill unusually high rates for it) - what they are complaining about is that, essentially, even if they were to offer delivering their traffic directly to all the central offices of the telco using their own lines, so the telco only would have to connect them to the DSLs of their customer, those telcos would refuse unless they were paid for it.
And I am not exaggerating: There have been cases where CDNs offered to place cache servers in the telco's network, to pay for the rack space and the power and their servers at normal hosting rates that other hosting customers would pay, so that the traffic originates as close to the subscriber as possible, thus reducing the network costs for wide-area transport by the telco as much as possible, and they refused.
They want to be paid for access to their customers - nobody would complain if they wanted to bill for wide-area transit at normal market rates, or even if they wanted to have the traffic delivered closer to their subscribers, as long as the content provider could choose their transit provider freely, thus buying the transit at market rates. But in practice, they want to be paid unless you installed your own DSL to their customer.
Also very interesting, though german only, unfortunately: http://netzneutral.init7.net/de/situation-init7.php - they report that they tried to get an offer from DTAG for direct peering, and what they got not only didn't match what they requested, but also had progressive pricing: The more capacity they would have wanted to buy, the more they would have had to pay per unit. That is to say: a 10 Gbit interconnect would have been more than 10 times as expensive as a 1 Gbit interconnect. That certainly is not due to costs, hardware for bigger pipes never gets more expensive the more you buy.
That depends. It's currently the case for 100gig (minus xconnect costs) depending on vendor platform. Cisco's yield on their CPAK modules is rumored to be in the low single digits. If you can actually get your hands on a working module, they're crazy expensive right now. People are hoarding them and saying "which customer is this really worth or will pay me the most?"
The Cisco 100gig situation is so bad right now, that there is a very large international carrier looking to replace their Cisco edge with Juniper.
I'd theorize that a decent percentage of all Internet traffic to residential ISPs is made up of HTTP requests for highly cacheable resources. However, it seems like the mythical idea of an automatic caching HTTP proxy at the ISP level never really materialized. Perhaps that was possible in the HTTP days, but HTTPS makes it a challenge.
How much better would Internet congestion be if there was a way for ISPs to cache commonly-referenced resources? They could even be video resources - if Netflix requests its videos across an HTTP connection by identifying a single common resource (a URL everyone will access who plays the video), then an ISP could simply cache the URL and serve it locally.
It seems like HTTPS is a barrier to schemes like that. Instead of an ISP running a generic cache for commonly-referenced resources, each website provider needs to run a local POP that's trusted and knows just their content. (Though maybe Cloudflare like solutions are possible where the POP is not fully trusted.)
If we set HTTPS aside for a moment, could we speed the Internet up with caching? How much of the traffic to saturated links are requests for the same content over and over again?
Is there a conceptual way to achieve some of the privacy benefits from HTTPS while allowing an ISP to cache content in a generic way? Or could an ISP offer a discount for using HTTP traffic instead of HTTPS, and then cache content in a protocol-compliant way?
They are not interested in solutions that do not involve paying them money.
Presumably you'd have to use a signed hash of the ciphertext as the identifier, and decouple the encryption of these blobs from the rest of the TLS stream (use independent key material) to ensure it remains secure. But this way, any ISP that sees the same blob appearing frequently can chose to cache it automatically and quickly start saving bandwidth.
Also, for maximizing privacy, I think this tag should be applied by the client only (can be disabled if you want to reduce metadata leaks) when the server tells the client "this particular data with identifier X can be auto-cached on the network level". One potential neat advantage here is that a LAN router/proxy could implement a cache for its own nodes by having the nodes send these cache requests directly to this router instead of letting the ISP see it, so you don't leak this metadata outside your own network.
You don't even have the choice of direct connections to all of the world with just a single server in Germany.
Yeah, you'll still need a lot of transit to reach people, but you can also peer with who you want.
Doesn't matter when we are talking about DTAG, which is the dominant ISP and will not peer for free. Yeah, you can peer with a lot of smaller networks which is nice, but not enough.
Same applies for the US market where the big players refuse to peer.
So maybe there point of view, beeing more important than any public exchange, is not far fetched as it is most certainly a business reality.
Hetzner making a stand for net neutrality is a joke. They are asking your money for better service.
BTW, nobody is denying that they in fact are an important ISP, where "important" means "lots of customers", which in turn means "lots of power". Someone holding a gun to your head also has a lot of power and thus is important to you - but power in neither case automatically implies that any offer the powerful party is making is a fair offer, be it that they ask you to give them all your money to avoid being killed, or that they ask you to pay massively above-market rates to avoid being unreachable for their customer base.
Most of their Servers are in east-germany where
* labor,land,power,darkfibre is cheaper
* they can harvest developement subs from the goverment [1]
[1] https://translate.google.de/translate?hl=de&sl=de&tl=en&u=ht...
2. You are aware that DTAG does not peer at DE-CIX, so whether "peering is most dense" there has exactly no influence on whether Hetzner gets good connectivity to DTAG?
3. You are aware that Hetzner does peer at DE-CIX with 200 Gbit/s, with an open peering policy?
4. You are aware that Hetzner even peers at AMS-IX in Amsterdam with 100 Gbit/s? That link almost certainly is even more expensive than the one to DE-CIX.
see also, peering list close to the end: http://wiki.hetzner.de/index.php/Rechenzentren_und_Anbindung...
2. dtag is ofc present in frankfurt and at de-cix, with little capacity for public peerings as mentioned.
This and your 3/4 each make sense, as they represent a legit business interest for the different sides. Hetzner preffering cheapass open and dtag preffering scalable private peerings.
Problem i see is Hetzner externalizing the cost of private peerings. Which in return plays into 1. and the example of them externalizing other cost factors.
5. You are also aware that they operate some 170 Gbit/s of private peering?
6. Which costs exactly are they externalizing with regards to peering in your opinion? So far, your argument seems to be "they don't pay as much as DTAG would like them to, therefore, they are externalizing the difference between DTAG's wishes and what they are willing to pay" - which isn't exactly how you determine externalities.
Sure, you save a lot by going east. But that means the real east, i.e. the new states.
I have coworkers in Berlin who are pitching me to move over there because it's so easy to live well there.
Edit: I think the problem with "high labor costs" is not monetary, but bureaucratic. It's damn near impossible to get people to work near 40 hours a week. It's hard to get them to do on-call shift work. It's also really difficult to fire people in .de
- It's not possible to announce a specific (premium) routing for just a single IP address.
- Hetzner has no DTAG Transit anymore. none. Not even for specific customers. See AS24940
- Current DTAG->Hetzner traffic uses AS33891. See https://f-lga1.f.de.net.dtag.de/
Of course this does not object the non-peering-policy claims and high prices of DTAG in any way. It's well known in the German ISP scene.
//EDIT: this thread goes back more than two years: https://telekomhilft.telekom.de/t5/Telefonie-Internet/Perfor...
You don't need a hoax for that. Everybody in the industry is already pissed at DTAG.
> - It's not possible to announce a specific (premium) routing for just a single IP address.
Wrong. Google source based routing. You'll get asymmetric routing, but that's rarely an issue for content providers.
> - Current DTAG->Hetzner traffic uses AS33891
That's exactly what the article states.
No. Hetzner is not able to provide direct Telekom traffic. Their offer is missleading.
Hetzner makes a profit of 36mio EUR/year. It's a shady move. Telekom sucks but Hetzner is not any better.
The article in no way states that Hetzner has a direct connection to DTAG. It very clearly states that they route the traffic to DTAG via Core-Backbone.
Now you are just outright lying and not reading the text of the article:
> This option costs € 4.20 a month for all of our servers. With this add-on, we will connect an uplink to DTAG via Core-Backbone
As someone else stated, its technically possible.
Similarly, they could require you to add an IP that is part of the block they've setup solely for customers that pick this option.
> Did anyone verify the information? I think it's a hoax to drive cusomers mad/against Deutsche Telekom.
It isn't a "hoax", DTAG does charge for private peering and their public peering is subpar.
> - Hetzner has no DTAG Transit anymore. none. Not even for specific customers. See AS24940
Its via Core-Backbone as they stated.
> This option costs € 4.20 a month for all of our servers. With this add-on, we will connect an uplink to DTAG via Core-Backbone
Core-Backbone sells private peering to DTAG. Given its an experimental setup, its not surprising Hetzner isn't committing to a contract with DTAG and going through what is essentially a reseller.
If DTAG allows it, they can accept an advertisement for a single /32 if wanted in their BGP tables, this would allow traffic to that IP address to flow from DTAG -> Private Peering-> Hetzner.
For the reverse, for traffic leaving Hetzner's network, Hetzner would have to set up source based routing which specifically sets the next hop for certain sources/destinations to the router that is peering with DTAG.
This is technically very possible, just complicated.
Bullshit. In the same breath they offer something that is entirely antithetical to their stated ethos, no matter how they spin it.
This page demonstrates what they say and what they do are two different things. I interpret this as they're making a goldbrick statement of support in favor of net neutrality with the hope that they won't face a backlash.
I will never touch Hetzner so long as they act against net neutrality.
Deutsche Telekom has been squeezing everyone for transit bandwidth. They're also the only network and ISP to NOT peer with DE-CIX. (they only added their first 10Gbps uplink just a few months ago) I've even seen Akamai bottleneck for weeks on end until they agreed to pay for more 10Gbps ports to DTAG.
What Hetzner Online is offering is merely an effective solution to alleviating a real problem. Supporting net neutrality won't fix your paying users video streaming QoS issues.
As frustrating as it is as a matter of principle, many companies will want to utilize an offering like Hetzner's.