Comment by lsc | Hacker News Reader

Comment by lsc | Hacker News Reader

Parent Full thread

>I never quite understood the love for premium vps like linode

This is true now (Linode is a premium vps provider) but it was not always true. Linode started out cheap, like almost everyone in our industry does. Linode was one of the first in the market; they've been selling VPSs longer even than I have.

>Its nice that they are now increasing the bandwidth cap, but it was on the low side to begin with. My takeaway from this is that "linode finally joins everyone else in network speed and bandwidth."

Eh, the way the industry works is that price pressure is applied by new entrants. Linode used to be that 'incredibly cheap provider' - but generally speaking? as you develop a reputation, you increase your resource allocations more slowly than moore's law would allow, and Linode has a pretty good reputation, so they get to charge substantially more than new entrants to the market.

So yeah, if the new entrants aren't a way better deal in terms of resources? something is terribly wrong (and that new entrant probably won't gain the traction they need to survive long enough to be able to let their prices rise to a higher margin level)

I mean, personally, I would think I have been letting my own prices rise (comparatively, I mean, by not increasing resource allocation) more than my reputation would allow me to... but I'm still gaining customers (though not at an awesome rate) - Linode announcing a network upgrade is putting /incredible/ pressure on me to upgrade my network. I can be okay with a new entrant flying below me, but Linode has a better reputation than I do; I have to provide more resources per dollar than they do.

Now, the big counterexample here is slicehost. Slicehost started out fairly expensive (compared to the industry standard at the time) and they went from zero to twenty thousand customers in like two years. It was incredible. I don't know exactly what they did, but they had a reputation for providing managed hosting level service for (premium) unmanaged hosting prices.

Now, how much of that reputation was marketing, and how much of that reputation was that their support really was that good, I don't know, but I do know that was their reputation.

Is linode announcing their own netblocks under their own ASN, though? My impression has been that you have more diversity of immediate upstreams than linode...

No idea. I'm on he.net and cogent right now; I'm working on getting more routes, but it's pretty irritating. I have been having some trouble reaching folks on comcast in the midwest, so I contacted comcast and asked about paid peering (they think they are hot shit, so there is no way they would settlement-free peer with me)

They want three grand for a 1g peering link. And they aren't on any of the silicon valley peering exchanges (meaning I'd need to pay for a connection direct to them, on top of the three grand.) that is /per month/ He.net charges around $750-1000 per month for a gigabit of transit to the whole internet.

It's pretty irritating. I mean, I'd pay a couple hundred bucks a month for reliable connectivity to comcast only, even though, traditionally, peering is free, and it would improve both my network and theirs. I mean, it was their customer who complained to my customer who complained to me.

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful.

Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgrade a peering link that is also in the middle of "nobody cares" - If I'm controlling it, I at least have a chance of doing something about it.

Also, the cost is 3x to 10x what you get direct from the super-low end big guys. Everyone fucks it up, and with two up-streams? It's my fault when it fucks up (and I can do something about it.) I guess having that preference is kinda stupid, but it is what it is; I am the network Engineer I've got... not the network Engineer I wish I had.

(that, and in my experience? competency of tech support does not scale with price paid. Neither does quality of routes.)

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down.

The opposite direction looks more problematic. Some ISPs will let you tag your routes with BGP communities that say ``when advertising to AT&T, prepend some extra copies of some ASN so that AT&T will be unlikely to send me traffic via this path.'' http://www.onesc.net/communities/as174/ does not list any communities which would affect how Cogent advertises routes to AT&T without affecting how Cogent advertises to all of their other peers. On the other hand, while 174:3001 is an extremely coarse tool, your traffic ratios may be sufficiently outbound only that sending most of your inbound traffic via Hurricane Electric might turn out OK, even with Hurricane Electric being only 1G and Cogent 10G (although this change might make your network less robust in the face of a DDOS attack, and you might want to drop 174:3001 when dealing with such an attack).

If the overloaded links are only overloaded in one direction, then you may only need to do one of these things, not both.

I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement.

Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?

I ended up learning most of what I know about BGP not via a book, which makes that hard.

But basically, back in the days of 16 bit ASNs, communities were just 32 bit values that were attached to routes being advertised by BGP, usually expressed as the two 16 bit halves separated by a colon, generally with the ASN that defines the meaning of the community being on the left of the colon, and that ASN defining arbitrary values on the right side of the colon for flags that they want to recognize.

Then, there are route maps or whatever the router vendor wants to call them that use the data from the communities.

Typically when the router is deciding what route to use, it looks for the route with the highest localpref, and if there's a tie, then it looks at how many ASNs are listed in the path, and if there's still a tie, then the router may use MED. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note... explains how this works in the Cisco BGP implementation with all of the additional factors that are considered; Linux and Juniper BGP implementations may consider a slightly different set of factors.

IIRC, Level 3 generally sets localpref to 86 on routes heard on non-customer peer sessions, and 100 on routes heard on customer BGP sessions, but if customers want localpref to be 90 or 80 or 70 instead, they can send community 3356:90 or 3356:80 or 3356:70. If you had two links to Level 3 and wanted to use one as backup only, you could send community 3356:90 on your routes on your backup link, and potentially no communities on your primary link. If you had a really slow link to Level 3 that was backup only and some other ASN was your primary, then you'd want to send Level 3 3356:80 or 3356:70. I doubt you have any desire to set your upstream's localpref values in your configuration, but this may shed some light on why they offer it to their other customers. Also, if you ever have customers speaking BGP who want to use you as backup only, you need to offer some way for routes they advertise to you to have localpref less than what you set on routes you learn from your upstreams; typically this seems to be done by letting your customers send you a community so the customer controls this, but you could just have a special route-map for customers who are backup only that would just set all of their routes to low localpref regardless of the BGP communities, and just apply the right route-map to those BGP sessions.

http://www.onesc.net/communities/as3356/ mentions that Level 3 supports 65004:[ASN] for four prepends when advertising to ASN. If you were a Level 3 customer, and traffic coming to you via one of their peering links was overloaded, sending a community of that form would be a good way to discourage traffic from taking that link by making the path have a large number of ASNs and thus look undesirable. I say this mostly to illustrate the sort of thing that it looks like you might find useful for Cogent to support, though it appears that Cogent doesn't support that.

It's likely that Hurricane Electric and Cogent will recognize some communities that you could send them that will influence their routing, but as far as I know, their peers won't necessarily end up accepting any communities from you, so the key is to either find documentation on the web of what communties Hurricane Electric and Cogent recognize, or ask them for it.

You can also sometimes get geographic data about which major city is closest to a network block by recieving community data from an upstrem, but it's not obvious to me why you'd want that. It may sometimes be used when hot potato routing is not desired, or when a CDN is trying to identify which geographically close server to use.

Also, playing with looking glasses can be helpful in understanding what's going on with BGP from the perspective of other networks. http://www.bgp4.net/wiki/doku.php?id=tools:ipv4_looking_glas... has a list, which doesn't seem to include AT&T. Going to http://www.nordu.net/connectivity/looking-glass/lg.cgi and selecting bgp and entering prgmr.com's IPv4 address isn't very exciting, in that it only lists one ASN in the path (it seems they have direct peering with Hurricane Electric, who is announcing the block that includes that IP address under their ASN). Putting in an IP address advertised under your ASN at the looking glass of a network that peers with both Hurricane Electric and Cogent might be more interesting if you manage to see routes via both.

Also, do a Google search for

bgp localpref

and at least skim some of the articles that come up there and some of the things linked to from those articles.

Also, you probably want to pick some community to affix to all routes you learn from your customers / originate yourself, and to make sure isn't affixed to routes you learn from your upstreams, so that you can then have a route map check for the presence or absence of that community when deciding which routes get passed along to your upstreams. (BGP doesn't directly solve the problem that an AS doesn't carry traffic unless the source and/or destination is ultimately a customer of that AS, it just provides mechanisms that can be used to make that happen.)

You may want to define communities to let your customers selectively decide to suppress announcements to each of your upstreams, and you may want to pass through the communties recognized by your upstreams in their official documentation if your customers send those communities to you, if you want to let your customers control their advertisements in that fashion. (The only reason I can think of you wouldn't want to let your customers do this is if it would let them move traffic from a link that's cheap for you to a link that's expensive for you.)

Do you have traceroutes that demonstrate that the path goes through Cogent and AT&T in both directions? Asymmetric routing is extremely common for things like this.

not now. Next time I get a complaint like this, though, I will attempt to document the debug process in more detail, in public.

http://en.wikipedia.org/wiki/Peering has a list of historic peering disputes, but doesn't really seem to have a good pointer to the details of the Exodus dispute.

Typically, national backbones use hot potato routing when peering in lots of different cities. If you, in Northern California, on Hurricane Electric, have a video chat with someone in New York City, let's say on Verizon, and it turns out to really be peer to peer, what typically happens is that the computer in New York City will send the packets through Verizon in New York City, to Hurricane Electric in New York City, and then Hurrican Electric carries the packets to you in California. In the reverse direction, Verizon gets to carry the packets across the country.

A key point here is that the two providers are equally responsible for the hassle of carrying packets across the country. This is why providers sometimes care about balanced traffic ratios.

Exodus was mostly pushing traffic out, not recieving very much traffic, and so with hot potato routing, they were doing a lot less than what many would argue was their fair share in covering costs of hauling packets across the country, which seems to have been the basis for that dispute.

When traffic ratios are balanced, hot potato routing has the advantage that one network doesn't need to know the details of where another network's individual IP blocks are geographically, thus reducing the amount of information big networks have to exchange and keep track of.

If Comcast were to peer with you near your location, they'd have to eat the costs of hauling the traffic across the country.

http://he.net/peering.html is Hurricane Electric's peering policy, and basically, anyone who's comfortable with settlement free peering is likely already going to have a good connection to Hurricane Electric, and my impression has been that Hurricane Electric's backbone is pretty sufficient for whatever long haul traffic it has to carry.

(And for all that Hurricane Electric claims ``Hurricane Electric expects you to peer at all exchange fabrics we have in common if we peer with you via public exchanges or all facilities we have in common if we peer via private interconnect with you. In other words, if you are peering in the US, Asia and Europe at the same locations as Hurricane you must peer with Hurricane at all those locations. This is prevent problems for your and our US and European customers so that customer traffic does not cross the Pacific or Atlantic twice to return back to the same continent.'', if a lot of their customers are data centers full of web servers, it's possible that hot potato routing might get them out of a certain amount of ``their fair share'' of long haul transit.)

Have you concluded that both Hurricane Electric and Cogent have trouble reaching those Comcast customers, and have you discussed the issue with Hurricane Electric and Cogent? It seems like they'd have better economies of scale in upgrading connectivity to Comcast than you would have going directly.

If Comcast is not directly a customer of he.net or cogent, then it is likely that each has multiple routes available that they could use to send traffic to Comcast. Have you explored whether getting them to pick a different one would improve things?

Or is the problem in the direction of Comcast to you, which is harder for you to control?

Perhaps it is because peering is not an entitled right. If it was, who would pay for all the costs of network growth?

AFA HE.net pricing, the reality is HE.net does not sell "their" network... they sell other ISPs network via their peering. Their costs are far lower as they don't carry the traffic more than 1-2 router hops.

HE.net does this so much that their peers are feeling that the relationship is no longer "fair" and are not interested in upgrading capacity to enable this business relationship. If HE.net had to pay as much as their peers for the network costs, the prices would be similar.

You get what you pay for....

>Perhaps it is because peering is not an entitled right. If it was, who would pay for all the costs of network growth?

My complaint here is that comcast charges as much for peering as for transport (with full routes) - I mean, if that's the case, why advertise paid peering services? Every time I've bought full routes I've had the option of asking the provider to only send me customer routes, or to filter on my end for customer routes. Hell, most people will give me a feed with customer routes, and another with full routes, so I could use them as a de-prioritized transport provider of last resort as well as gaining all the benefits of being a peer if I bought transport from them, so yeah, advertising paid peering at transport prices is just silly.

My second complaint is that they aren't on any peering exchanges. Peering exchanges are important because individual interconnects in a datacenter are expensive; the data center charges you a hefty monthly fee for every connection you have to other people at the same datacenter (like half what you'd pay for a shitty but local room in a shared house) /and/ then if you use direct interconnects both parties need to maintain a switch port.

With a peering exchange? you maintain one switchport for all your traffic that goes over the exchange. There are even route servers you can subscribe to if you want to automatically peer with everyone else on the route server at that exchange, making peering a nearly zero work, zero cost thing.

(Of course, if you are sending a /lot/ of traffic, then a individual link probably makes more sense, sure, but I'm not sending a lot of traffic.)

>If HE.net had to pay as much as their peers for the network costs, the prices would be similar.

But they are similar. after negotiation, he.net is more expensive than Cogent. to the tune that commits on he cost the same as overages on Cogent.

HE.net, for me, is actually pretty competitive with Cogent, who I think is their closest competitor, quality-wise. After negotiation, Cogent was slightly cheaper. (The interesting thing about he.net was that they started with a number that was fairly reasonable and stuck with it. It was clear that I didn't have to spend three months dicking around to get the real price, which I appreciate.) Cogent started off asking about as much as I'd expect to pay for L3 after negotiating well, but came down hard.

>You get what you pay for....

If you believe that... well, I've got some Premium VPS here for you. just $100/month per gigabyte ram. They are "premium" - way better than the regular stuff. I'll even wear a suit while I provision you.

People are always willing to charge you more for a shitty product. Always. Professionals spend huge amounts of effort getting you to pay more for the shitty product. (More for the good product, too.)

Hell, in the bandwidth world? It's completely normal to end up paying 1/5th the asking price after negotiation. Yeah, 20% of what the salesperson initially quoted is about what I normally end up paying for transit.

Do you think service is different for people who pay the asking price vs. people who negotiate down to the real price?

I mean, sure, buying transport from L3 is going to get you better connectivity than buying you transport from Cogent, and L3 is generally going to cost you more than Cogent. How much more? with my negotiation skills, about 10x more, but I only got L3 down half from their asking price, and I got Cogent down to 1/5th their asking price; I spent considerably more time on Cogent because I thought that was, well, more realistic, and at the time, Linode, the people I see as my competitors to beat, were mostly he.net, which, quality-wise, is on par with Cogent (I think he.net may be a bit better, and there is some argument if cogent or he.net is slightly better, but I think most people would agree that they are both near the bottom of the pile.)

So yeah. uh, I guess there is some relationship between "real price" and quality... but it is not nearly as close a relationship as people say.

Hurricane Electric seems to have built their network by installing a router at every major peering point in the world, connecting each router to the local public peering point switch, and then interconnecting those routers. If you want reasonably direct connections to all the world's tier 2 ISPs, Hurricane Electric seems to be an excellent way to get that, and as long as they're not charging too much, they may make actually showing up at the major peering switches yourself not worth the trouble when you can just be their customer and get connections to more peering points than you'd probably directly connect to yourself.

Cogent has a fairly substantial network directly into random office buildings (though if you have an office in a random office building, it's almost certainly not one of the ones they've lit), and it's not clear to me that they do as good a job at peering with other networks as Hurricane Electric does.

No. Linode use IP space provided by the upstream they use at each location.

That was true in the past, but more recently they've been using their own IP space allocations, but having their upstream announce it as part of their (the upstream's) AS.

See: http://bgp.he.net/search?search[search]=Linode&commit=Se...

It's entirely possible to get IP space from one upstream provider and then announce it to multiple upstreams under your own ASN.

Reply on news.ycombinator.com