Hacker News Reader

Comment by sevensor | Hacker News Reader

I’m still a bit unclear about how an IX is situated relative to the internet and to end users. Per the article, it’s not meant to have desktops or print queues plugged into it, but it’s also a LAN. What sort of computer is meant to participate in an IX?

View on HN

The idea of an IX, or IX peering LAN is simple in concept. It is a LAN (a flat, layer2 network), to which multiple ISP's can plug in routers.

Like your home LAN might have 192.168.0.1 = router, 192.168.0.2 = laptop, 192.168.0.3 = phone etc, a peering LAN will have things like 195.66.224.21 = HurricaneElectric, 195.66.224.22 = NTLI, 195.66.224.31 = Akamai, 195.66.224.48 = Arelion etc ...

So instead of all these ISP's that want to exchange traffic with each other having to assign ports and run cables in a full mesh (which quickly would get out of control), everyone connects to the "big switch in the middle" with that peering LAN on it, and they use that.

Back in the day, that might have been an actual single big switch, or a stack of switches. Now IXP infrastructures are much more complex, but the presentation to the end user is usually still a cable (or bundle of cables) that goes into something that looks to them like a "big switch".

There is a LOT more to know about this space (Peering vs Transit, PNI's, L3 internet exchanges, what Google are doing by withdrawing from IXP's), but I wanted to write a comment that didn't turn into an essay.

An IX is the internet, or at least a small part of it. It's sort of a network where each device shares their "routes" with the others, and then propagates those routes back through their own network. They're exchanging their respective pieces of the Internet with each other.

The devices themselves are just routers, though much larger and more complex than what you'd see in a house or office. Instead of one route, "put all the traffic through eth0", they'll have hundreds, thousands, or millions of routes depending on their location relative to the "rest of the internet".

It's a big peering LAN (not dissimilar to a LAN party, but you know, with equipment costing more than a nice car), scaling from a switch in a rack somewhere (eg. FCIX [1]) up to hundreds of switches in dozen of locations not even in the same city anymore (eg. DECIX, AMSIX).

You connect to the IX either over a cross connect cable from some equipment nearby (in the same DC or same building) or across town via some leased line / dark fiber / lambda / etc. But usually what's at the other end of a connection to an IX is your edge router, on which you will then run BGP to all the other folks on the IX.

However, it's not always this simple - for example, you might have another switch-like device in between the router and the IX, to maintain some level of flexibility for your own services, or because you're actually being bundled together alongside multiple customers into IX access by some provider. You might actually go into a whole MPLS backbone first, because that's how your provider is selling you transport to the IX. Or you might've set up some peering LAN bridging on your router to set up some hot standby and then plugged it into some switch for convenience to run it across the office LAN to your desk and then...

What ends up happening, is with this amount of complex network devices along the way, and with how network equipment is generally provisioned (you SSH into it and then you do some mutable changes, then you remember the update the orga docs), mistakes happen.

In a critical case of misconfiguration (also stale configuration and VLAN identifier reuse) it's not unimaginable for an IX peering LAN to then get accidentally bridged into some VLAN that actually reaches an old Windows laptop that was at some point plagged into another VLAN for troubleshooting. This is especially likely for customers that co-locate their office equipment, AD servers, web servers and edge router in the same building across the same infrastructure.

[1] - FCIX is also a good source of how this sort of stuff actually looks like IRL, instead of the bullshit marketing 'rack of neat racks' renders on providers' websites: https://pbs.twimg.com/media/FIsd-JsVgAQLUh9.jpg?name=orig https://pbs.twimg.com/media/FP72xaiWQAsyPUM.jpg?name=orig

Ideally, participants in an internet exchange purely exchange routing information, usually using the Border Gateway Protocol (BGP). This implies that devices connected to an internet exchange are routers. These devices receive routes for other autonomous systems (AS), and (selectively) publish their routes to other parties on the internet exchange.

(Internet exchanges typically offer a route server, such that every participant of the IX can easily publish routes for other participants, and simultaneously receive published routes of all other participants.)

The _effect_ of exchanging routing information is that IX-local participants know where to send traffic destined for certain IP ranges from other participants.

An autonomous system internally "knows" where each of their routers are located, and all these routers are typically connected with each other. When several routers of an AS are connected on different IXes, this means they can take informed decisions on where to send traffic destined for other ASes. It could be that AS 64496 is only present in IX-A, while AS 64510 is only present at IX-B. Suppose AS 64499 is connected to both IX-A and IX-B, traffic sourced from AS 64499 (e.g. endusers or "eyeballs") and destined for either 64496 or 64510 knows, through internally exchanged routes, where to send that traffic.

Scale this to even more autonomous systems and IXes in different geographic reasons, and you'll find it becomes a network of networks, or: the internet.

>Ideally, participants in an internet exchange purely exchange routing information

I would hope they exchange Internet traffic as well.

That really depends on local policy. It could be that an AS x present on multiple IXes (assuming another peer AS y is present on the same IXes) prefers sending traffic for AS y on IX t, while AS y prefers sending traffic for AS x on IX v.

Of course, the goal is to transport traffic in many directions, but in essence only routing information is explicitly exchanged.

In an IX you have mostly routers, firewalls and servers. But sometimes people will need to connect a "personal" device to the internet or the internal servers network, and instead of using a NATed/firewalled zone, they will use a public internet port (mostly by mistake), and then that kind of stuff can happen.

https://en.wikipedia.org/wiki/Internet_exchange_point

it's a network for ISPs and organizations with big networks to hook together to let traffic flow more directly, mostly only routers talking BGP

Say you're trying to go to google.com (dig resolves this for me right now as 64.233.177.100). Your packet leaves your laptop and travels over wifi to your access point, where it goes into a switch. The switch sees the IP, recognizes it's not part of your LAN (because you've set your network up as 192.168.168.0/24 and that IP's not part of that range) and forwards it to its gateway, the router. (Often the AP, switch, and router are colocated in one device, which can cause shortcuts.) The router forwards all its packets to its upstream, which depending on if you have fiber, cable, etc may mean a modem or something first. Say it goes to a modem, where it's put onto a coaxial cable, the other end of which is at your ISP.

Now the packet is in the ISP's network. Logically this will be a big pile of switches with some servers attached, such as CDNs and the like, and connected at points to other ISPs/backbone providers at IXes. Physically it's a bunch of neighborhood racks feeding to ISP datacenters scattered around the country and interconnected wither by the ISP's own cables or backbone providers. Your packet will traverse this network in a number of ways depending on the specifics. Maybe the ISP's switch recognizes that 64.233.177.100 is a CDN on a server in the ISP's datacenter, so it forwards it there. Maybe the address is outside of the ISP's network. The switch that your modem is plugged into will pass the packet along to a router of some sort which will have a routing table (which is distributed and updated by internal routing protocols like OSPF or IS-IS) which will tell it where to sent the packet. If the address is outside of the ISP's network, then the router will direct it to one of the IXes based on its routing information.

Now we're at the IX. Your packet has gone from your laptop, to your AP/switch/router, to your modem, to the ISP's switch, to the ISP's router, to several other ISP switches and routers, and is now at the ISP's router in the IX. The physical configuration of an IX is like any other datacenter: just a bunch of racks, this time with routers in them. There'll be Comcast's routers in one rack, Cox's in another, Lumen (a backbone provider) in another. Then each router will be cabled to each other router, just like you plug your desktop into your router at home. The routers will exchange routing information with BGP, and that information will include who has address 64.233.177.100 connected directly to their network, or if nobody has it then who has a connection to the network it's directly connected to, or if nobody has that then who has a connection to a network that's connected to the network that 64.233.177.100 is connected to, etc. Like your ISP's network, each other network is a logical mass of switches and routers, and a physical connection of datacenters, neighborhood routing racks, and IXes.

Maybe in this case 64.233.177.100 is in a datacenter connected to another ISP at the IX; in that case the packet will go to that ISP and through its internal network to the datacenter. Or maybe nobody connected to this IX has 64.233.177.100. In that case the packet will probably travel through one of the backbone providers' internal network to the IX where the ISP that does have 64.233.177.100 connected to it is connected.

Since physically an IX is just a bunch of routers in a rack, you can have things plugged into them other than other routers, although this is generally frowned upon. Many IXes have locked racks to prevent that

> The switch sees the IP, recognizes it's not part of your LAN (because you've set your network up as 192.168.168.0/24 and that IP's not part of that range) and forwards it to its gateway, the router.

This may be nitpicky, but assuming we’re talking about a switch in the strictest definition (a layer 2 switch), this is not correct. Your computer sees that the destination IP address is not in its local subnet, and addresses the packet to the MAC address of its default gateway (the router). The switch receives the packet and forwards it to the appropriate interface based on the destination MAC address.

Even if we are talking about a layer 3 switch, then we would be assuming that the gateway resides on the switch, and it is still the computer that makes the decision to send the packet to its default gateway.

Given the rest of your comment I’m assuming you already know this, and I’m not posting this as a correction to you, but rather for the benefit of others.

Reply on news.ycombinator.com