Hacker News Reader

> my laptop could just make a request to Service X on AWS

I was looking for the "just" that handwaves away the complexity and I was not disappointed.

How do you imagine your laptop expressing a request in a way that it makes it through to the right machine? Doing a traceroute to amazon.com, I count 26 devices between me and it. How will those devices know which physical connection to pass the request over? Remember that some of them will be handling absurd amounts of traffic, so your scheme will need to work with custom silicon for routing as well as doing ok on the $40 Linksys home unit. What are you imagining that would be so much more efficient that it's worth the enormous switching costs?

I also have questions about your notion of "centralization". Are you saying that Google, Microsoft, and other cloud vendors should just... give up and hand their business to AWS? Is that also true for anybody who does hosting, including me running a server at home? If so, I invite you to read up on the history of antitrust law, as there are good reasons to avoid a small number of people having total control over key economic sectors.

> How do you imagine your laptop expressing a request in a way that it makes it through to the right machine? Doing a traceroute to amazon.com, I count 26 devices between me and it. How will those devices know which physical connection to pass the request over?

That's my whole point. You're thinking of it from an IP perspective where there are individual devices in some chain and they all need to autonomously figure out a path from my laptop to AWS. The reality is every device between me and AWS is owned by my ISP. They know exactly which physical path ahead of time will get a message from my laptop to AWS. So why waste all the time on the IP abstraction?

> I also have questions about your notion of "centralization". Are you saying that Google, Microsoft, and other cloud vendors should just... give up and hand their business to AWS?

AWS is just an example. Realistically a huge amount of traffic on the internet is going to 6 places and my ISP already has direct physical connections to those places. Maintaining this complex and byzantine abstraction to figure out how to get a message from my laptop to compute in those companies' infrastructure should not be necessary.

And in general the more important part is within AWS' (or Microsoft's or enterprise X's) network why waste time on IP when the network owner knows exactly which host every compute process is running on?

Instead of thinking of an enterprise network as a set of autonomous hosts that need to figure out a path between each other think of it as a set of processes running on the same OS (the virtual infrastructure). Linux doesn't need to do BGP to figure out how to connect two processes so why does your network?

> The reality is every device between me and AWS is owned by my ISP. They know exactly which physical path ahead of time will get a message from my laptop to AWS.

Neither of these are true in general. And suppose AWS (or GCP, or Azure, or Cloudflare...) decides to add a new POP. How do they broadcast to your ISP and all the other ISPs in the world how exactly to send datagrams to it?

> That's my whole point. You're thinking of it from an IP perspective where there are individual devices in some chain and they all need to autonomously figure out a path from my laptop to AWS. The reality is every device between me and AWS is owned by my ISP. They know exactly which physical path ahead of time will get a message from my laptop to AWS. So why waste all the time on the IP abstraction?

IP is concrete, not abstract. Whatever the form the network takes, when you make a request, your ISP is going to have a make a decision on how to route it over their physical assets to get it to the desired destination. Unless you are talking about your ISP provisioning a physical circuit directly between you and Amazon, with no multiplexing and no equipment on it, you are going to have those hops whether you use IP to choose the route or not. That is not really negotiable, or you're not describing a network (or something even remotely viable) at all. Maybe that path is invisible to you, but it exists.

And in fact, in many or even most carrier networks, this is abstracted in much the way that you describe within that particular network using MPLS. But this approach doesn't scale to the scope of the Internet, requires all edge devices have complete knowledge of every necessary path in the network, and makes inter-networking more difficult because every endpoint and its end-to-end path to every other needs to be shared and synchronized. This is actually more complex, and much more brittle, than the current implementation. And for what? I still have yet to understand what advantage you think would be gained here. Right now, if you want to talk to s3, you send a packet to s3, and your ISP does all the 'complex and byzantine' work. What do you care how they do it?

Ignoring the long tail is also silly. FAANG might represent a majority of traffic on the Internet, but the long tail is huge, and you can't just hand-wave it away like that. Enabling it is what makes the Internet what it is, and if your proposal doesn't account for it, it's dead in the water.

> And in general the more important part is within AWS' (or Microsoft's or enterprise X's) network why waste time on IP when the network owner knows exactly which host every compute process is running on?

Knowing that is the easy part. You still need to figure out a path and actually route the packets along it. You still need to deal with path selection, load balancing, fault tolerance, and synchronizing any necessary state. You still need devices along the path to know what to do with a packet they receive, somehow. It turns out that hop-by-hop routing is an efficient and viable way to accomplish this.

> Instead of thinking of an enterprise network as a set of autonomous hosts that need to figure out a path between each other think of it as a set of processes running on the same OS (the virtual infrastructure). Linux doesn't need to do BGP to figure out how to connect two processes so why does your network?

Because the 'network' you describe is not a network? It's processes running on the same machine? This is not analogous at all to a large distributed network like the Internet.

> That's my whole point. You're thinking of it from an IP perspective where there are individual devices in some chain and they all need to autonomously figure out a path from my laptop to AWS. The reality is every device between me and AWS is owned by my ISP. They know exactly which physical path ahead of time will get a message from my laptop to AWS. So why waste all the time on the IP abstraction?

The ISP can internally use MPLS to do routing exactly the way you suggest: build a "circuit" between you and Amazon and then route the packets internally in their networks through this circuit, instead of using IP. This works because the ISP has a global view of their network and as such their routers don't need to work independently. MPLS was needed in a time where IP routing was too slow, but nowadays you can get full speed without MPLS.

But anyway, it doesn't matter how packets are routed internally within each network, IP is still required for routing between different networks. Which is actually super common!

Sorry, this still sounds like architecture astronautics to me. You have an intuition that things could maybe be simpler, which is not a bad place to start, but then you have to ground it in the actual reality.

> You're thinking of it from an IP perspective where there are individual devices in some chain

What do you plan to replace the individual devices with?

Looking at the traceroute in question, something I'd suggest you do for your own, I see one router owned by me, 6 by my ISP, 2 at an interchange, 5 for a backbone, a number of intermediate ones of mysterious ownership, and finally one owned by Amazon, presumably with a bunch of other Amazon hops that are hidden from me.

These are physical devices connected by physical links (wires, cables, fiber). Wires that people installed. Wires that break. Connected to devices that break. What in your proposed grand vision will happen there? Please start with the happy path and then give detail on the failure cases.

A similar problem applies in Amazon. The abstraction they hand you is pretty convenient. But that abstraction is made out of many millions of devices connected up in cunning ways.

Linux can connect two processes because the kernel has total control over a modest number of CPUs and RAM. That just doesn't compare to literal billions of interconnected devices with no central control. It's like saying that because your dad knows your mom's name and how to reach her, he should be able to do the same thing for everybody in his country.

> So when I want my services to connect to each-other within AWS why am I still dealing with these complex routing algorithms and obtuse numbering schemes?

> AWS knows exactly which physical hosts my processes are running on and could at a control plane level connect them directly. And I, as someone running a business, could focus on the higher level problem of 'service X is allowed to connect to service Y' rather than figuring out how to send IP packets across subnets/TGWs and where to configure which ports in NACLs and security groups to allow the connection.

You shouldn't be? Doesn't AWS number your machines for you automatically and give you a unique ID you can use with DNS to reach it? And also provide a variety of 'ingress' services to abstract load balancing and security as well? I'm not a consumer of AWS services in my dayjob, but isn't this their entire raison d'etre? Otherwise you may as well just run much cheaper VMs elsewhere.

> Similarly my ISP knows exactly where Amazon and CloudFlare's nearest front doors are so instead of 15 hops and DNS resolutions my laptop could just make a request to Service X on AWS. My ISP could drop the message in AWS' nearest front door and AWS could figure out how to drop the message on the right host however they want to.

Uhm, aside from handwaving away how your ISP is going to give you a direct, no-hops connection to AWS, this is pretty much exactly what your ISP is doing. Hell, in some cases, your ISP has abstracted the underlying backbone hops too using something like MPLS, and this is completely invisible to you as an end user. You or your laptop don't have to think about the network part of things at all. You ask to connect to s3, your laptop looks up the service's IP address (unique ID) in DNS, sends some packets, and your ISP routes them to CloudFlare's nearest front doors.

There are some good arguments to be made for a message-passing focused rather than connection focused protocol model, but that doesn't seem to be what you're talking about. What you seem to be talking about is doing away with routing altogether, and even in a relatively centralized internet, that just makes zero sense. We will continue to need the aggregation layer, we will continue to have multiple routes to a resource through multiple hops that need to be resolved into a path, and we'll continue to need a way to uniquely identify a service endpoint.

The IP addresses you see as an AWS customer aren’t the same used to route packets between hosts. That said, there’s a huge amount of commodity infrastructure built up that understands IP addresses and routing layers, so unless a new scheme offers tremendous benefits, it won’t get adoption.

At least from a security perspective though ip acl’s are falling out of favor to service based identities, which is a good thing.

You can see how AWS internally does networking here: https://m.youtube.com/watch?v=ii5XWpcYYnI

This service pretty much does what you were asking for: service name and it handles the routing under the covers:

https://aws.amazon.com/blogs/aws/new-amazon-ecs-service-conn...

> Rather than being autonomous it is controlled by a single global control plane and uniquely identified within that control plane.

By default, sure. You can easily bring your own IPs into AWS and use them instead, and I don't think it's hard to imagine the pertinent use cases and risk management this brings.