Vint Cerf on mistakes he made in TCP/IP
spectrum.ieee.org
spectrum.ieee.org
[1] https://youtu.be/mZo69JQoLb8?t=815 "Google IPv6 Conference 2008: What will the IPv6 Internet look like?"
Test the thing you want to test!
1. Not scary like fear, just like, I don't know, astonishing.
It's hard to see the future.
You don't treat this by whining that the prototypes are winning. You treat it by finding ways to cleanly evolve prototypes[1], and by collecting intuition (c.f. the linked article) about common mistakes and how to avoid them.
[1] Once upon a time in an age lost to history, this was the idea behind something called "agile". The thing we call "agile" today is pretty much the opposite.
I forget the name of it (there are probably several by now), but there's a UX prototyping toolkit where all the components look hand-drawn, so that nobody expects to be able to interact with them and then gets mad when nothing happens. It also prevents you-the-designer from being tempted to add interactivity logic at design time.
There's nothing stopping someone from creating a programming language that works the same way — something intentionally constrained to e.g. run on an abstract machine that uses a 24-bit address space (and builds this into the design, using the other 40 bits of each pointer on 64-bit VM implementations for other important info) so that only prototype use-cases can be implemented and tested, while the system would inherently be unable to reach any kind of scale serving multiple concurrent customers.
While I don't think anyone's ever tried creating a runtime like this for general-purpose software prototyping, I know of at least one domain-specific example:
The PICO-8 (https://www.lexaloffle.com/pico-8.php) is a game runtime that's kind of like this, designed to put constraints on game development that resemble, but are orthogonal to, the kind of constraints imposed by old consoles like the Gameboy. (For example, it is programmed in high-level Lua, but has an 8192-lexeme source-code size limit, to achieve similar complexity constraints to old games that needed to fit their object code on 16k ROMs, while not forcing you to compile to/write in an assembly language, nor impacting source-code readability.)
So I wrote some scripts in bash to do things the "worst" way possible. Directly accessing the production DB, minimal safety checks, etc. Nobody else on the team liked writing bash, so there was no fight to replace these awful tools with proper APIs.
In practice, in many industries, prototyping tools are used as the first step in the design process. The constraints built into these prototyping tools force those using them to spend less — often orders of magnitude less — time, and effort, and money(!) prototyping, than they would if they allowed themselves to prototype with production-oriented tooling.
Consider painting. A painter — unless they're working off of a photographic reference — will almost always draw a pencil sketch of the scene they want to capture in paint, before they begin the actual painting. A pencil sketch has no need to consider color (or how to mix to achieve particular color effects), or light and shadow, or dimensionality (how real light reflects off of built-up paint on the canvas), or any of that. They just need to concentrate on proportion, perspective, anatomy, etc. The sketch focuses only on (a subset of) the design of the painting, while inhibiting any of the implementation details specific to the medium of paint, from being worked on. Which means the sketch only takes a few minutes, rather than days.
Once they have this sketch, they can show the sketch to the client who commissioned the painting (or to the master of the studio, if you're painting for gallery sale), and the client/"product owner" can point out places where the sketch does not align to their vision for the painting, which can be used to iterate on the design.
There are many other things to get right once the "real" painting starts, but if you don't get the "bones" of the thing right, the client won't want the painting. The sketch lets you evaluate just the "bones" of the painting before even considering the meat on those bones.
A prototyping tool for programming should be the same: something to let you consider the "bones" of business logic, preconditions/postconditions, etc., without the "meat" of the particular library APIs and data-structure juggling required to glue things together and achieve scalability in a particular language.
The best prototyping tools are ones that pare down your focus to the smallest useful kernel of design, and thereby allow you to iterate in near-realtime. An experienced painter will become skilled enough at making sketches, that they can sit down with a client and sketch in response to the client's words, changing the sketch "interactively" in response to the client.
In fact, some prototyping tools are streamlined enough to allow the client themselves to iterate and ideate on the sketch by themselves; and then only submit it to the productization process once they're happy with it!
Game-development example again: RPG Maker. As far as I can tell, RPG Maker as a software product was never really expected by its vendor to be used in the production of commercial games (although it has been repeatedly marketed that way.) Until very recent releases in the series, it was far too constrained for that — unless you entirely eschewed most of its engine [as most of the RPG Maker "walking simulators" like Yume Nikki do], it gives you a very static set of engines: battles that work exactly one way, menus that work exactly one way, etc. A tool that's actually for building role-playing games as commercial products, would have an almost-monomaniacal focus on letting you customize these systems to make your game distinctive; but RPG Maker is entirely the opposite. Rather, I believe that in its idiomatic use, RPG Maker has always been intended as a tool to allow a client to "sketch out" the narrative(!) "bones" of an RPG. That's why it gives you so many built-in assets to work with, but also why these assets are so generic, and also why ASCII/Enterbrain never created an "asset store", nor documented the asset formats: the assets are meant to act essentially as wireframe components. You're not meant to re-skin an RPG Maker game; you're meant to just use the generic assets to build a generic-looking game, because the look of the game isn't the point, any more than the look of a pencil sketch is the point.
And they repeated the same mistake with the IPv6 design by not making it an evolution of or backwards-compatible with IPv4.
Literally neither of these was built with scalability in mind.
And in case it wasn't clear enough: it's called IPv4 because the value of that version field is 4, and it's called IPv6 because the value of that version field is 6.
To me this sounds like the winner's bias: in hindsight.
It is hard to balance a prototype against a "works under all conditions no matter what they are" production-ready solution without growing into a hard to handle monster solution. On the other side I know off quite some overhyped apps, that rest on a overpowered machine, because the specs did not account for one thing: no one cares about the product.
Extensibility is the key to balance certain weaknesses in a spec.
- If it fails: welp, we were right to not spend months on this discussion.
- If it wildly succeeds: welp, it's now too costly to change for so many users.
Having a mild and steady usage growing curve is a blessing that's too often overlooked.
IPV4 address contention has a bit more real-world impact.
http://catb.org/~esr/writings/taoup/html/ch15s04.html
Which "certain type" are Stuart Feldman and Steve Johnson?
The growth ended up to be exponential over time. So maybe a better margin would be expressed exponentially along the lines of "build it for 2^(doubling_constant * number_of_years) the load."
In this case probably the right number of address bits would have been 16. That would be big enough to to build test networks (real or simulated) with more hosts than were currently on ARPANET, but small enough that it would not take too long for ARPANET growth to hit the limit.
With 32 bits the running out of addresses problem is far enough in the future that someone choosing to go ahead and put the experiment in production is not making a problem for themselves. They are making a problem for whoever has their job long after they are retired.
With 16 bits the running out of addresses problem is soon enough that they can see that it might be something they might have to deal with.
It’s astronomically larger than 32, not double.
IP6 didn't add one byte, it added 12! I think 64bits is plenty (as mentioned astronomically larger than 32), and still looking for reasons to change my mind.
IPv6 is _dramatically_ different. Its actually possible to get a /48 or a /56 - sometimes just for free as part of general operations. That leaves 8 or 16 bits for the customer to create hundreds or thousands of subnets. Unlike with IPv4 where most customers don't get more than one IP address, you don't have to use a NAT setup if you have a /48 or a /56. Even if you only get a /64, that still leaves 64 bits to give every device its own globally routable address without having to setup NAT.
Do we need 64 bits to identify individual devices in a subnet? IDK, maybe not. but if you are doing an apples-to-apples comparison, with IPv4 you have 32 bits for global routing. With IPv6 you have somewhere between 48 and 64 bits. The remaining 64-80 bits of the IPv6 address don't have a good analog in IPv4. So, in a lot of ways, IPv6 is a lot like taking IPv4 and expanding it to something between 48 and 64 bits and then the remaining bits you can think of basically almost like extra fields to encode information that IPv4 doesn't support.
Is 64 bits enough? Maybe. Maybe not. But if you're wanting to future-proof, then setting an arbitrary limit is not the way to go.
32bits is just about large enough (within an order of magnitude sense) if the space was used more efficiently.
128bits I've heard described is like "every atom in the universe" big. If so, then 64 is probably enough for every atom on Earth.
Now I've just thought of another angle, similar to UUIDs. They are used because they can be assigned randomly without worry of collision. But I don't think IP6 addresses are being assigned randomly, hmm.
Well, not really. Between just the populations of the US, Europe and China (places with high levels of internet connectivity), you have over 2 billion people (this site claims over 5 billion internet users: https://www.statista.com/topics/1145/internet-usage-worldwid...).
> But I don't think IP6 addresses are being assigned randomly, hmm
That is, in fact, exactly how IPv6 addresses are assigned using SLACC with Prefix Delegation. Your ISP assigns you a prefix, and your computer randomly picks an address within it. You can also self-assign a (non-routable, like 10...*) prefix from the fc00::/7 ULA block by randomly filling in the remaining 57 bits to form a /64 subnet.
You have to consider the context: back then, multi-user computers were common. Each user didn't have their own computer; instead, they had a terminal to connect to a central computer. So a single computer would serve tens or hundreds of people, and as computers became more powerful, you could expect each computer to be able to serve even more people.
Sure, if you want your refrigerator, oven, and dishwasher publicly addressable on the internet it isn't enough, but you don't actually want that.
Further, 64 bits is many orders of magnitude overkill already. So what does 128 bring to the party, besides making addresses harder to type?
Also, random with a prefix is not really random.
Earth mass divided by 2^64 is roughly 357 tons. There are roughly 2^33 humans on earth, so 2^64 is "only" a billion or two addresses per person; tha's far fewer addresses than the number of human cells out there.
Because then you don't really have enough bits to do some nice things:
1. It would be nice for your upstream network provider (and you) if they can delegate some network prefix and thus don't need to concern themselves with the address plan inside your subnet. That means, if there is an address collision on your prefix it's not their problem.
2. Assuming you have been delegated a network prefix, having at least 48 bits for within-subnet addressing greatly simplifies self-assigned addresses:
2.1 You can use the data link layer's address (the 48 bit MAC address for ethernet) which often has weak guarantees of uniqueness (though this isn't really advised today)
2.2 You can also randomly generate addresses or generate them by hashing things, and be reasonably confident of not colliding with another station.
2.2.1 Of course, you want to check that no other station is using the same address, but there's always the hidden node problem: How does station C cope with stations A and B both claiming address XYZ?
2.3 You can (if you want) still use DHCP to assign addresses if you want. But you don't have to.
3. It would be nice if each station was not limited to a single address.
3.1 Among other things, having multiple addresses vastly simplifies building a multi-homed network. For example, you can (today) sign up for two (or more!) ISPs that support IPv6 Prefix Delegation and have your router(s) issue Router Advertisements (RAs) for each prefix.
If a link goes down, your router(s) don't need to do anything or share any state... it just works (yes, really, I've tried it!). Each of your routers, of course, only routes packets for the prefix it was delegated by its upstream.
(BTW, IPv6 also has some neat RFCs for letting foreign hosts know about a prefix change, so you can even transfer existing open connections initiated from an address on prefix A through a different router with connectivity on prefix B)
3.2 For privacy, wouldn't it be nice if you could generate a new address for every external server you connect to? You can do that with the 64 bit subnet address space--remember, your ISP has no say about the address plan within your network.
4. With 128 bit addresses, IPv6 allows you to delegate a /64 to yourself from the ULA block (the IPv6 equivalent of 192.168.. or 10...). You can just randomly pick a 57 bit suffix to append to fc00::/7, and you can be pretty darn confident that nobody else will pick the same one. And you get all the same advantages of self-assigned addresses (mentioned above) within that prefix.
4.1 Having a unique* local prefix maybe doesn't sound like such a big deal until you've tried to merge two enterprise networks that both have hosts on 10..., and clients hard-coded to connect to those hosts.
Finally, think about how a 64 bit global address space would be split up:
We're running out of IPv4 addresses, even with* NAT (and "CGNAT", which is just regular NAT with a fancy name). Most of those endpoints are users, not servers--you need at least 30 bits to give one prefix to each household with a current internet connection. Between the expected growth of internet users and all current enterprise networks, you need well in excess of 40 bits for the prefix.
So you'd realistically get no more than /48 prefix at home (16 subnet bits). That's not enough to do the cool autoconfig and privacy things I mentioned. More likely, you'd get a /56 because 256 hosts "ought to be enough for anybody" and you can't do the cool stuff anyway.
As a user of OpenWrt (which does exactly this) I strongly disagree that it is the right solution. If you announce on the LAN the prefixes obtained from a fast fiber and a slow LTE link, then devices will, well, get an address for each prefix. The problem is that they have absolutely no information to choose the source address correctly - it's all just numbers! So, just by bad luck, they choose the source address from the LTE prefix and waste the LTE data and my money if I use the default setup. Which is why I don't. My network uses IPv6 NPT, but announcing one prefix at a time would have been even better (because I only want fail-over, not real multihoming), although impossible with OpenWrt.
Forum discussion, which also serves as a proof that it is really hard to explain the "don't overcomplicate fail-over by generalizing it to multi-homing" notion: https://forum.openwrt.org/t/ipv6-wan-fail-over-without-ipv6-...
IPX had a lot of convenience features in a similar number of bits. (80, but 24 were wasted on manufacturer, so 56 useful bits)
A rotating address number will not provide privacy if the prefix attached to you is the same. Would be like saying a rotating port number would provide it, but not the case.
No organization will ever need a /64 (not even close, not even wastefully-that's 18 quintillion).
The slow uptake in IPv6 seems to imply that it's over-engineered and people don't care about these potential additional features. I'm a lifelong geek and can barely get interested. Network engineers are maybe .01% of the population.
I guess bits are only getting cheaper in the future so why not spill them incredibly wastefully at anything we can think of, is ultimately the answer. Although it doesn't answer why 64bits was not even an option.
uhm. counterexample: they need more dollars than that
TUBA (replacement of IP with CLNP, running TCP and UDP on top) mandated use of max size addresses always, to make it simpler to decode, as part of "Internet Profile CLNP".
Vint mentioned the lack of encryption as a mistake, but even on private networks, the 16-bit TCP checksums are too weak to protect against bit-flips. Many orgs noticed that enforcing encryption reduces "random" crashes and errors, because it also enforces a strong checksum, typically 128 or 256 bit. However, even a 32-bit checksum would have sufficed for ordinary use.
Almost all "real" protocols built on top of TCP eventually end up reinventing the same wheels: message boundaries, logical sub-streams with priorities, and strong checksums or encryption. Examples include Citrix's ICA, HTTP 3.0, gRPC, etc...
By being just a stream of bytes you are free to do whatever you want with the protocol. Sure that may mean implementing your own form of blocking but that's better than being stuck in a protocol where you are forced to work around the message oriented features even though you are just streaming data.
The checksum does hail from an earlier era where 16 bit machines were commonplace and the total amount of data sent is minimal. Also, there is some assumption that the underlying transport is going to have its own checksums. You will note that IPv6 ditches the checksum entirely.
This seems like nearly nothing, but when you think about the lost compute power and network bandwidth throughout the history of TCP to this distinction, it's actually a very significant chunk of power, CO2, and wasted human life. Not to mention the lost human life that comes from HOL blocking.
The byte stream abstraction idea, I think, was a suspect decision at the time (born out of a desire to look exactly like a serial teletype port despite the fact that it's the wrong abstraction layer to use for that), and clearly wrong for a transport layer today. Almost everything built on top of TCP re-introduces messaging semantics.
Also, IP never had a message checksum, only a header checksum. The checksumming is at the TCP layer and it is still alive.
IMO something like SCTP would have been better. You generally want multiple independent streams (to avoid head of line blocking), message and stream transmission, and optional reliability on a per-stream/message basis.
This sort of thing is very very common in real time games (e.g. enet). Actually enet is built on UDP, but if you compile it to WASM then the Emscripten UDP implementation uses the WebRTC data channel which uses SCTP!
Not entirely. Head-of-line blocking is an inherent problem with TCP so you can’t have real priority “streams” like you can with eg QUIC. Another “missing feature” is datagrams, ie simply opting out of the retransmission part of tcp.
That said, I still agree with the general statement.
> As a result of implementation differences and middlebox interactions, new applications SHOULD NOT employ the TCP urgent mechanism. However, TCP implementations MUST still include support for the urgent mechanism.
Of course, basic ftp:// clients (the only kind that are really used today) would probably rather just close the control connection, which has the same effect.
It turns out that what people wanted was application protocols, not transports.
This is why HTTP(S) is basically the only protocol used inside and outside of the data centre these days. Even storage protocols are migrating towards HTTPS, with the public clouds using S3-like protocols on top of HTTPS even for virtual disks.
It's ironic that in an effort to only allow web traffic on the network, these administrators have instead made it impossible to block anything else, because their actions forced all of those other application to disguise themselves as web traffic.
I personally stopped using raw TCP because it is almost always the wrong solution. If I need a bare bones protocol where I want to send raw bytes I now always use websockets because they work exactly as they should.
Streaming is such a niche usecase that is only natural if you don't semantically interpret the data, i.e. arbitrary file transfers but the vast majority of the complexity isn't there.
> It's easy to add features to a simple transport, it's not so easy to work around features when you don't want them
No, in my experience, you will spend most of your time working around TCP warts.
In the context of 2023, this is a defensible position. I can quibble, but it's defensible.
However, are you sure you want a 1974 take on a "message based protocol"? Trust me, you're not getting SCTP out of that, in a world where having two streams instead of just one is a noticeable resource impact. Heck, even by the 1990s there were still a lot of really bad things coming out of the world of "message based protocols".
You wouldn't be here in 2023 with a message-based TCP. You'd be here in 2023 with some other protocol having become the foundation of the Internet, and it wouldn't be "message based" either. There's no option where someone in 1974 is building with the nearly 50 years of experience that haven't happened yet, and I'm not going to grade TCP on that basis personally.
It doesn't seem like you've actually moved away from using TCP.
Maybe it's just too low of a layer for your needs?
TCP and IP used to be one thing, and they split off TCP in what we now call Layer 4 (where UDP, and others, also exist). So they did think ahead in separating things to a certain extent.
The fact that we have a whole bunch of middle-boxes that don't allow DCCP, SCTP, etc, is hardly the fault of Cerf et al.
Byte streams are not ideal to deal with for protocols. As soon as you have encryption, like tls, you have discrete messages anyway (records) and the stream is largely an illusion.
That said, I still like TCP as a compromise between simplicity and versatility. It’s been holding up incredibly well for a huge variety of use cases.
Routinely looking up things in hashes was not common back then. Hashing was computationally expensive when routers ran on little machines with well under 1 MIPS. A huge flat address space was not feasible with memory costs at the time. Nor was keeping huge routing tables in sync well understood. There had to be some cheap way to make routing decisions based on the IP address.
I dont really think this is a mistake. Its not like there are great options even now for end2end security on the transport layer (like we have vpns but they mostly require having preshared secrets). It would have been pretty impossible back in the day and added a lot of friction
[1]: I mean like really not. At all.
* TCP should have been a layer above UDP, not beside it. Or UDP and the urgent/out-of-band option in TCP should have been equivalent. This would prevent the blocking of UDP in corporate networks and countries trying to stop P2P networks.
* Both the network address and checksum should have been variable-length.
* NAT should not exist, because variable-length addresses would have negated most of its usefulness (other than for censorship).
* TCP should have been designed for networks with high latency (minutes/hours/days) to be ready for use in space. Optimistic delivery (is this the word?) should have been used instead of handshakes, but this might have required encryption from the start, to prevent sending sensitive information to the wrong recipient before it's verified.
* Address negotiation should have been built-in, so that peers IDs stay connected if the network changes, regardless of their IP addresses. TCP is a connected protocol (unlike UDP which is connectionless) so this was never really considered, but connected protocols simply don't work on the mobile web without yak shaving or embedding the stream in a tunnel that handles reconnection.
These issues are all severe enough that we probably shouldn't be using TCP directly. I know that they would haunt me had I designed it. It would be nice if the web provided a WebSocket that wasn't terrible, that handled everything mentioned above. Also I wonder if we scrapped all NAT workarounds, what it would take to provide something mathematically equivalent to direct connections, perhaps with homomorphic encryption, through open matchmaking servers kind of like Tor exit nodes.
Edit: I forget to add why these discussions are important. There's a tendency today to drink the Kool-Aid and assume that standards are perfect, when in reality they are often highly-opinionated, which creates a heavy burden on people who think differently. Flawed standards are a form of injustice.
NAT is a great idea in principle to connect networks in a scalable way. There are various UDP hole punching techniques that (depending on the devices) can be used repeatedly to get through multiple layers of NAT. Vs something like UPnP, which from what I understand, has poison pills which prevent it from communicating past 1 layer of NAT. So some of the features I complained about were probably engineered intentionally through a great deal of effort, and I might have even supported those efforts at the time. They just didn't know the negative effects those solutions would have on the open internet for stuff like networked games and P2P file sharing (which the status quo just loves).
For people today, it's hard to understand how expensive everything was back then. 1K of RAM in 1974 cost about $307 dollars. Yeah, you're not going to be putting that into your router. So those extra bits have a super-high hard dollar cost.
We're lucky he didn't go to 16 bits.
I don't recall what Vint Cerf's position was. But attributing the last few decades of IP dysfunction to 1970's choices seems... incomplete.
Yeah, I don't really think Cerf's decision really applies to the blunder that is IPv6. There were other options available to the IPng team that were foregone for SIPP knowing there was absolutely no transition/migration plan whereas the others had at least given this area some consideration.
I'm sort of expecting in year 40 of this attempt to transition to IPv6, we'll collectively realise it's a losing battle and look at future Internet architectures as salvation.
Most experiments, prototypes, and even full blown systems designed for the long term die well before 30-40 years and those extra addresses and features would ever be needed.
Much more human effort is wasted prematurely 'future proofing' systems and trying to predict the future. The brilliance of TCP/IP is its simplicity - if it had got more complex perhaps it would have been replaced in its entirety. We could be using Gopher over OpenDECNet.
Hell even AWS didn't start seriously supporting it until the mid 2010s... And heck I don't know if they support ipv6 across the board or not
In the late 70s / early 80s there was a ton that the public didn't yet know about cryptography. Few, if any private citizens could have gotten security remotely right at that point. Remember, Diffie-Hellman was invented in 1976, and RSA in 1977, and PKI came some years after that, and DES was also from 1976/1977. As well the public's understanding of the need to do authenticated encryption came later, and the industry didn't really start catching on until the 90s, and AEAD cipher modes didn't really become a thing till the 00s, and didn't start to get used widely for another decade, and so on and on.
The biggest benefit for most users is having a large enough IP space that devices can self-assign a unique address with the networks prefix with very low risk of collision. Many of the problems seen on wifi or home networks are due to various ARP issues and devices not properly releasing their own DHCP addresses. It's still very common. I had a work laptop get booted off my network after 10 minutes for a while until eventually figuring out another device was stealing its assigned IP.
IPng (IPv6) was designed when the original paradigm existed, not what we have now-a-days.
So, if we actually had been on track to do IPv6 in the decade it was defined, we could have gone a totally different way. Instead due to the rapid growth of the Internet in the mid 90's, IPv4 was morphed into something different and IPv6 had its own model until we started to need it again.
Ip routing is kind of like a tree (not a graph) in the ip regards, how would you get around it?
I admittedly don't know as much as I'd like about network, hence why I'm posing this question ;-)
If theres a way to efficiently model hierarchical domains for stuff like DNS/IP i'd love to know. Like, consistent hash rings maybe, but then we'd be reintroducing ring topologies...
Oh wait duh though. A cell tower is owned by a single company. Yeah that's prolly doable, the company can always buy beefier routers if they need the memory. And realistically speaking the virtual tree could be closely aligned to the physical one given how cell towers work.
However the correct ip way to do it would to have tcp(layer 4) decoupled from ip(layer 3) such that an ip address could change while maintaining the tcp stream. unfortunately this was not done, tcp has too many layer violations and is firmly interlocked with ip. I don't know much about the state of the art, does http over udp (quic) allow for ip address change?
Having letters in IPv6 feels wrong, at that point why not just have words, reinvent DNS at a lower level. You are already living on Cherry Picked Street, number 15, let your computer have the IP `cherry.picked.15.1`.
Nobody is going to start saying IPv4 addresses as hex, people aren't used to base 16. "Just set your gateway to co.a8.0.fe" said nobody ever.
A 128-bit dotted quad would be equally unwieldy. Again "The IPv6 local range is 252.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000/7" said nobody ever.
Honestly 128 bits was too much. They should have just done 64 and stuck to dotted quad and ARP. That would probably have got more acceptance than what we have now.
This feels familiar somehow...
[1] "What’s wrong with what3words?", Chapter 3, "offends.people.easily", covers it, https://www.youtube.com/watch?v=SqK0ciE0rto
"AAA.CA" becomes 4143.414141, routers will find the best route for 4143 and anycast ti 414141.<more specific address>, the apex AAA in this case would also be the ASN equivalent for bgp type routing and the TLD CA would also be the PKI certificate authority used to validate both routing updates and address ownership by the applications on either end. No port numbers either, the full address should describe the layer4+ addresses, so the full address user types instead of https://AAA.CA:8443, it becomes HTTPS.8443..AAA.CA (.. meaning anycast to the closest endpoint).
A good protocol is the easy part, the hard part is getting all the big networking companies and their engineers to agree on something.
Arbitrary length but fixed hierarchial address partitioning.
He made the right choice. the world will adapt, eventually.
I'd be willing to bet IPv6 wont be the last network upheaval we'll have as a civilization. (assuming we dont wipe ourselves out first)
What do you think will happen if we generate some really really big AI, and they realize that they'll be super reliable if we made every synapse have its own IP, then we network all the AI's together as each AI to behave as its own synapse node :-D
IPv256?
I mean, the estimate on the number of atoms in the observable universe is "only" 10^80, for comparison.
That wasn't an issue back when IPv4 was being defined though.
That said I agree that the world likely would have sucked it up and just gone with the 64 bit addresses, but there would have been a whole lot of grumbling for decades about the memory use. It's hard to imagine these days, but back then memory was outrageously expensive and enormous amounts of engineering went into minimizing memory use wherever possible. Having these addresses where we wouldn't even touch half of the bits for decades would have been unpopular with a lot of people, even if he would have been praised for being so forward thinking in the end.
On the other hand, the transition to IPv6 does give us a chance to fix some of the longstanding defects in IPv4 that would otherwise be baked into the protocol until the end of time, so it's not all bad.
Worse perhaps, people, algorithms, hardware, ... will all start to assume those bits are zero and start optimizing for that. We have seen this happen repeatedly throughout history (eg. Lisp pointers would stuff tags in the "spare" bits in addresses, making implementations non-portable).
These are all hard problems, but the lesson I take away is to
1. Always have a plan for evolving the design/protocol/API/...
2. Aim for a watertight design that supports 1. (Example: TLS 1.3 puts random stuff into reserved fields so nobody can assume they are zero).
3. Really stretch your imagination about future usages (assume 50+ years)
4. Have a ridiculously comprehensive test suite for both clients and servers
It seems clear that in terms of lifetimes, hardware gets used much longer than expected, software even more so (Y2K anyone), but protocols/formats/specs/... literally never dies
https://datatracker.ietf.org/doc/html/rfc791
there was no such thing as a router in the sense of dedicated hardware; a router ('gateway') was just a host with two or more interfaces
the ram used by ipv4 addresses was the same kind of ram used by the rest of the ip header and indeed packet
i mean originally you did have the imps but they were just looking at the arpanet node number; they weren't gateways
the switch from variable-length addresses to 32-bit addresses happened between ien 80 in february 01979 https://www.rfc-editor.org/ien/ien80.pdf and ien 111 in august 01979 https://www.rfc-editor.org/ien/ien111.txt
cisco was founded five years later, in 01984, at which point regular ram cost on the order of a dollar a kibibyte, so the content-addressable memory you need to accelerate a router was maybe ten or twenty dollars a kibibyte
would people complain about the ip headers having unnecessary junk in them? certainly, but there is plenty of that in the actually adopted protocol header design too, so clearly it wasn't a showstopper
Doubling from 32 to 64 would have had a significant impact on routing table size and it would have dropped the back pressure to keep it from growing too quickly for TCAMs.
It would certainly be nice today, but it’s not a sure thing it wouldn’t have been killed off as “bloated” when there weren’t even 2^12 hosts.
This is likely the #1 issue I hear from people in the field.
#2, for better or worse, NAT became peoples’ comfort blanket. It’s a boneheaded stateful firewall regardless of how long people can scream “NAT isn’t security”. By forcing people to take both changes at the same time, adoption was setup for failure. So many places in the US turned it off “because nothing I use is v6 only” and they weren’t quite sure if they were exposing their end hosts accidentally.
#3, link local addresses, guids, etc all assigned at the same time is a steep learning curve. Which will be used for which flows in the network? If my ISP changes my prefixes, do I know have to reconfigure all of my local network firewalls?
#4, prefix delegation, should each client behind my router get its own /64? My ISP only gives me 128 of them to hand out which is a problem for IOT. Guidance here is weak.
Burning all of the early adopters with the slaac vs dhcpv6 vs privacy extensions didn’t help either.
Nobody gives a shit about fragmentation. The upgrade UX was a disaster.
#2 is people being nervous, but seriously the "DENY incoming on $WANIF if not in state table" rule is all you need. This always struck me as "I refuse to learn anything new, no matter how little I need to learn".
#3 Again, people worried about the firewalls when they really don't need to be.
#4 A /64 is a subnet. Normally all of your devices would be on the same subnet, although with IPv6 you get a whole bunch which is nice. You can put the IoT devices on a different subnet that is partitioned from the rest of your network because they are IoT devices which means they are adorable little security vulnerabilities.
I think your last point is a good one. DHCP6 was poorly named. People thought it would work like DHCP did in IPv4 but it's really not intended for that. It's really only meant for routers, not for end hosts.
You missed a couple of points where I think IPv6 did have some flaws. SLAAC originally lacked a lot of the extensions that DHCP offered, like DNS server advertisement. In theory you could anycast your DNS queries, but DNS is a dodgy enough protocol already and this was really not a great idea. It's one of those things that works great in the lab, but is kind of a nightmare in the real world. It also offers no way to communicate back to a DNS server what IP address you have chosen, so if two hosts want to communicate with each other, especially if they are not in the same subnet, it becomes difficult to determine what their partner's address is. All in all the IPv6 committee considered DNS to be an application protocol and outside of their scope, but application developers consider it part of the network and so it got left out in the cold.
Using the MAC address to create the IPv6 address was also a bad idea from a big data standpoint. Luckily that was one of many options so it was easy to switch without breaking anything.
This didn’t exist. Many of the early implementations of “support” for ipv6 was a checkbox that said “enable”. These cheap routers (which is what the majority of people without deep pockets are on) rarely even gave you an explicit stateful firewall UI.
If you got a “firewall” option at all, it was to block stuff from getting out of your network, not in. Know why? Because “NAT already did that”. If you wanted inbound unsolicited traffic you used port forwards or the “DMZ”.
It’s not people being nervous, it’s the vast majority of network vendors not making the UX any good. I pushed people to try v6 hard, it was not the rosy transition it was supposed to be. v6 for a while ended up being a great accidental exfil path for malware for administrators that screwed this up. Don’t try to downplay it, it just makes you sound like an armchair quarterback, not smart.
> #4 A /64 is a subnet. Normally all of your devices would be on the same subnet, although with IPv6 you get a whole bunch which is nice.
A /60 is also a subnet, so is a /127. Subnetting is just breaking up larger IP spaces.
Anyway, you missed the point of the comment. What I was getting at is that doing prefix delegation for home users for anything more than a /64 is immensely wasteful. Yet a bunch of very large ISPs do just that.
Want to guess why? Once again, bone-headed implementations in off the shelf routers that do give a /64 per WiFi client. Untold waste in the v6 space because addressing guidance was (and still is) so poor.
This is where you see a lot of corporate pushback on IPv6. The Deep Packet Inspection vendors have been very slow to adopt so corporate policy is often just to block all IPv6 period.
I won't argue against a lot of home routers having egregiously bad firewall configuration UIs though.
In IPv6 a /64 is special. It's the smallest network you are supposed to allocate. Some people think it is the equivalent of an IPv4 single address, but this isn't quite right. It is still a full subnet. It is better thought of as the IPv4 /24 behind a single NAT address. Home administrators are expected to put all of their hosts on it. Assigning a /64 to each client is not supposed to be a typical use case, and would be mostly to avoid having the clients inter-communicate. The problem however isn't wasted space, it's just that you'll exhaust your typical home /56 too quickly.
I can see you're concerned about running out of IPv6 addresses due to excessive waste, but we've not even scratched the surface on them. The address space is mindbogglingly huge. A lot of the optimizations we have to do to save space with IPv4 are simply not relevant in IPv6.
If you can't implement it easy enough for it to be optimal once in place, it's not optimal. Whatever the advantages of ipv6 are, it wasn't enough to suck everyone in. You have to consider your environment. And re-training sysadmins is part of it and just telling them "lol it's easy" isn't going to do much to get them there.
By the time we get fully switched over, I wonder if we won't be pining for whatever comes next.
IPv6 could have been a lot more pragmatically backwards compatible, absolutely. It would have been much more doomed as a temporary solution in that case. The IPv6 we got was designed to be a more permanent solution, which makes it feel much less pragmatic. That's somewhat how trade-offs work.
Right - I’m not a network admin, but back a few decades when I was hanging around the CS lab and setting up home networks, I sure had to type in IP addresses by hand a lot. If they were more than four bytes long that would have been painful!
This is thinking which appeals to people who don't understand the problem because it feels like a compromise, but you can't negotiate with silicon. If it expects a 4 byte address payload, then that's all it will ever handle.
Adding more bytes is exactly the same amount of redesign and replacement work as adding more. If the problem was solely software related, it would've been done by now.
Add a one-byte option into the header, give it an unassigned value (e.g. 6) and assume that any box that does not understand it will pass it through unchanged. Use it as the most significant byte of the address, and obsolete all routing equipment that doesn't understand it from the backbone so that they can be used to handle routing in one of the new 32-bit super-A class networks.
Sure it's ugly as hell, but it is just backward compatible enough to work.
As many have found out, unfortunately "any box that does not understand it will pass it through unchanged" is frequently false. What often happens is that a box which does not understand it silently drops the whole packet, under the mistaken impression that it's an attempt to invade the network.
> Use it as the most significant byte of the address, and obsolete all routing equipment that doesn't understand it from the backbone
You'd have to not only obsolete all routing equipment, but also all end hosts. Consider what happens when host A, which does not understand the option, receives a packet from host B, which has an address which requires the option; the reply packets from host A will never reach host B, since host A doesn't know it has to add the option.
These "just add another byte" solutions are often proposed in these discussions, and look deceptively simple at a first glance, but fall apart once you start looking at it in detail.
Routers don't run packet addresses through software, they run then through silicon ASICs which know the address is 32 bits, that's it.
Your solution is literally how Cisco implemented IPv6 on a bunch of it's switches: the IPv6 routing path goes through the firmware and is slow as hell compared to the pure ASIC IPv4 handling. It didn't catch on because it was terrible at any scale (we were building Hadoop clusters, 100+ big data nodes with constrained throughout is money down the drain).
Your claim was that any change to the ASIC was the same design cost. But really? A 40-bit routing table design, where there is a 32-bit fall-back to preserve the top 8-bits vs a 128-bit routing table design. Why would these be the same cost?
When would you have done it? When the hardware was 5 years old? 10 years old? I've been on change operations where the hardware was over a decade old and our biggest concern was making sure we had replacement PSUs on hand since the thermal shock of a reboot might take them out.
This is the problem, you have absolutely no idea what anything costs or how it works - you're simply thinking "lifting those bits sounds heavy, what if we made them lighter?" as though a committee of engineers, designers and industry didn't spend a lot of time considering exactly that problem.
If you could add 1 byte in software and be done with it, it would have already happened.
You are arguing that all change has the same cost. I don't see your reasoning for this. A smaller change to a system has a lower cost in R&D, and a lower cost in manufacture.
Although it is probably true that the cost of updating all equipment dominates the cost of redesign, this is only an issue for a big bang update rather than a rolling upgrade.
If you can articulate how this magic wand would work in practice we're all ears since you solved a worldwide problem, but I'm betting my salary that you can't.
0. Gating mechanisms should live at lower and higher levels in the stack. We have 801.1x for wifi/port security and TLS between client and server.
1. DHCPv4 (and conflation with BOOTP) is messy and doesn't have a trust model. There ought to be a mechanism to validate that the DHCP server authentic, its data is valid, and it's supposed to be there. (Rogue DHCP servers can be thwarted with participation by the network gear, but requires more engineering than client simply being able to ignore bogus DHCP servers. Networks still must police and shutdown rogue endpoints nonetheless.)
2. DHCPv6 (stateful -SLAAC, stateless +SLAAC, DHCPv6-PD), SLAAC, and M and O flags make it more painful to deploy than it ought to be.
3. DNS DNSSEC/DANE hasn't really caught on (yet?). DNS needs a top-down trust model or at least make it user-friendly enough (tools) to deploy.
4. EBGP security. (Let's not talk about that.)
In general:
i. Optional features don't get used.
ii. Complexity is a barrier to adoption and a source of security issues.
iii. Over-simplicity is another source of security issues and a source of lacking functionality.
Naive early implementations could have been nearly the same, just using the top 32-bits to route to a host, and let the host handle all lower 32-bits (effectively a big port number). But later revisions could have split it into e.g. /48 for host and 16-bits for service selection, or take to an extreme maybe even /56 for host and 8-bits for service.
The stupid thing is that the vast majority of IPv4 addresses are only used for a handful of public services, almost always HTTP and HTTPS. So even though we have extreme pressure on the 32-bits of IP address, the 16-bits of port number is mostly wasted and underused.
At the time, before IPv4 was introduced, IP addressing was 8-bit, and this was mostly a single machine per site. Moving to a 64-bit service address, early implementations could have kept their 8-bit address and had 56-bits for service ID, most of which could just be 0 and unused.
Some sites might then have gradually move to /8 for site and /16 for site+machine and 48-bits for service ID, most of which would still be unused.
Later on, we might see bigger networks taking a /8, /16 for network+site, and /24 for network+site+machine, etc...
Up to the present-day, we could easily see how /48 for network+customer, /56 for network+customer+machine and 8-bits for service would still satisfy almost every need. Given that IPv6 /48 are being given out like candy and a /40 is trivial to get for a larger organisation, there's not really a massive difference.
So, that largely covers inbound services, but there's also outgoing connections to consider. Given that IPv4/IPv6 already has to consider address+port on both sides to identify a connection, we could still have something like a /56 for network+customer+machine leaving 8-bits for discriminating between connections to a /64 remote service.
It's possibly interesting to consider that this is actually kind of how the original IP stack works - a connection is just the pair of 32-bit addresses plus the protocol and then UDP and TCP bodge in some extra port numbers as extra data within their protocol-specific data. If we'd instead had an IPv5 that extended the addresses to a pair of 64-bit addresses plus protocol, that TCPv5 and UDPv5 might not have even needed port numbers. This would then have shifted the burden of supporting TCP and UDP from the kernel (because it needs to be there for IPv4 to check access control to different ports) to potentially allowing user-space protocols to be implemented because the OS can assign the socket its 64-bit address and wouldn't really need to care what happens to it after that.
The MTU should have been reified to the IP layer. And it would have been so easy!
When a system sends a packet, it sets the "forward MTU" field in the IP packet to be equal to the MTU of the outgoing link. Each router along the way inspects the packet and if its outgoing link has a lower MTU, it replaces the "forward MTU" with its own.
Once the packet reaches the destination, the target host copies the "forward MTU" field (which now has the lowest MTU seen along the path) to the "reflected MTU" and sets it on the next reply packet.
So the originating system can discover the limiting MTU by the time it gets the ACK packet.
Sigh. Instead, we have a Ruby Goldberg machine with fragmentation, ICMP and all the related unreliable crap that has forever cemented max MTU as 1500.
The destination will detect that the packet has been chopped, send the updated reflected MTU and everything would be fine. Of course, the minimal allowed MTU (say, 512 bytes) should provide enough information for the target to find the correct TCP flow to which the packet belongs.
No need for out-of-band ICMP signalling or fragmentation.
And in practice, topology changes happen rarely, so this should be very infrequent.
Retrofitting features like this into the middleboxen takes decades, and never finishes, so you either get them in on day zero or you never get them.
UDP-based applications would have to do MTU discovery, but that's much easier when you're receiving truncated packets.
I work with magnetic tape and have seen tape block sizes grow from ~512 bytes in the early 1990s, to 10s MB today, resulting in huge performance improvements. Network packet sizes haven't changed by anything near that amount in 30 years.
That's not a mistake in TCP/IP.
The earlier attempts to replace it like SCTP didn't work out because too much middleware blocks it.
Crypto is too hard, it should be offloaded to AI.