Fun with IP address parsing
blog.dave.tf
blog.dave.tf
This was the source of one of my favorite “bugs” ever. I was working on multiple mobile apps for a company, and they had a deep link setup that was incredibly basic: <scheme>://<integer>, which would take you to an article with a simple incrementing ID. This deep link system “just worked” on iOS and Android; take the URL, grab the host, parse it as an int, grab that story ID. Windows Phone, however… the integers we were parsing out were totally wrong, returning incredibly old stories!
Turned out that the host we were given by the frameworks from the URL was auto-converted to an IP in dotted-quad format, and then the int parser was just grabbing the last segment… which meant that we were always getting stories <256, instead of the ~40000 range we were expecting.
I bet it was not “I love it” sentiment when you had to debug this kind of issue though, haha :)
Thankfully, I caught this one while building the feature in the first place; I don’t imagine I’d have such fond memories of it if I’d had to recreate it from user reports!
According to RFC 1123, hostname could legally be entirely numeric, and web browser shouldn't attempt to "correct" it (as it is a valid URI) for schemas it doesn't know anything about - as it doesn't know the rules for hostname for a given protocol. This is also not a valid IP address according to RFC 3986 (which specifies URI syntax), as this specification requires #.#.#.# format with three dots.
That said, using authority for something that isn't technically a hostname is misusing the field. I think using `<scheme>:<integer>` would have been a better idea.
For anyone using Ruby, I'm the author of a gem [1] that comprehensively protects against SSRF bugs. For anyone using Golang I recommend this [2] blog post.
[1]: https://github.com/arkadiyt/ssrf_filter
[2]: https://www.agwa.name/blog/post/preventing_server_side_reque...
(My solution at home is to blanket block IPv6 entirely)
IPv6, with end-to-end connectivity, is how the Internet is supposed to work. It's how it did work in the early 90's, even with IPv4.
If you want to secure your servers, use a firewall. Maybe it's a host-based firewall.
(I know the weaknesses of NAT but the cat’s out of the bag at this point...the question isn’t really “why should we use NAT”, it’s “why should we go through the pain of breaking it”.)
No, the way the Internet is supposed to work is that you have one routable address space. If you need to expand it, the previous address space is imported as a subset of the new one.
https://cr.yp.to/djbdns/ipv6mess.html
I will never forgive the IPv6 for not making the 32-bit IPv4 space a subrange of the 128-bit IPv6 space. Years after winning the IPng wars they admitted their mistake and standardized NAT64, but it was too late. NAT64 should have been part of IPv6 from day one, and every IPv6 router acting as a default route gateway should have been mandatorily-required to offer NAT64.
If I'm on an ipv6-only host and blast UDP at ::ffff:1.2.3.4, they should get delivered to 1.2.3.4, no?
The actual, real-world problem with 4 being a subrange of 6 is that 4-only hosts are blissfully unaware that the super-range exists, so have no mechanism to send packets there. This is of course where you're right about NAT64 and the state requirements.
No, it won't necessarily! That's precisely the problem. Until NAT64 was introduced it was in fact impossible for an IPv6 router to deliver your packet to the IPv4 host 1.2.3.4. NAT64 still isn't mandatory (and likely never will be), so if you're writing software you can't assume those packets will get through even if you have an IPv6 network connection with a default route.
NAT64 didn't come about until long after IPv6 was finalized, and NAT64 support from default-route IPv6 routers is still is not mandatory. That's why we have this mess with dual-stack hosts: you cannot safely assume that your IPv6 router is willing to deal with the IPv4 world on your behalf.
The ::ffff:1.2.3.4 address space does, in fact, date back to the early days of IPv6 (it came from RFC 2765, about one year after IPv6 was finalized), but it was not meant for letting IPv6 clients share a single IPv4 address -- it was only for servers which for some reason had their own IPv4 address but couldn't speak IPv4. Yeah, back in the early 2000s people thought this problem might happen.
The IPv6 committee was viciously hostile to NATs. The way they saw it, NATs were the problem that made IPv6 necessary, so no way were they going to allow any NATs to pollute their precious IPv6. If that meant that the whole world had to run two separate internets (IPv4 and IPv6) for the rest of eternity just to keep the IPv6 network puritanically NAT-free, then so be it!
It took them more than a decade to realize how stupid this mindset was.
I possibly have some sympathy with the anti-NAT view taken at the time, even if it ended up being the wrong thing to do it hindsight. Adding more mandatory complexity to implementors would have harmed adoption rates, and I've seen some weird edge cases with NAT64 - it's not necessarily a trivial thing to implement correctly.
I fault the IPv6 proponents for not forseeing our current situation. DJB saw it with crystal clarity in 2001. Lots of people warned them that this would happen.
Since packets aren't APIs, you should never see ::ffff:0:0/96 in packets on the wire. A v6-only host can't use this prefix to send v4 packets to v4 hosts.
(What would the source address of those packets even be?)
Not requiring backwards compatibility in IPv6 guaranteed that IPv4 would be around forever. IPv4 is never, ever going away because of this.
The only people who couldn't see this coming were from Bell System backgrounds where you could use centralized schemes like "Ma Bell says tomorrow is the Flag Day, flip the switch". In a decentralized system people don't stop using the old system until you give them a new system that is backwards compatible. Then you drop the backwards compatibility in a second, separate upgrade much later, on a timetable dictated by adoption, not flag days.
Your proposal would force maintaining IPv4 for longer and in more networks: every IPv6 router would have to have IPv4 connectivity and probably a routeable IPv4 address, so it wouldn't even solve the address exhaustion problem for long (perhaps not even at all).
> Not requiring backwards compatibility in IPv6 guaranteed that IPv4 would be around forever. IPv4 is never, ever going away because of this.
IPv4 has already been eliminated from newer edge networks, and for those networks the vast majority of upstream traffic is IPv6. No doubt those networks will have to maintain 464XLAT for a long time as the long tail of upstream sites that are only v4-accessible, but they'll be able to have a smaller and smaller pool of 464XLAT servers and outsource the v4 connectivity support further and further upstream (just as with Usenet), until eventually v4 connectivity becomes a paid add-on and then goes away entirely. Home routers for use with PCs will probably have to offer 4over6 for a long time, because it's hard for an ISP to be confident all their users are up to date, but that doesn't actually reduce the benefits that much (all your internal network management can still be v6, only the little home user LANs are v4), and organisations that manage all their endpoint devices don't even need that much.
> If you want to secure your servers, use a firewall. Maybe it's a host-based firewall.
Firewalls do not solve this problem because a you do want service to service communication. What you do not want is code that crawls to user supplied URLs to access your internal services. Do you need application level protections. With IPv6 you're basically forced to declare your CIDR explicitly whereas with IPv4 you could easily achieve a secure by default system.
Just to note, this should be ::ffff:169.254.169.254
https://github.com/skx/remotehttp
I've found, and reported, a whole bunch of services which take user-supplied URLs and don't filter out access to localhost:8080/server-status, and similar local resources.
A common route to attacking these is to access the AWS metadata URL endpoint. Something at least the Google cloud prevents, by forcing the use of the `Metadata-Flavor: Google` header.
I think I will quote this article any time I see someone using regex to validate or parse IPs.
1.1 which to me is the shortest useful IP address I am aware of.
1: https://unix.stackexchange.com/questions/99336/how-does-ping...
But do a "strace -econnect nc 0 22" and you'll see that yes, actually, a connect() syscall to "0.0.0.0" does connect to localhost.
if (!fl4->daddr) {
fl4->daddr = fl4->saddr;
if (!fl4->daddr)
fl4->daddr = fl4->saddr = htonl(INADDR_LOOPBACK);
...I'm never writing anything that positively accepts 127.1, or 0127.000.000.0001 as a valid address no matter what garbage implementations do.
The issue we have with this are situations when we have to accept only inputs that are domain names which are sure not to be treated as an IP address by some software downstream of us.
But we'll see how well that works... I just fed the first 4 google results for "ip address converter" with 10.1: Three converters gave an error message and one came up with 0.0.0.10.
people doing lan-partys around 1995 called ...
just kidding, great you discovered it
I have to disagree with this conclusion. I see it very frequently on Linux. It turns out that programs can bind their listen address to just ::, and the kernel will still allow connections from IPv4, with the address mapped to ::ffff:0.0.0.0/32 -- outbound connections use the same notation.
This is only true if the sysctl bindv6only or socket option IPV6_V6ONLY is 0, and is defined by RFC3493.
No, IPv6 explicitly rejected that idea at first. Most of the other IPng proposals did have a backwards compatibility mechanism like that. I'm still sore that the least backwards-compatible proposal was the one that won.
Later the IPv6 cabal admitted their mistake and published NAT64, but at that point it was too late to make it a mandatory required service offered by any default-route router. So now we have all of this crap about dual-stack hosts instead of simply being able to upgrade to IPv6 and trust that you will not lose any connectivity.
This is basically why, twenty years after it was standardized, IPv6 is still merely the "internet of cellphones" and no closer to replacing IPv4.
As usual, DJB saw all of this decades ahead of time:
Wait, what? 77.77.88.88 is in dotted decimal. It doesn't correspond to 7777:8888 in hex.
edit: Somebody else already noticed on Twitter:
> And as @alanjmcf noticed, I messed up one of the representations above.
> 1:2:3:4:5:6:77.77.88.88 means 1:2:3:4:5:6:4d4d:5858, not 1:2:3:4:5:6:7777:8888. I missed out a decimal-to-hex conversion in there.
I got to find that notation useful once, to make a shorter one-liner... without even knowing that there were different classes of IPv4 address, and that I was looking at one of them.
It's a tiny function that gives me the IP address of my machine in the LAN, for either Linux and Mac:
# Get main local IP address from the default external route (Internet gateway)
iplan() {
# Note: "1" is shorthand for "1.0.0.0"
case "$OSTYPE" in
linux*) ip -4 -oneline route get 1 | grep -Po 'src \K([\d.]+)' ;;
darwin*) ipconfig getifaddr "$(route -n get 1 | sed -n 's/.*interface: //p')" ;;
esac
}
Full disclosure, I got the "1 is shorthand for 1.0.0.0" from here (which didn't get into explaining why it is a shorthand): https://stackoverflow.com/a/25851186
What I wanted to express here (and did badly) is that crossing paths with this arcane Class-A style IP address is something so strange nowadays... in my case in more than 10 years professionally working as a developer, I had seen it exactly once and even then, didn't recognize it for what it was.
The code snippet was just an extra curiosity in case anyone found it useful.
Ugh.
% ping 1 PING 1 (0.0.0.1): 56 data bytes
% ping 1.0 PING 1.0 (1.0.0.0): 56 data bytes
unless an exception to the rule exists somewhere, and 'ip' is actually doing it right!
/* This uses a non-standard parsing (ie not inet_aton, or inet_pton)
* because of legacy choice to parse 10.8 as 10.8.0.0 not 10.0.0.8
*/
(the entry point to start tracing down to the above inner function is right around here: https://git.kernel.org/pub/scm/network/iproute2/iproute2.git... )
By the way 4.2BSD was being compatible with older or contemporary implementations, like ITS which was running TCP before any Unix was.
For example plenty of machines back then used octal as a preferred human representation. In fact that’s why octal is the default format of numeric constants in C: C, like Unix, was initially developed for an 18-bit (six octal digits) PDP-7. The smaller 16-bit PDP-11 version came later.
It was a surprising amount of work to figure out all the different formats an IP address can be shown in and convert a given IP into all those formats.
$ ping stairway scavenger tracker upcoming
PING 209.216.230.240 (209.216.230.240) 56(84) bytes of data.
64 bytes from 209.216.230.240: icmp_seq=1 ttl=50 time=68.2 ms
64 bytes from 209.216.230.240: icmp_seq=2 ttl=50 time=69.5 ms
64 bytes from 209.216.230.240: icmp_seq=3 ttl=50 time=67.2 msAnd in this case "209.216.230.240" translates to "stairway stupendous tracker upcoming".
Thanks for sharing.
Nitpick: missed a single zero in the middle there.
No, I did not add dotted quad notation to the parser. No, you can not have more than four hex digits in a single quad; 00000001:2::3 is a syntax error. It supports “normal” stuff like ::, ::1, 2001:db8::1, and even non-normal stuff like “2001-0db8-1234-5678 0000-0000-0000-0005” (to be compatible with the really basic IPv6 parser I put in MaraDNS’s recursive resolver nearly two years ago), but does not support any of the IPv6 corner cases in the linked article.
The IPv6 test cases in the automated test for the parser are at: https://github.com/samboy/MaraDNS/blob/master/deadwood-githu... (The final three lines are supposed to return errors)
http://openbsd-archive.7691.n7.nabble.com/inet-net-pton-seem...
Love it! No conversation about SUS is complete without Theo bashing up the absurdity of some historic bugs being documented as features. :-)
---
I do like the hex specification, though. Especially in the age of /29 and such, it's way easier to deal with space using such notation than the decimal numbers, which make little sense for network boundaries in such case. It looks like ping supports most of these (try `ping 0x08080808`, or `ping 0x08.0x080808`, but note that 0x0808.0x0808 is not valid, only 0x08.0x08.0x0808 would be), but `dig @` doesn't.
BTW, I guess this finally explains why the netmask is often shown as `inet 127.0.0.1 netmask 0xff000000` on the BSDs, which is actually a valid IP address notation, as it turns out!
That said, many of those representations no longer make sense in the modern world, and I'm actively choosing to not support them. That doesn't mean I don't understand why they came about in the first place, au contraire! I'm explicitly deciding that their historical reason for existing no longer applies.
> The familiar 192.168.140.255 notation is technically the “Class C” notation. You can also write that address in “class B” notation as 192.168.36095, or in “Class A” notation as 192.11046143. What we’re doing is coalescing the final bytes of the address into either a 16-bit or a 24-bit integer field.
According to this:
https://www.digitalocean.com/community/tutorials/understandi...
Which details my understanding, classes refer to the ranges, not so much grouping the latter part
Happy to be corrected!
> Traditionally, each of the regular classes (A-C) divided the networking and host portions of the address differently to accommodate different sized networks. Class A addresses used the remainder of the first octet to represent the network and the rest of the address to define hosts. This was good for defining a few networks with a lot of hosts each.
in the ntp config file, you will have stuff like this:
server 127.127.1.0 # local clock
server 127.127.20.0 minpoll 4 iburst prefer # gps clock
here's a page explaining the clock types:
https://www.eecis.udel.edu/~mills/ntp/html/refclock.html
but basically it's a weird anachronism. I'm not sure if NTP will actually bind to those addresses using the tcp/ip stack, or if it someone just got lazy and coopted the ip address parser for off-label use.
I’ve never had someone tell me, “see if you can ping 143267841”. I’ve worked in networking for coming up on 30 years now and just haven’t found the use.
Or worse, in one case I've had to deal with both (plus another surprise twist):
public class IP {
byte[] value
// if true value is a variable-length ASCII dotted octet,
// if false it is length 4 - with LSB in value[0].
bool isString
}Internally, I would imagine that almost every IPv4 stack uses 32bit ints to represent an address. Its not that crazy to think this would leak out somewhere.
I've written (un)parsers where we would just treat IPv4 addresses as integers because A) that is how they were treated in the binary data and B) given what we were doing with the data, we didn't actually care about the IPv4 field.
So as I see it - 143267841 (or 0x88A1801) is the address, and quad-dotted decimal is a (slightly more) human-readable representation of it.
Not a boomer myself (I'm just a poor millennial) but I was lucky enough to enjoy the early days of the internet.
May he rest in peace.
If you count mixed representations, there would be 120 possibilities, but the tool doesn't generate those.
One of my motivations for writing the library was being able to grep for IPv6 addresses in text files; it's surprisingly difficult to match all valid representations of a simple IPv6 address as seen in the example here:
https://twitter.com/beaugunderson/status/527393872909828096
I also maintain a site for examining IPv6 addresses that may be useful to people working with IPv6:
Guilty party was Poco::Net library if I recall correctly. I can maybe see this making sense if you provide four octal digits (0377), but not three, and I have a hard time believing anybody has ever used this on purpose.
This is how embedded stacks (LWiP) store IPv4. Didnt' know browsers could respond to it thought.
Mixing IPv4 and IPv6 is just evil.
There clearly should be a common library to take care of these things, which are way too complex for most of developers.
https://www.rapidtables.com/convert/number/ascii-hex-bin-dec...
Oh, yeah, and 1.1 is the only DNS server address I memorized.
I’m sad that there probably won’t be any memorizable addresses in ipv6.
I remember it was a few days after they came out with 1.1.1.1 and 1.0.0.1 that it dawned on my that I could drop the zeroes. I’d been wondering why they hadn’t chosen 1.2.3.4, but once I realised 1.0.0.1 was just 1.1, it became fairly obvious why they had chosen it.
(P.S. mtr’s stripchart with latency information is super great for this sort of thing; I have MTR_OPTIONS=--displaymode=2 set in my environment.)
Unless there is a big reason, never drop backwards compatibility. In this case, supporting all those forms would be very do-able. The best way to support them would be to find some old BSD parsing code and port it, then you can be sure every corner case is handled the exact same way. Handling corner cases differently is a great way to introduce security vulnerabilities and crash/DoS bugs that every user of your library will have to be aware of.
Maintaining such code isn't really a good excuse here either - the code is only going to be a few thousand lines, is self contained with no dependencies, is easy to test, not going to change much with time, etc.
Basically, there is no benefit to removing this feature, so don't break what isn't broken.
In cases where backwards compatibility is needed, just use inet_pton() and let the libc maintainers deal with the bug reports (I believe inet_pton() dropped octal and hex support for ipv4 addresses)
Correct.
It also doesn't support truncation unlike inet_aton. e.g. inet_aton considers "1.2.3" and "1.2.0.3" to be the same address.