netaddr.IP: a new IP address type for Go
tailscale.com
tailscale.com
I think your statement reflects more on your own biases than reality.
What I'm saying is that the design flaws of net.IP, the absence of which would have benefited the whole ecosystem, fall into a familiar pattern that can be learned from. It is good to see that addressed, even if by third parties.
It wouldn't surprise me one bit if this ended up in the stdlib for Go 2.0.
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.
I had to work around one of the issues here in my first use of the built in ip type - I needed to parse ip addresses and pass them to a command that needed a tcp4 or tcp6 flag depending on the address type.
The "net" standard library is still very nice and works extremely well.
The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
Kubernetes and Docker are not networking. They orchestrate some networking components that do the actual network processing in the kernel (iptables, ebpf) or other high performance user space c/c++ (dpdk, etc).
And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them.
As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
type IP struct {
hi, lo uint64
z string
}
But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are.
And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that.
But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
In either case, having the interning wrapped up in a reusable library is great.
Who uses IPv6+zones and why?
I'm confused what zones are about. They don't seem to be part of IPv6 packets. Where do they come from? Why does IP type have to deal with them?
It’s relatively common to encounter them in some network configurations where the routers use link-local addresses rather than global addresses in their RA packets. They are also used in SLAAC, because a device can configure a link-local address without knowing anything about its environment.
[ I think in the past there might have been an idea to use them to disambiguate site-local IPv6 addresses, but network interfaces are not sufficient in that case, and site-local addresses were problematic in lots of other ways, so they have been dropped from the specifications. ]
https://en.wikipedia.org/wiki/IPv6_address#Scoped_literal_IP...
Apparently it's possible to embed interface name into an address:
fe80::1ff:fe23:4567:890a%eth2
The article also points to an alternative solution - embed interface index into an address:
fe80:3::1ff:fe23:4567:890a
Still unclear why and where do zones come from.
It's in a PR here: https://github.com/WireGuard/wireguard-go/pull/11/files
Which mentions:
> Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake.
It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it.
[0] https://github.com/OJFord/terraform-provider-wireguard/blob/...
(There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
The programmer's mantra.
I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest.
Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
Update: never mind, I re-read the docs of SetFinalizer to get the explanation:
SetFinalizer sets the finalizer associated with obj to the provided
finalizer function. When the garbage collector finds an unreachable block
with an associated finalizer, it clears the association and runs
finalizer(obj) in a separate goroutine. This makes obj reachable again,
but now without an associated finalizer. Assuming that SetFinalizer
is not called again, the next time the garbage collector sees
that obj is unreachable, it will free obj.Or how F-Secure is shipping secure USB keys with bare metal Go firmware.
Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
I could clarify in the post.
> use a zone mapping table, mapping between zone index integers and zone name strings. That’s what the Go standard library does internally. But then we’re left susceptible to an attack where an adversary forces us to parse a bunch of IP addresses with scopes and we forever a bloat a mapping table that we don’t have a good opportunity to ever shrink. The Go standard library doesn’t need to deal with this, as it only ever maps interfaces that exist on the machine and doesn’t expose the integers to users in representations; its net.IPAddr.Zone field is a string.
Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
Why would it be better to use tailscale.com?
It has nothing to do with Tailscale. It's just an IP address.
A neutral package name seemed best.
The former looks like the name of a Go type (which is what it is) and the latter looks like a domain name or something.
I'll even give you guys leading lowercase, which breaks the usual HN title conventions. Don't let it go to your head.
that's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?
... Tailscale...
You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off.
That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator.
UTF-16, perhaps? LOL
It's covered in the blog post.
> The underlying type of a net.IP is just a []byte
> ...
> It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.