rklaehn

They built something amazing and now have a nice exit. The code remains open under a non profit foundation. I don't think it is that bad. If working conditions are too bad people will just quit and work for the foundation.

If you want to compare iroh with something out of the IPFS world, iroh is basically "what if you build libp2p on multipath QUIC".

If you want full ipfs functionality, we have a number of protocols you can combine: blobs https://github.com/n0-computer/iroh-blobs and gossip https://github.com/n0-computer/iroh-gossip .

There is one thing we do not have yet - global content discovery. The reason for this is that so far we have not found a way to do it in a way that just works.

It is a very hard problem, but I hope we can come up with a solution so we can cover the use cases that initially got me excited about IPFS many years ago.

Main author of iroh-blobs here.

We realised that there are many more use cases for p2p streams than for blobs, so in the last year we have focused on getting iroh to 1.0, which involved implementing our own multipath QUIC implementation based on quinn. This was a lot of work and took the full focus of the team.

That being said, we plan to continue to work on blobs and get it to 1.0, which will involve some API changes and some internal changes, primarily to the blob store.

The blobs network protocol itself is just BLAKE3 verified streaming with a few tiny modifications (chunk groups) for efficiency, and hasn't changed since 2 years. Not because we don't have time for it but because it is done.

My personal goal is to have blobs working so well that you can forget about it, and also provide a solution for global content discovery.

There is this https://github.com/bittorrent/bittorrent.org/pull/174

We would love this to gain more traction so we can use mainline also for content discovery. But we have to be realistic: mainline moves slowly, and for good reasons.

There are some technical differences since we build on QUIC, not on wireguard. We think that QUIC offers some advantages for demanding use cases.

But everything we do is open source as well. Everything in the core is MIT and Apache2 licensed, including the relay binary/library.

We have not implemented this yet, and there are some things to consider. But there is an open standard for load balancing QUIC connections, and we have hooks in our QUIC implementation noq to generate connection ids that should allow us to work with this open standard.

https://datatracker.ietf.org/doc/draft-ietf-quic-load-balanc...

Get in touch if you have a demanding use case and want us to help.

You don't have to use it. But here are some answers:

Re-issuing keys is as simple as generating a new Ed25519 keypair.

let secret_key = SecretKey::generate(); // takes less than a millisecond

Iroh as of now has no fleet management. So the concept of revoking a key is something you would have to add yourself.

We have extensive logging for iroh. You can enable trace logging and even enable qlog for detailed connection logs. You can view the logs in any qlog viewer. We have written one, but there are others. It is an open standard for QUIC logs.

https://datatracker.ietf.org/doc/draft-ietf-quic-qlog-main-s... https://github.com/n0-computer/qlog-viewer

The iroh relay library and binary are open source just like everything else in the core. You can of course run a relay in your intranet, but the exact details depend on the use case. Get in touch if you have a demanding use case and want us to help.

This might not be a giant difference in practice.

But headscale is a community project. The iroh relay code is by number0 just like iroh itself and lives in the same MIT and Apache2 licensed repository. You can even embed it into your webserver if you have a special use case - it is very modular.

We do have browser support, but as of now it is relay only.

We have made some progress reducing the set of patches that is needed to get iroh to run on an esp32 with SPIRAM. We just need a little dependency reduction for it to fit, but other than that iroh 1.0 works out of the box now.

You only need to know the public key of the target endpoint.

It will work even on very restrictive firewalls. Even if they outright ban UDP packets, we will fall back to the relay connection which is https/websocket.

Note that here is not a single keypair per machine, but per endpoint. You can have multiple endpoints on one machine.

That’s the beauty of it. The app end user doesn’t have to do anything.

The iroh endpoint will determine its location in the world by probing the configured relays using QAD (similar to STUN). It will then choose a home relay and establish a https connection to it.

When another iroh endpoint wants to talk they first briefly talk via that relay, then hole punch and establish a direct connection.

All of this happens in the background without the user even noticing. It’s also very lightweight - runs on an embedded computer with 2 megabytes of RAM.

> I might have assumed io_uring would be the high throughput kernel interface for Linux.

We might do an io_uring based linux only implementation at some point. For now we do care about performance very much, but also want to have a single code base for all supported architectures and platforms.

We do support a lot out of the box, which is hard enough as is with a small team. And while io_uring is a bit better than the current sendmsg with GSO / recvmmsg with GRO setup, it isn't orders of magnitude.

> Can iroh run on a proxy server which forwards requests to backends that don't integrate with iroh directly?

We have a tool called dumbpipe that has options to forward local tcp services over an iroh pipe. Something like global netcat.

And there are plenty of tools that do something similar for specific services, e.g. there is iroh-ssh.

> What is the CPU overhead at link saturating speeds?

We don't have exact measurements, but CPU is not the bottleneck usually.

Yeah, that’s from a friend of mine. If something like this would gain traction that would be cool.

It might be an option to provide a good migration path for projects that build on libp2p.

But in the long term you probably don't want two fully featured p2p networking stacks in your dependencies.

Yes, exactly. Basically we chose the bottom edge of zooko's triangle.

https://en.wikipedia.org/wiki/Zooko%27s_triangle

A mapping from a scarce but human readable name to a non-scarce but not human readable name would't be that hard, but we haven't done this yet.

If we do it it will probably be an additional crate reusing some of our infrastructure, not built in to iroh itself. There are a lot of use cases where the non human readable names work perfectly fine.

We would implement two versions, one using DNS and one using an appropriate decentralized system like ENS.

I think we do very well with devices devices with limited bandwidth and changing connections. We are able to saturate a 1 GiB link from a normal desktop PC or good phone, but have some work to do to saturate a 10 GiB link with a single process.

We don't have a comparison benchmark, but we are fast enough that we are not the limiting factor for many use cases.

In many cases the performance bottleneck is the interface to the kernel to send and receive UDP packets. Our QUIC implementation is using all available tricks to make this as fast as possible. For example on OSX we use the sendmsg_x syscall to send multiple UDP packets in one syscall. On Linux we use GRO/GSO and recvmmsg to send/receive as many packets as possible.

But to be completely honest, in some cases TCP is still faster for raw throughput on server class hardware. Decades of optimisation have gone into TCP. But QUIC/UDP is quickly catching up. All the major cloud vendors bet heavily on QUIC/UDP and are optimizing it.

Since we just do p2p QUIC we benefit directly from all improvements in this area.

That is correct. Iroh connections are at L6, individual protocols such as blobs or gossip are at L7.

From the OSI point of view, QUIC itself is a bit of a layering violation. It covers transport (L4, Reliable ordered delivery, stream multiplexing, congestion control, ...), session (L5, connection establishment and lifecycle, path migration, ...) and presentation (L6, encryption).

And of course below that we have the ability to provide custom transports.

This was done intentionally in QUIC to provide more control. The application layer doesn't have to care about what goes on below, but for some advanced use cases it can know what's going on and even influence which path is being used.

QUIC/TLS being such a comprehensive and well tested package allows us to delegate a lot of the work and just add a tiny bit of logic to make it peer to peer.

Although delegate is not exactly right, since we ended up having to write our own QUIC implementation, noq, to support QUIC multipath...

Previous similar attempts were often not pragmatic or frugal enough. They cared about peer to peer purity more than about it working under all circumstances. They also frequently overabstracted things.

So we got into the sad situation that people associate peer to peer connectivity systems with having to frequently debug the entire stack and having recurrent performance or connectivity issues.

A part of the motivation for the iroh team is to change this notion by being very pragmatic and minimalistic. E.g. the use of relays vs. enlisting other peers to help with hole punching.

We wrote a small demo app called sendme to show off iroh. It is cli only, works on all operating systems that we use internally, and we frequently use it to send around qlog files:

https://www.iroh.computer/sendme

There are several projects inspired by sendme that use the same protocol but add mobile device support and a GUI:

https://www.altsendme.com/en

https://github.com/ARK-Builders/Drop-Desktop

https://github.com/zignig/sendme-egui

I love mainline and also really appreciate how lightweight it is by just using UDP packets. But UDP packets have their limits. With the maximum MTU you can expect to work, the payload can't be more than about 1000 bytes, which is the limit of bep_0044.

But there are some use cases where you want to store a bit more than 1000 bytes. Not giant amounts, but maybe 4 KiB. Mainline won't ever be able to do this.

Iroh 0-rtt is very lightweight. On the network level you see one or two UDP packets flying in one direction, one or two UDP packets flying back, and that's the end of the interaction. You close the connection, and all state you have to retain is a session cookie on the client side.

It is still much heavier than bare UDP packets, but much lighter than a normal 1-rtt QUIC connection, which is already pretty lightweight.

Here are some details about 0-rtt. The API has changed slightly since then, but the basics remain the same of course: https://www.iroh.computer/blog/0rtt-api

We don't have the post 1.0 roadmap very fleshed out at this point. The team has been extremely focused on getting 1.0 out of the door for the last year. This took longer than expected, as expected :-)

But I can say that some form of global content discovery is in my personal definition of done for the project. I'm the BLAKE3 guy at n0, and I don't consider blobs done without global content discovery.

But it is a hard problem, and our standards for "it just works" are pretty high.

If they are both behind the same CGNat, you can use mDNS to help them find each other.

If they are behind different CGNat, you need a party that is reachable by both to help with hole punching and/or relay traffic. This is fundamental, there is nothing you can do about this.

Some p2p protocols try to enlist other peers to be that third party. E.g. holepunch.to . Iroh uses dedicated relays for this.

The relays can either be the n0 public relays, a n0 paid relay network, or self-hosted relays. No matter which option you choose, every iroh endpoint is able to talk to every other iroh endpoint unless they are fully airgapped from the internet.

We have some customers that do use iroh inside data centers.

You get the simplicity of being able to freely move nodes within the data center or even across data centers without having to reconfigure ip addresses. And once a connection is established the performance is comparable to a normal QUIC connection.

Regarding client server architectures: I frequently build systems where you use p2p connections but have clearly defined client and server roles at the application level. Absolutely nothing wrong with that, in fact I think a big problem with existing p2p projects is that they try to be p2p at application level and overcomplicate things.

At the lowest level it is a creative way to leverage all the work the major cloud vendors have poured into QUIC for p2p connections.

If you look at an iroh connection in wireshark it is just a QUIC connection. If you configure a SSLKEYLOGFILE so wireshark can actually look into the packets, you will see a few TLS extensions and somewhat unusual packets flying by during the handshake, but once established it is a completely normal QUIC connection.

That is also why we are relatively confident regarding encryption security. It is just TLS. And we can also leverage new encryption like post quantum key exchange with just a few config changes, without any code changes.

See https://www.iroh.computer/blog/iroh-post-quantum-handshakes

One thing that is genuinely novel is that we use QUIC multipath to keep the different paths (relay, various direct IP paths) separate. This has some technical benefits because the congestion controller does not get irritated when the underlying transport changes. Each transport has its own congestion controller.

Very cool project. I seriously wish one of the printer vendors would use iroh to come up with a network printing standard that just works.

We constantly have issues with our printer. I have fewer issues with my bambu 3d printer than with my (also very expensive) epson inkjet.

These people need to get their shit together, seriously. The way things are going we will have full AGI before we have working printers.

If somebody want to do this, please reach out to us. We are eager to help. But this is one of the most annoying things ever with computers.

Iroh endpoint ids are just Ed25519 keypairs. We don't want you to charge per id, and even if we wanted there is no way we could.

We charge you for hosted relays and additional insights into your deployment, but the core is and will always remain free. Everything is licensed MIT and Apache2.

Yes, exactly.

Our commercial offering provides more insight into your iroh deployment as well as a hosted relay network. At the enterprise tier you can also get priority access to our engineering team.

Obviously we want to make people aware of these services. But we also have projects that use iroh at large scale without using iroh services.

DynDNS does not work for devices behind a NAT unless you manually set up port forwarding.

Iroh gives you the connectivity of DynDNS without any manual configuration.

We did have golang bindings in the past, but had to pause all bindings because keeping them up to date was not viable. Now that we have a stable API, we will revisit this.

If there is enough serious demand we could publish go bindings. Iroh is a rust library that is very easy and efficient to embed into golang binaries.