I wrote my own rsync [video] | Hacker News Reader

I wrote my own rsync [video]

youtube.com

118 pointszdw22 comments

The slides: https://docs.google.com/presentation/d/1XqWF7h8ScimDFbQgLcXP...

The speaker Michael Stapelberg had a few blog posts about his home 25 Gbit/s fiber connection that have been discussed here previously

"My upgrade to 25 Gbit/s Fiber To The Home": https://news.ycombinator.com/item?id=31134534

"25 Gigabit Linux internet router PC build": https://news.ycombinator.com/item?id=27792738

I read through the slides and I still don't know why he wrote his own rsync.

It's a bit easier to follow in the video since he talks from the slides more than presents them but as built up earlier in the deck (11-15) the router7 system us is written in pure Go, i.e. there is no other userspace, which comes with several neat advantages over a traditional system but he wanted to merge his routing box and server box as part of the upgrade needed to support 25G. Later (20-21) he talks about how all of the other software already had popular Go alternatives that could be easily dropped in except rsync and how there was a kludge way to get rsync on the box but it broke the advantages of doing the router7 Go only approach to do so and as a result he set out to implement enough of rsync to meet his needs.

The journey and results are much more interesting than the why though. Router7 has some interesting diagnostic capabilities and workarounds specific to issues he's had from the ISP. Also the implementation of rsync itself is interesting in that it outperformed canonical rsync of the time doing the same task without "better performance" being a goal while writing it. There are limitations with this implementation though, it was only made to meet the particular use case not be a general rsync replacement and it doesn't have the latest protocol/hashing changes rsync has which allowed rsync to retake the performance lead since.

The short sum up would be "because he had an open router userspace written in pure Go and rsync didn't have a suitable pure Go implementation at the time". No taking over the world or anything like that just good ol' hacking for personal use.

For those who want to avoid youtube:

https://media.ccc.de/v/gpn20-41-why-i-wrote-my-own-rsync

Casually having 25 GBps internet speed... Remind me why I live in the country that has the biggest tech companies in the world + most venture capital, yet my ISP is currently offering 1 GBps max :) (I think I know the answer, yet this is painful to think about)

the answer is regulatory capture.

Usually I'd agree with you, but Switzerland has the advantage that 2/3rds of its population live in extreme density for European standards (426 ppl/km², [1]). The denser the population, the more cost-efficient the buildout of any infrastructure is.

Additionally, Switzerland is all but drowning in money (in fact, they have to fight to keep the Swiss Franc affordable [2]) because everyone and their dog who can afford it parks their dark money there. That influx of wealth is reflected in the services and wages that the citizens get.

[1] https://www.avenir-suisse.ch/wie-dicht-ist-die-schweiz-besie...

[2] https://www.handelsblatt.com/finanzen/geldpolitik/schweizer-...

Although the density thing is true, the entire population of the country of Switzerland is less than that of a large metropolitan area (e.g. London) so while it might not be possible to have Swiss level infrastructure in rural US areas it should be perfectly possible in major cities.

Put more concretely: the population of NYC proper (ignoring the broader metro) and Switzerland are pretty comparable (both around 8-9m people), but NYC is about 50x more dense (300 mi² vs 16000).

But the Swiss poulation is not evenly spread over it's land area.

While that's true, the two densest cities in Switzerland are Geneva (~12k / km^2) and Basel (~7k / km^2).

NYC sits at about 11k / km^2. Unless 10Gbps internet and up in Switzerland is only available in Geneva, there's clearly more than just population density at play.

The funny part is that parts of the rural US have better internet speeds than the cities. Rural Electric Cooperatives do like their fiber. Carrington ND has a billboard proclaiming the availability of 10Gbe fiber to the home.

These are really half-baked thoughts.

1. As you may know half of Switzerland is filled with mountains, so building infrastructure there is actually not that trivial (no, the high speeds are not just in the big cities, they're everywhere).

2. Dark money in Switzerland hasn't been a thing for a while. The wealth comes from sensible policies like a sovereign currency, low taxes and reasonable levels of regulation and bureaucracy. All that money also makes everything expensive, so the extra money kind of cancels out when it comes to doing stuff inside Switzerland.

> that has the biggest tech companies in the world + most venture capital, yet my ISP is currently offering 1 GBps max

These two facts are completely related. A wide array of competition is to be desired, not "biggest in the world."

Lucky to get 25Mbps max here :-(. And no, not rural. In a metropolitan area.

germany?

Same country, but restricted to DSL speeds because the sole vendor who provides near-gigabit service in this neighborhood is Spectrum, whose service quality is so low that AT&T DSL is still a better option.

Go two miles in any direction from our neighborhood, and you have multiple providers with gigabit or near gigabit speeds, including AT&T.

Regulatory capture indeed.

why not use unison as a starting point?

While I enjoyed some of this video, I felt it was, at least, 20 minutes too long. It covered too much of his other projects, and it's title, perhaps, should be charged to note that.

Have to agree. His motivation was interesting but also could have been left out.

In fairness, the title of the video is "Why I wrote my own rsync", not what's reported here.

I have to say I have enjoyed every minute of the video. Just watching what Michael Stapelberg has achieved leaves me awestruck. With his subtle humor I find it really a joy to watch.

Reply on news.ycombinator.com