Fixing the Internet for Real Time Applications, Part II | Hacker News Reader

Fixing the Internet for Real Time Applications, Part II

engineering.riotgames.com

88 pointsrayascott20 comments

The big cloud providers also do similar kinds of things to bring down latency on their networks, but I wonder how effective they are? I've never seen latency comparisons between cloud providers - but there must be differences. None of them seem to use latency as a selling point either. I'd add latency to one of the criteria when choosing between providers if there were a way to compare them. I'm not talking about their latency to me or any arbitrary point, but their latency to internet users in the US in general. Like 50% of users within 80ms RTT. Also packet-loss rates would be nice to know.

Yes, I was actually suprised that all graphs were going up :) Number of users increased -- great but ping times and packet loss should be at an all-time low now and these are the graphs that we want to see!

I think you read the graphs wrong, they are % of players with better than 80ms ping. So going from 30% to 80% was an incredible improvement!

Hello,

I am the author, you got that right. And thank you for taking the time to read it.

Peyton

I wonder if there's a long-term goal here of becoming the AWS for gaming and other realtime network needs.

About to go back and read Part I so please forgive if that's covered in the first part.

I'm from the VoIP world and we share this pain. Kudos, this must have been a huge undertaking.

It was, but so much fun and really worth it. :) I am the author. Thank You for reading :)

Peyton

How is this different from Akamai's IP Application Accelerator (https://www.akamai.com/us/en/solutions/products/web-performa...)? Honestly curious.

I would love to see a consumer videoconferencing service built on top of one of these networks.

Hello,

I am the author. The difference is that Akamai is working on optimizing bandwidth to their customer from a pipes size perspective, not a latency perspective. In other words, they want the widest highway possible, maybe not the fastest.

Peyton

Interesting, thanks for that.

You guys should really consider making this a platform that anyone could use!

I'd be surprised if they haven't already considered it. Building a national backbone is a large endeavour, and now that it's built, a great way to diversify revenue. Games only stay popular for a finite period of time.

Interesting read. Could you blog about the math/techniques/algorithms you used to determine the best fiber paths?

Consider writing a SIGCOMM paper about it!!

We had to RFP each link, meaning we reached out to every fiber/wave vendor that was present in the data centers we wanted to be in that we could find, and had them submit routes with latency data. Then we tested to verify their data. It is really manual unfortunately.

How is this "fixing the internet"? This is just good old network optimization. And a local one, not "internet-wide" by any means! Colossal undertaking, sure, kudos to Riot team for pulling it off. But... I don't see what makes this different.

I'm pretty sure that phrase is not meant to be taken so seriously.

Hey,

I am the author. We are building it out world wide, but you are correct, this is for one application. The same blueprint could be used for any other application though.

Peyton

> A reduction in the number of participating routers would lower latency (yay for LoL players), but it would also lower the number of customers they could support.

This is because ISPs oversell their networks. That's great and it makes sense... as long as they can get away with it. But when services like online gaming and netflix gain popularity, the ISP business model is insufficient for delivering quality connections.

Also, nice "pipe dream" pun in the second paragraph. ;)

What prevents you from using GCE or AWS and running chunks of your applications near the edge?

1) Databases - if your app servers at the edge need to talk to your database back at your main datacenter, you've just made performance worse. Replicate the data to the edge? Now it's harder to keep consistent.

2) Too many edges - if you regionalize too heavily, you end up restricting your matchmaking pools to smaller groups. Matchmaking theory says that either (a) quality of matches decreases or (b) time to matchmake increases as your pool shrinks. At some point, to have a healthy matchmaking pool in a game with a wide variety of skill levels, you have to consolidate. Riot already had Euro datacenters (with separate accounts, which solves #1).

3) Performance - cloud low-latency UDP performance isn't nearly as good as you can get yourself. Plus, you have more control (and leverage) doing it in-house.

4) Perception - some level of latency becomes imperceptible, 50ms is getting pretty close to that number.

The TCP/IP internet is irreparably broken for real time applications. The barriers to new entrants wanting to do the same thing as Riot Games is too high.

I've proposed a scalable mesh network, see IsoGrid.org

Reply on news.ycombinator.com