The Technology Behind A Low Latency Cloud Gaming Service
blog.parsec.tv
blog.parsec.tv
Either you have insufficient capacity to meet day 1 load when everyone piles onto a hot new title, or you over-provision, meet that demand and have much of that hardware doing nothing during doldrum seasons (or when a title bombs).
Probably you need to figure out how to make GPU capacity useful when it's not rendering games, and sell that as a service as well (GPU-based machine learning?). It doesn't help that OSes have been prickly about letting processes share GPU resources; I imagine there are a fair number of thorny security problems, even with GPU MMUs.
This doesn't seem like something a cloud gaming start-up can really tackle; lots of capitalization, with lots of competition from entrenched providers, and no really compelling reason to put games in the cloud to begin with. The bigger (console) players probably realized that having consumers buy their own compute is not only cheaper and more resilient depreciation-wise, but also causes a nice platform lock-in effect once the customers have purchased a few titles.
1. Some games just aren't built for any latency. I'm talking fighting games(1 frame windows), majority of FPS' and anything that has a reaction instead of a prediction game model. These can be made network compatible with a time-rewind gamestate resolution(which many good ones do), however they aren't drop-in.
2. The market that's interested in this tends to be much more demanding than your traditional consumer. Drop ~150ms of packets at a critical time? You've got a really frustrated customer where a game designed for network degradation will handle this gracefully.
Kudos to them for trying but you're not dealing with just technical issues, there's fundamentals of design that need to be adjusted for that can't be done without direct developer involvement.
2) Yes, anything real-time is like this.
The purpose of streaming is to remove the need for the simulation and rendering at all. The client is a dumb terminal that just renders frames and records input.
The moment you start making the client smart again — making it aware of a subset of the world state, making it do its own rendering based on the input — you've just reinvented current client-server gaming.
You still get to run a physics model and game logic though and that is a non-neglible part of modern games.
You can only move forward in the frames you display. You can't walk back in time to evaluate a player's action in relation to their latency and then reconcile game state with all other clients.
That's exactly what properly engineered network games do. However we're talking about video streaming here, in which you don't have access to any game state. It's not possible to do without the developer going in and adding specific feature support for your streaming service.
As for the adjustment... Yes bringing a character back to life after a momentary error, in one place on the map, as seen by one person, is a small adjustment that happens rarely.
Game design is built with these latencies into account, if people were to see the raw network updates they'd be shocked at how jerky and unplayable it is.
Generally if you're building a game to be tolerant of network latency you want a design where you're trying to be predictive than reactive. The former tolerates latency well since you're trying to guess where something will be in the future. The latter has latency in the feedback loop which is incredibly painful without some complex(time-rewind) mitigation measures.
So if you're talking about the "highly latency sensitive" players, then by definition those are only PC gamers, and only a small fraction of those players. The addressable market of gamers who tolerate 100+ ms of latency is very large, definitely large enough to build a business on top of.
Latency sensitive players are actually large part of the market, any action based multiplayer game will fall under that category, which includes FPS games which make up a large portion of game revenue. About 10% of the market makes 90% of the revenue so missing certain use cases excludes a large chunk.
What's the 99th percentile for that? How about packet-loss?
None of this is new stuff in gamedev, we've been build action games successfully since the days of 28.8 modems and 200ms pings.
Maybe that makes me a bit of an entrenched player(which is a poor position to argue on HN) but I've yet to see anything fundamental in these technologies that will address the latency issues in the same way you can with an interpolated client-side simulation.
It doesn't have to be as good, it just has to reach a certain level. And for a lot of games it works fine to have a bit of lead time on controls.
You set up a fixed delay, that is large enough to contain your jitter. It doesn't matter if one packet takes 80ms and the next takes 105ms when you don't display it until the 120ms mark. There will be no visible jitter.
If you were to take a console gamer that was just fine playing a game at 100ms+ latency, and let them play the same game, while magically removing 95% of the latency, they would probably be amazed, and never want to go back to their previous setup with horrendous latency.
If you have a consistent latency of, say, 100ms, you can account for that. People adjust. They can plan for being a little slower to react.
On the other hand, if you're playing a shooter and you're constantly bouncing from 20ms to 100ms, you're likely going to feel pretty unhappy. You'll be forced to constantly adjust for actions happening slightly too fast or too slow depending on the direction/severity of the jitter.
I've been wanting to give this one a real try: http://store.steampowered.com/app/369530/
Sadly my laptop sucks too much for it. It can run it at ~40-60 fps, but i really feel the input latency already.
I'll try and see what it feels like with their AWS AMI, playing on a 25mbit line from hannover, to an EC2 instance in Frankfurt.
Edit: Turns out there's an issue with my laptop hardware, so results from me will be delayed until a fix can be found.
But certainly over wired LAN it does offer near native performance for most games, past the threshold where many gamers would even be aware they were playing remotely. The real questions are: how good will the internet get, and how quickly will it get there?
I think this model has many big features that makes it a worthy, even if it's not for everyone. Here's a few obvious ones:
- No download or even load time, jump straight in the game - Can play on almost any device, even mobile - Online co-op/multiplayer for local only games - Easy to spectate friends (don't need good upload speed)
So if you are a gamer who plays a moba/fps every single day, then sure, having your own rig is nice. But if you're someone who likes playing the latest single player story game, or some indie games, but don't want to invest into a gaming PC/console every few years, this model could be very useful.
In that world, this sort of technology could be adapted to shipping video a short distance, say less than 10ms away. As a platform instead of a service these sorts of things might work as well as a port of the game to your platform of choice.
You throw your playstation 5 into a closet in the basement and you can play games anywhere in the house, or maybe at your friend's house a couple blocks over.
But that's just Steam. Expand that to everything in a couple of years, and you're there. Maybe Valve could get into that business themselves.
A major issue with virtual filmmaking is the datasets are huge, which makes it very, very difficult to work with a remote team, because you can't effectively get the dataset to people's machines even just to work on.
What I'd like to do is host our dataset AND the 3D content creation apps on our own hardware at Switch (in Las Vegas), and then have our employees and contractors throughout California access 3D content creation apps remotely, via something like your streaming approach.
Thoughts?
On the business side of view, how do you plan to make moniez?
Making money, we plan to wrap up the cloud experience for you nicely and take a margin on it. But for personal use it will remain free.
I have a beefy home PC that is showing it's age and I'd prefer to have a SaaS solution that I can access everywhere rather than role on my own!
So if you guys do release in the next 4 months or so...consider me a customer.
Server side I'm not sure. I know the Windows API for grabbing frames is really good, and I assume there's a similar API for Linux. Shouldn't be too bad. But you will likely see a macOS server first...
E.g. do you multiplex into a mpeg transport stream? RTP? UDP?
If i understand correctly you also have a Streaming Server to install in the local network, so it's basically doing what Steam in Home Streaming does ? How does it compare to that in terms of performance ?
A future advancement that could be made possible by this technology is the complete elimination of "laggy hitbox" type effects: if the server that handles physics is also handling rendering, it can make sure to show a consistent picture of the environment to every player.
The "laggy hitbox" effect only happens because the client predicts non-player actions before the action data actually arrives. The alternative is to have actions "officially" occur when the packet hits the server instead of when the player actually acts, which means that players with high latency will be at a disadvantage. With streaming video it seems this problem is unavoidable.
With client-side rendering, the client draws a muzzle flash as soon as you fire, and there could still be an enemy in the line of fire at the time the muzzle flash goes off, and the shot still not hit if the enemy has already jumped out of the way.
Signed up and will be trying it out this weekend.
The absolute lowest ping I can reliably maintain from my house to a major datacenter are those in CA, and it's around 60ms. So, I'm effectively attempting to play a game at 15fps. Workable for some 4x games and a few single player action games, but nowhere near pleasant.
Since my real world pings to game servers are much closer to 100/150ms (10fps) with the occasional burst of packet loss (especially in the evenings), cloud gaming is unrealistic (PS Now basically says "No, you can't use this service").
I'm getting 60ms averages from work (which is on fibre) to AWS-East (where parsec hosts their website from), so there's no way this would ever be viable for me.
Seems like it would have limited appeal to non-casual gamers, if I'm honest. You have to near a major datacenter with reliable broadband, and you could never play competitively in a broad set of latency-sensitive games. On the other hand, I could definitely see the appeal for those who want to play the latest Peggle or Parking Dash games.
- use a video encoder variant with low latency and CPU/GPU usage
- keep video output in video memory as much as possible
- no solution to the problem of getting low latency high bandwidth internet to the customers
Playstation bought Gaikai and turned it into Playstation Now. They also use the technology for PS4 Remote Play.
Its awesome and you should check it out :D
As you admit yourself you can't do anything about network lag and reliability. Even if you get down to 0ms video enc-dec this will be not viable for any games where the camera moves.
Also did a 3 hour raid night on the new Legion content with it on WoW, lot of dynamic camera movement there. Cleared Nythendra on Heroic as the 4th DPS.
May not be to everyone's liking, but worked for me.
While the speed of light isn't getting any faster, bandwidth and hardware encoding both are (relative to the speed at which we're increasing resolution and image quality).
And the really nice thing about it is that you can afford to spend an order of magnitude on the hardware if it's shared around and played casually than if it just sits in your closet most of the time. And you can game on laptops or even mobile devices easily.
The experience could actually be fantastic, and hugely social; imagine reading a review with screenshots from the game, and clicking on them allows you to start to playing in under a second from that exact moment, just like you can send a YouTube link mid video.
The trouble is, there are a hell of a lot of cities in the world. In Chicago, everyone could be gaming like this in ten years; but in rural New Zealand?
IIRC, Gaikai was aiming for that exact application (live game demos) before they got acquired by Sony.
That said, 15ms network latency isn't too bad at all, considering that it takes your TV 40-80ms to display a frame through HDMI. Fortunately, with the advent of VR display manufactures finally took notice of this built-in lag. So that's getting better, too.
I found some people on dslreports forums with Comcast's DOCSIS 3.1 service, and their ping times were around 15ms to a local server, which is still ok enough.
The upgrade process is gong to be slow, but 4 years after OnLive died, there's a significantly higher number of people who have a good enough connection to use it. It sounds like there's a lot of improvement on the server side too, as well as in hardware assisted compresison/decompression. Monitors have also been encouraged to decrease processing latency too, and everything adds up.