Building multiplayer games with Socket.io and HTML5 Canvas
github.com
github.com
- Ignore socket.io, do pure binary websocket
- Do not serialize JSON, or use msgpack or protobuf - write your own protocol.
- Write tests for protocol, you need fast iteration. It's super easy to write tests for codec.
- Quadtrees are kinda lousy compared to r-trees.
- Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view.
- Keep client updates under MTU payload. This is basically impossible with JSON.
- Mutability is the way to go for performance.
- Share code with backend and frontend, you'll need it for client prediction.
- Prefer fixed entity speeds.
- Don't log per-frame in production, journalctl will eat up your memory.
- Write game bots to test your load non-synthetically.
- Test on your VPS, you'll be surprised how underperforming CPUs are in most providers. Neighbours are probably serving some CRUD and are very likely not to be as noisy as you are.
Now, all this might not apply depending on exact gameplay mechanics you have, but it works for me. I have Node.js serving 200 clients and 10k entities with vision (can run at or from you) at 20 frames per second that keeps under 100MB RAM with no hiccups. I think I could use something like Golang for some more advanced physics though.
A question to all reading - I'm considering adding an option to mine Monero in my next game. It'd probably be a flip-switch between ads and mining right there on the home screen, with some intelligent throttle/thread decisions depending on the kind of device game is running on. I'd appreciate any feedback regarding the sentiment on that.
Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.
This was invaluable to almost all multiplayer games I did.
Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).
Do what you've gotta do to get paid for your work, but if I found out this was happening it would strongly discourage me from remaining on the website.
"We're cool with adblock, but to continue on this site click here to enable mining, here for paid accounts, or just disable adblock and report inappropriate ads so we can forward that to our syndication."
I'm betting on hearing some horror stories fairly soon as the first gen is rolled out.
Mining => You WILL murder the battery of your users. You WILL cause overheating of the devices. You will also grind the browser to a halt on all hardware of moderate speed or already busy for reading a youtube video in the background.
Personally. I'd consider that any website that start mining when opened should go right into an adblock list, if not blocked as malicious entirely.
For experiment, I've turned on a miner in one tab of my brother's 7-year old modest PC setup. He went for several hours without noticing it, while doing blender rendering as well as somewhat intensive photoshoping.
Chrome is going toward more and more aggressive CPU throttling of tabs in the background. If mining becomes a thing, we're going to a world where background tabs will stop being rendered at all.
I did this on my brother, for data gathering. It'd most certainly not be implemented non-transparently in the end-product.
> Chrome is going toward more and more aggressive CPU throttling of tabs in the background.
That's why you mine during active game. The more people love it, the more they mine it. It can certainly be made mostly win-win scenario, which is why I aim on, again, transparently experimenting with it.
> ...just buying the coins directly?
Realistically, there's never such a thing such as "just buying coins". You need payment processor / security / refunds / authorization / ...
There is, however, something close to "just mining coins". This is one of the fundamental reasons cryptocurrency is attractive in the first place.
Don't get me wrong, I've considered similar things, but I'm pretty uncomfortable with the idea for the above reasons. Plus, whatever people might tell you, even the absolute best JavaScript engine is slow by comparison with other languages.
Example: in the last few weeks due to #reasons I've ported a measure calculation engine for market research data from JavaScript to C#. At first, in many ways, it was a like for like conversion, in that I preserved the data model and general mechanism and flow of calculations, albeit that I applied a much stronger separation of concerns. Nevertheless, with no optimisation, it was roughly 2x the performance of the JS version (V8, of course) running on the same machine. I then started to take advantage of .NET's strong typing for numbers and strings, switched out dictionaries for strongly typed arrays using (obviously) integer lookup, rather than using integers as keys in dictionaries. I also fixed a couple of issues where I was inadvertently using exceptions for flow control, and recalculating strings to generate hashcodes (doh!).
What's surprising is that those changes get me to about 15x faster, and that's really just taking advantage of strong typing and more appropriate data structures that are available in .NET. (JS has arrays, and engines like V8 do optimise for them, even though they're strictly just objects with special properties. Whereas in C, C++, C# arrays are the absolute fastest data structure to access, bar none.) Also worth pointing out that judicious switching from dictionaries to arrays absolutely slashed memory use.
I also implemented the kicker - which happens to be language agnostic - which was to change the moving average calculations to preserve and reuse intermediate results. This got me a 10x performance gain on top of the rest, taking me to about 150x. Like I say, that would probably have been the case in JS as well, so it's not really a fair comparison.
Nevertheless, bear in mind that whilst JavaScript is certainly a lot faster than it used to be, it is not fast, even by the standards of other managed runtime languages. And it's absolutely not fast by the standards of well-written C++ or C compiled down to native code.
I'm not sure how much benefit you'd get from a WebAssembly approach, although possibly something worth looking at.
[?] I think they've removed it from their homepage, most likely because they thought it's not fitting to include it there. :/
Probably. But the point is that it's not an alternative to mining for yourself, but an alternative to showing people ads or having people explicitly pay, which is likely to be even less efficient for very small transactions.
If you do it behind the users back, then sure, it might be ethically grey (though how many users care about how much/little electricity the game they play cause their computer to consume?), but the alternative is to clearly show a message saying you do this instead of showing them ads and/or offering them a button to switch to ads instead.
ArrayBuffer and DataView are fairly fast for my standards.
[1] https://gist.github.com/synthmeat/d23595db732d366f658a1981e9...
The problem is less the client, you're right with that. The problem is that server bandwidth and CPU scale up really quickly if you try to do 30fps, no matter what method you use to keep state in sync. If you want any significant amount of players in a realtime game, this will be one of your first bottlenecks, probably before any encoding and decoding.
For less realtime-y games, JSON and also socket.io can be perfectly adequate.
Protobuf is just insanely slow to encode/decode client side. So that's out too if you want fast paced games. And for slower games it is not worth the effort.
Do you have any code samples, please? (Don't need to run or compile, just an example would be handy.)
Also, can you post a link to your game, please? I'd like to give it a try.
In either case, when you're having them doing the mining, I'd recommend full disclosure AND I'd recommend some explanatory text that goes into the details of what you're doing and why.
Something like, "Click here to learn more about mining." That should do the trick. I'd just put said link below the quick blurb/switch/invite to use mining.
If mining generates more income, then you may wish to offer some sort of in-game bonus for those who enable it. You'll have to figure out some way to ensure they really have it enabled, of course. Giving them a reward may entice more users to choose to do mining.
Another post mentioned that this would be a battery killer. You responded that you'd only enable it for certain devices/browsers. If you want, you could probably enable it (as a choice) for all devices and browsers BUT I'd absolutely ensure that you made it very clear that doing so would mean a rapid depletion of stored energy on battery powered devices.
The reason I'd suggest you do allow it is because many people only use mobile devices and, in some cases, will operate them while they are plugged in. For example, I'm typing this on a tablet that is plugged into an outlet. If mining makes you income, I'd be inclined to want to do mining for you - even while on a mobile device. It'd be even more tempting if there were an in-game reward for it.
Sort of related: That might make an interesting funding option. Free games but mandatory mining on behalf of the creator. Obviously, full disclosure is required if one is concerned with ethics. Inside this idea, they might be able to get rewards directly related to how much they mine on the creator's behalf.
One interesting trivia though - for few undisclosed games I've checked the data, with following conclusion - they would've made order of magnitude less with mining than with ads. However, if you include currency trend for those periods, they would've made almost the same. :)
I think this might be indication of a possible close-future tipping point in favour of this model. We'll see.
https://npmcharts.com/compare/avsc,thrift,msgpack-lite,msgpa...
Yes, basically this. I think most of us solve the same sort of problems day in and day out with our jobs; it's unbelievably fun to be dropped into a totally new problem space and have that feeling of complete ignorance again.
1) "Do not serialize JSON, or use msgpack or protobuf - write your own protocol." - Why should you write your own protocol insead of protobuf? If your game has more than 10-20 types of messages/objects than it becomes very hard to implement your own binary encoding for each object. Plus with protobuf is easy to use and the packets generated are really smaller (most likely smaller than your own protocol).
2) "Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view." - Profiling is something that you should always do, with both synthetic and real loads. In your case, my guess is that your diffing code was inefficient or that the load was so small that both computing the delta and sending the message were almost instant, so comparing them would give inaccurate results.
3) "Prefer fixed entity speeds." - I think this is a game design decision and you shouldn't limit your game to this if you don't have too. Also, I don't think there's any issue with entities being able to have a different speed each tick.
And, a question. What VPS are you using? Do you recommend any specific VPS for hosting Node.js game servers?
2) Sure, I might go the way of the delta one of these days - it does still smell like good design. That's why I was underwhelmed to see it underperform first time around.
3) Yeah, design and implementation are never separate. Neither is content and presentation. :)
DigitalOcean works for me now. Wouldn't mind slightly better CPUs, as even High Compute droplets are not all they're cracked to be.
I also used DO for a low-tickrate game (2ticks/second), but it doesn't seem like the servers are able to handle multiple game servers running at 60FPS.
If you were to implement the game again, would you rethink your networking so you don't have 2TB of data transfer for only 200 players?
2TB of data is really not that much. Say you run at 30fps, streaming only 2D position (say, 2 x uint32) of 100 entities to 200 clients...
(64 * 100 * 200 * 30 * 60 * 60 * 24 * 30 ) / 8 / 1024^4 ~= 11TB per month.
You bring some sanity to that with less frames, less entities, naive delta not sending updates for entities, etc. It's really not surprisingly big number.
I'll definitely explore world state deltas in next game, but this works good enough for now.
To add my 2 cents: I think mining Monero is actually very reasonable (you seem to have thought through the impact on your users), and you've nailed the most important thing: making it opt-in.
I'm fine contributing CPU cycles when playing a game, as long as it's in the foreground and I agreed to it.
What's in the 100MB?
Doing some napkin math, take 10k hypothetical game objects that each have as state: pos, vel, acc, rot, ang vel, ang acc, name (32 chars), color, health, mana, armor, level, type, energy, kills, timers, inventory (256 bytes)... etc etc, all together comes in at around 512 bytes each. Even if that's double buffered it's still just 10MB.
Curious where all that memory is going!
well I'm not sure how it could only be 100MB considering that our Angular2/4 Web Application (a big enterprise CRM/ERP) uses 400 MB (no joke).
If you don't mind me asking, Why is this the case? I would have assumed world entity deltas is probably how I'd approach things first...
Yeah, game design and tech to support it are tightly linked, and there's approaches to it from either side.
I don't think anyone has really done it well yet but everyone is hiding what they have figured out so far. I have noticed that certain engines tend to have better success at this, for example source engine seems to have drastically improved, allowing games working on this issue like Insurgency to make some noticeable strides towards being better.
This has me remembering UT over dialup lag...
1. Rooms
2. Disconnects and reconnects,
3. Fallack solutions
for free.
Although not for a game. I trust that its just not made for that kind of performance.
But for a messaging app its actually quite decent.
Now compare that to how much you'll gain from ads. And soon you'll realize one huge issue you're going to have is to maintain your own servers a little bit everywhere in the world.
It's good that you mention this, since I've heard "horror" stories of people working in this space earning about $10k/mo but spending as much and more on AWS.
But single-digit TB monthly outbound per VPS was not that surprising to me. Almost triple-digit one in naive implementation, scared the hell out of me first time around. Spent hours double and triple-checking and measuring. :D
I basically stuck some visuals on top of the example "counting" snippet on the Elm site (http://elm-lang.org/examples/buttons) and made it multiplayer. First player to count to 50 wins.
I didn't use a tick approach, instead the state of the game was only updated on an event sent from one of the clients. I then broadcast the new state of the game to the other clients.
This is what I made http://retrorace.neillyons.io/ https://github.com/nwjlyons/retrorace. Something is not quite right in the backend code. I think there are problems with waiting for the mutex to unlock. The game feels like it freezes sometimes, but it could also just be a latency issue. The server is in London and I've only tested it with people who are several hundred miles away.
I also had plans to build multiplayer snake but felt like latency would make the game too slow.
[1] http://lance.gg
AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players. Is there a library which abstracts away tcp and udp connections?
And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat? How do you prepare for this? I therefore advise against writing your own network stack if you intend to make the service available to the public. Or am I too cautious?
Socket.io's most discerning feature was fallback to polling. There's also some useful abstractions there, like reconnecting, rooms, as well as some useful ecosystem around it, like socket.io-redis adapter which enables you transparently scale horizontally. If your time is at premium, you might find socket.io work for you, but it's not really hard to write everything better yourself, and better tuned to your use-case.
> AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players.
For fast-action multiplayer, there's currently no sensible offering besides WebRTC's DataChannel, with all it brings on board (I've yet to study it in-depth). I'm having high hopes for QUIC to gain more ground.
> And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat?
From what I've experienced first hand, bots/scripts are most frequent enjoyment-spoiler for other players. I've never ran at scale where I'd be interesting target for actual attacks.
This needs more investigations imho, I wonder where this fails so badly; is WebRTC aiming at ~100ms latency?
This is a tcpdump after loading the page:
\time sudo tcpdump -c 1000 -ni eth0 udp and port 9777
1000 packets captured
1000 packets received by filter
0 packets dropped by kernel
0.02user 0.05system 0:12.45elapsed 0%CPU (0avgtext+0avgdata 5472maxresident)k
0inputs+8outputs (0major+1508minor)pagefaults 0swaps
\time sudo ping -f -c 1000 188.166.46.128
PING 188.166.46.128 (188.166.46.128) 56(84) bytes of data.
....
--- 188.166.46.128 ping statistics ---
1000 packets transmitted, 996 received, 0% packet loss, time 14195ms
rtt min/avg/max/mdev = 28.839/38.641/50.433/5.152 ms, pipe 5, ipg/ewma 14.209/41.787 ms
0.02user 0.13system 0:14.31elapsed 1%CPU (0avgtext+0avgdata 2248maxresident)k
0inputs+8outputs (0major+1183minor)pagefaults 0swapsthis is after start up.
TLDR: Websockets are overengineered and waste CPU cycles doing nothing but marshalling. HTML is better than canvas because it can be GPU accelerated and you get components and z-order for free.
Also, what are the limitations by the hosting provider regarding concurrent connections, data transfer, etc.
I am currently working on a poker app using free Heroku dynos but would like to have a budget in mind for when going live.
1. $20 staging at all times, already live [1]
2. $20 static nginx
3. 3x$5 for socket servers on North America / Europe / Asia
4. $5 master server with WS connections to full mesh of socket servers / orchestration / future.
All Debian.
Have no clue what's Heroku like for that, but last time I've checked, it wasn't as good choice as DO was and is for me.
[1] WARNING: might break at any time, by choice or not http://staging.munchies.io
[1] https://bit.ly/ShowdownFacebook (FB+Flash)
[2] https://bit.ly/ShowdownApple (iOS)
[3] https://bit.ly/ShowdownGoogle (Android)
Maybe one day when I need it, and certainly evaluating at any point.
> Also it's not nice that you didn't mention the demo contains a crypto miner...
Hopefully it turned on only for Desktop + Chrome and didn't drain your battery? I simply didn't think it'd come as surprise, considering context of the discussion.
I also did a more action asteroids clone (http://triangle-wars.com), and this would only support I guess 50 players on a $10 server.
The limiting factor seems to be the number of packets / sec that need to be broadcast to all players. If each player is changing state 2/sec, and you have 50 players, then the sever needs to relay 100 packets/sec...
Unlike some other comments in this thread, I didn't find JSON parsing to have any effect on performance.
Rabbit-hole, I tell ya. They're fun, though. :)
Sorry, gfysfm. I hope you enjoy the discussion though!