An Embarrassing Tale: Why my server could only handle 10 players | Hacker News Reader

An Embarrassing Tale: Why my server could only handle 10 players

medium.com

87 pointsjachitla51 comments

I feel like a lot has been lost since the days when I was in my parents garage making TCP/IP games to run over dialup.

1. Binary was the only choice

2. You had to assume packets would get delayed (sometimes 10+ seconds) thus you had to do some fun stuff like try to use math to predict where the player would be based on what they were doing last time you got an update.

3. You had very little RAM to work with so growing memory would be immediately noticeable.

#2 was the primary reason in old game you used to see other players walking into a wall for 20 seconds before jumping half way across the map. Darn n00bs who couldn't afford 56k (joking).

I fondly remember the first time I played a multi-player online game I actually had to dial into my friend's modem over POTS. I then spent the next two months reading telephony documentation trying to figure out how to use the modem to do things like that myself.

</nostalgia>

#2 hasn't gone away, the gaps have shrunken a lot, and we've gotten a little better at predicting actions, but it's still the exact same concept.

Definitely, but other methods, such as lockstep, are also used in some games.

That depends on the kind of game, mainly on whether it is practical to replicate whole state relevant to player at least occasionally and whether the latency of lock step is acceptable. For RTSes the whole game state is usually impractically large and latency is not that important (and can be masked) while for shooters the state is relatively small and latency is critical.

Just a side note; what you're saying is true (as in RTS vs FPS) if in general if I'm just attack moving my army into yours and then ignoring it.

But if I'm micro-ing for example my single marine vs your single zergling then latency is just as important as in any FPS. The interesting part with RTS for me is that a person can only control so many entities, so only 'touched' entities need real-time syncing, if its for example off-screen a server simulation can just tell you the results afterward.

I quite enjoyed this classic old article (1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond): http://www.gamasutra.com/view/feature/131503/1500_archers_on...

In RTS you can get away with latency between player input and the thing actually happening on the order of 100-200ms with visual tricks and thus serializing all the player inputs and replicating only that to independent simulations for each player running in lockstep is viable. You can't do that for games which need faster simulation updates than network round-trip.

Don't worry, it hasn't been lost by game developers in the professional space. The article is presumably from a web developer implementing their first game. No offense meant by that statement, but if you're reaching for Socket.io (or even TCP/IP) for a real-time multiplayer game then you're probably going to have a bad time.

Thankfully we have free resources from professionals in the industry to keep us on the right track:

https://gafferongames.com/

I remember reading it some time ago, and I gave it another pass, but it seems the only existing solution is WebRTC which, as the author himself admits, is extremely complex for the needs of a simple game.

And netcode.io, the solution developed by the author, which has a major drawback: it needs the installation of a browser extension to work (only chrome supported at the moment).

All things considered, WebSockets still seem to be way to go for most use-cases.

Sure, if you're targeting the browser. But even then you probably don't want Socket.io vs just using the vanilla WebSocket API.

Basically the browser isn't a good platform for real time twitch control games like this one seems to be.

I use Socket.io just fine for my game, though I do implement it differently and send binary packets of my own protocol instead of JSON. I can see what you mean for new people in the field using unmodified events though.

#2 is called Dead Reckoning [1], in case anybody wants to know more about it.

[1] https://en.wikipedia.org/wiki/Dead_reckoning#Dead_reckoning_...

http://knckout.io/ is the link for anyone just wanting to play his game.

Also, is collision detection currently working? I mean, just trying to keep on the platform is actually pretty zen, but running into other players doesn't seem to actually do anything right now.

Collisions only do something if you boost into the player (press space).

Or the up key-- It's a nice, simple game. Good job.

Thanks both of you, I didn't notice this.

It depends on whether the players are boosting. I think it goes like this:

- If neither is boosting, the players just pass through each other

- If one player is boosting, they'll knock the other off

- If both are boosting, the players just bounce off each other

I find the controls extremely unsatisfying. Top down driving games (which this fundamentally is) really need a sense of momentum and drifting.

The lag is so high that I can't "boost" without going out of bounds and dying

what region are you in?

Really enjoyed playing this game. It's really hard but pretty amusing. A good example of multiplayer gameplay distilled down to the very basics.

I noticed that currently json is used to encode messages between clients and the server and also a lot of information that is not actually needed in every update is included, such as the color of each player's car, their name or the number of lifes left.

How much could be gained by transferring only the information that is actually needed and using a different serialization format, perhaps protobuffers or the like?

I got 2 to 4 orders of magnitude improvements across the board last weekend just by rolling my own binary protocol over ws, with world deltas. Didn't even bother to space-partition it to clients.

Using JSON as serialization format for real-time multiplayer game is being very silly, unless it's just a prototype.

yes json is crazy. protocol buffers are ok. but custom bit packed format is still helpful for twitch games

Using JSON for a realtime game is extremely bad. Each request must not only be parsed (json-> object -> json) but each serialization requires a byte by byte encoding pass. Meaning that you're going to use a lot of CPU. Binary is a much better choice.

could have used msgpack http://msgpack.org/

Messagepack still contains field names as strings, which can easily double or triple your data, and explodes parsing time.

A game is a pretty good use case for predefined schemas like Protocol Buffers, which is used by CS:GO and Pokemon Go for example. Other games like slither.io use binary over websockets without any of these libs.

Have you thought about using a profiler?

Precisely this. The advantages that can be gained from detailed and effective use of a profiler can't be underestimated.

You no longer need to guess at where in your code something is taking up CPU or memory, just look. I'd say the ability to effectively use a profiler is one of the most important skills for a software dev (yet one that many people overlook).

I've found that in the length of time people have spent speculating about what's slow and what's not slow, I've measure it, and have proof. Another advantage of profiling is that it improves your intuition about what will be slow. When I profile my existing code add, I tend to see the same things come up over and over again - vector resizing, overly broad mutexes, wrong data structure choice. Seeing these every time means that I can often pre empt these issues, but I always profile it to get hard. Numbers before and after.

If you aren't using a profiler and benchmarks when analyzing performance problems you're just blindly stabbing in the dark.

Just a comment on the gameplay, the characters are moving way too fast for the size of the screen, the whole game is just too unstable and random.

What does OP mean when they say "an io game"? They reference a few games that have .io domains, but the domain has nothing at all to do with the technology or style of game (or even whether it is a game at all), so I'm a little confused.

Usually a free to play multiplayer HTML5 Canvas Game.

There's not really a meaning of "io game" other than by example. It doesn't necessarily have to have a .io TLD. Here is a community with some examples. https://www.reddit.com/r/IoGames/

Multiplayer long-running competitive games in which you kill/consume other players to become stronger, generally. Agar.io was among the first, and is mostly representative of the "genre" - people wrote a pile of clones, improvements, rethinks, etc, mostly with .io domains, so they wound up being called "io games".

Check out https://developer.valvesoftware.com/wiki/Source_Multiplayer_...

Quick suggestions on usability, maybe start off the game with players on a slower speed so they get used to moving around first and then gradually speed up by accomplishing actions that you want them to do in a training sort of manner so that players start learning and don't get confused.

Side note: I subscribe to Mark Brown's Game Maker's Toolkit on Game Design and Game progression, here is an example:

https://www.youtube.com/watch?v=2u6HTG8LuXQ&t=4s

I think it would definitely help

I had thought of that. When onboarding a new user, they fly off the map A LOT. Only problem with this idea is, I think it would be a lot of friction for users that have played more than 1 or 2 games (assuming you have the same thing happen to all players on start).

I have been looking for a continuation mechanic though. Speeding up after some knockoffs sounds like a plausible candidate.

Can you show the specific code that was leaking memory?

tl;dr: programmer makes game server for webgame, experiences massive CPU spikes, instead of profiling he randomly changes stuff until he finds the actual problem (not deleting past events).

lol, nailed it

It looks like he's still a student, so this is just the type of learning experience that breaks those bad habits. Hopefully his methodology will be a bit more refined after all this.

People are welcome to see about how many players my alpha multiplayer server can support: www.emergencevector.com

The theoretical limit as configured is 460 concurrent players. Be sure to read the instructions, or you won't be able to shoot, and you'll drain your energy and become helpless.

Sounds like you found an usability issue. Have you considered changing the controls and/or warning the players when they do it wrong?

The former.

Sounds like some form of load testing could help you pinpoint issues. Loader.io has a free tier.

Horrible game mechanics.

Took way to long to understand how to control it and collision doesn't work

use the up arrow to "boost" and knock players off. There is no collision if neither player is boosting.

The controls could definitely use some work though

if OP is reading this, can you share your technical stack, frameworks etc that you're using. I feel like you kind of left this information out. (I don't even know what language you wrote this in.) Thanks.

NodeJS. If you click on "Acknowledgements" on homepage of knckout.io, you can see some modules I used. You will also see what was used to kickstart the project. Email me at jasonchitla@hotmail.com if you have more questions.

Space partitioning when there are 4 players is a bad idea.

Poorly written article about a poor programmer writing a poorly written code.

Reply on news.ycombinator.com