DoS attacks against my online game | Hacker News Reader

DoS attacks against my online game

hookrace.net

171 pointsdef-95 comments

I'am a Developer of an larger custom modded GTA Server (Roleplay Server), with 500-700 online users in peak hours. The community is very toxic. In the beginning, we received several DDoS Attacks daily, so we decided to build our own protection. We are working with IPtables and IPSets, as you are. But we dont bother ispecting the contents of a packet: Our users are logging in to the Server over HTTPS to a dedicated Server, which is protected by Cloudflare. Once they verified their identify, the Server sends a Request to the Main Game-Server, which then adds the User's IP (from which the Login Request came) to an IPSet. We setup IPTables to only accept packets from IP Adresses in this IPSet. When the IPSet is updated, the Login requests finishes and the User is connecting to the Main Gameserver. Also we are unsing UDP only for Game related traffic, so we completly blocked SYN and TCP related packets in our Hardware Firewall on the Gameserver Port (its a Hetzner Feature). Since then, we had no more major outage, besides sometimes some clients are dropping, because an attacker clogs our 10GB line. Of corse, this can't hold against every attack, but so far this is enought to protect us.

> because an attacker clogs our 10GB line

Yeah that's the problem, attackers get 10+ gigabits UDP flood quiet easily these days and with that they simply hammer down your line, well before any of your software protection mechanism could react.

I worked on a mid-sized online game a few years ago and we experienced several DDoS attacks. I recall one employee tracking down the specific botnet that was rented for the attack against us and we calculated the attacker probably spent a couple of hundred dollars based on the rates.

IIRC, we eventually used AWS Elastic Load Balancer to just soak up the attack, which was a pretty basic SYN flood. Then we waited the attacker out until he got sick of spending money. That temporary redirect definitely impacted performance and cost us some money but it pretty well mitigated the issue. We also spent a bit of time optimizing our servers to drop obvious nonsense requests as quickly as possible but in the end the ELB handled most of the issue for us.

Fascinating that there are people out there with disposable income for the sole purpose of griefing someone’s project/product.

I think it can be cheaper to initiate these types of attacks than to defend against them. Just pissing someone off with an extra $20 to spend can give you a bad day

Well damn. The more you know.

Real life amplification attack, eh?

Offense in cheaper than defense even in non internet related situations, hence societies with high levels of trust between its members are able to advance more quickly due to not needing to waste so many resources on defense.

Low trust societies have their advantages. They are less likely to develop into dictatorships because factions don't trusts other factions enough to give them too much power.

I doubt it's their income they're spending.

I don't know about gp's particulars, but over the years I'm pretty sure I've seen competitors do this on strategic dates to impact the userbase game perception.

If it's an online game, people might do it to boot other people out of the game, especially enemy groups. Kind of as a joke.

Or so I've heard

Out of curiosity as well as sheer ignorance; isn't AWS Shield designed to help with DDoS? was it more complex or expensive or not applicable?

Why was no attempt at legal action taken?

They mention attempting it in the post. It didn't go anywhere one time, as it was a minor, and the other they don't have anything to tie the attack to a person. This is a pretty common problem for small games/services that experience outsized problems like this. It's incredibly time consuming to deal with and legal action typically costs money. If you want to go after someone and you think you know who it is you need a lawyer and you need to get a case into discovery so you can get a lawyer subpoena power to contact ISPs and such. Even if you have their real IP-- reporting to police, or ISP can result in no action, even if you put in a lot of work.

Not sure if this is helpful, but I remember reading about the tribes network protocol years ago. IIRC it also UDP based and essentially had a mechanism to request a proof of work depending on server load before accepting any complex packets from a client. You could probably require some proof for their source IP utilizing the HTTPS request already used during server browsing.

Edit: Found it. http://opentnl.sourceforge.net/doxydocs/history.html (the „puzzles“)

The proof of work idea is cute, but at this stage it's not necessary as profiling seems to show attacks don't get too deep into the netcode before getting dropped. It's hard to know without testing but I'm fairly certain any kind of PoW would cost more to check. This reminds me of a blogpost that is at least fun to read: http://ithare.com/udp-for-games-security-encryption-and-ddos...

Regarding the HTTPS request thing, we already did this for an event and I'm currently getting it set up on all servers. The blog post forgot to mention this one :)

> any kind of PoW would cost more to check

Cost more to check than it would for the attacker to generate an _incorrect_ proof. I'm sure that's what you mean, but my brain first went to NP problems.

And actually, now that I think of it, maybe, maybe not. A PoW can be pre-calculated (possibly using off-peak cloud resources for very cheap) and then stored in a lookup table. It can be reused until the attacker actually solves the problem. Then _maybe_ you could offload that verification to a cheaper, harder to DDoS service, like a cloud function that won't charge you for SYN flooding.

But then the hard part is letting legitimate users bypass the check after doing their PoW, but not letting an attacker through.

[removed]

Prime numbers are expensive to generate and writing a reasonably efficient factorization algorithm isn't that easy either. It's far faster on the server side to just spit out a nonce and have the client generate a partial hash collision up to an arbitrary number of bits (ie, the same proof of work many cryptocurrencies use)

Amusingly every other page on that site is giving me a Cloudflare access denied error, saying "This website is using a security service to protect itself from online attacks."

You should look into OVH. They have those gaming server and they handle the DDOS protection. I was getting hit by DDOS before, but since I moved there, nothing (except an email from OVH to let me know that my server is being attacked and that they are filtering my trafic). On the server itself you just don't feel anything.

Edit: I should add that the DDOS protection is included with the server rental and there is no limit on the size or duration of the attack.

> Instead of cheap VPS servers we have tried getting dedicated servers at larger European hosters like OVH, Hetzner, ihor and NFOrce. The idea is that we have exclusive resources, so the chances of us impacting other customers is lower, and thus we won’t get nullrouted so easily. Largely this works, but the available network bandwidth (usually 1-10 Gbit/s) as well as CPU usage become the limit.

I don't think OVH is viable in this case, they do mitigate the attack but in my personal experience they also mitigate legit traffic during the attack.

Mind you, this is a process using a single port, with only around 100 active connections. You'll easily see half if not more lose connection during a DDoS attack.

I did notice that a few years back, but now when the filter activate I no longer see any drop in bandwidth usage or any customer complaints. How long ago did you experimented with their filter?

Hmm good point, I haven't had any complaints for awhile. Maybe I'll look into it again see what happens.

Thanks for the heads up :)

I feel like I have to throw OVH a bone here.

From running a service with 50-100k concurrently active connections on a single VPS on OVH that has shrugged off a lot of attempted DOS attacks over multiple years, I have the impression that OVH handles DOS attacks exceptionally well. Specifically I've never seen it drop (a lot) of legitimate traffic.

In comparison to OVH, Hetzner (which this game seems to be using), is utter garbage when it comes to responding to any kind of incident well, or at least predictably. Their responses range from doing absolutely nothing, to nullrouting you, to terminating your service. With OVH I at least know how they'll respond to various things and they're (with few exceptions) professional about it, even if I don't like it.

I'd say you get what you pay for, but OVH (when comparing dedicated servers) aren't much more expensive.

Funnily enough our experience has been much the reverse. Hetzner will let us use the 1g dedicated link they promise however we want. Most other hosters will put blanket filters that are too broad or their smart ddos filtering will kick in, which it turns out is not smart enough to learn an arbitrary protocol :)

OVH has consistently been filtering legitimate traffic for us each time we tried them and we’ve tried almost every tier of service they offer.

We've had several servers with OVH, including their kimisufi line, So You Start GAME line, their standard GAME line and their standard servers. While I'm sure these are great for common games their DDoS protection seems to get confused by our very non-standard protocol, ending up blocking most if not all traffic from non-connected players.

Might sound strange, but you could always contact @olesovhcom on Twitter. He is the CEO of OVH and he made change to their DDOS filter base on what we reported to him. He is always interested in improving is offering. But that was maybe 5 years ago, now maybe he will put you in contact with someone else, but back then they were actively looking for feedback to improve their filter.

Edit: you could always contact their support as well. Fighting DDOS on your own it's an expensive/difficult battle. But their DDOS filter is fully custom (mostly Asic and some Arbor as well).

A shot in the dark but maybe implement a wrapper for the protocol in something that looks more like http? Websockets perhaps? Otherwise I think you will have to build your own countermeasures specific to your protocol.

The blog post does mention this possibility. In a similar vein we can also try to mimic a protocol that is well supported by hosters, like source or minecraft but I'm fairly certain that would be the single most ugliest piece of code ever written :)

DoS attacks are something I’ve had to put up with too while hosting game servers since 2008. I run fshost[1] and we see attacks almost weekly. Even though we host mainstream games, we still see legitimate traffic being filtered.

Do you modify any kernel options? net.ipv4.conf.all.rp_filter=1

[1] https://fshost.me

Sounds like modern-day “protection rackets.”

I know that, if you own a gambling site, you can look forward to meeting exciting slavs. I didn’t realize they were taking it to other types of games, but I guess that makes sense. Wiseguys coerce Grandma’s Bake Shop, just as they do Moneybags National Bank.

If you received anyone asking whether you need services that defend against DDOS, and if you refused, those are probably the guys who initiated the DDOS attack.

From the article, it sounds like they know at least some of the attackers already.

I don't know much about DDoS protection, but wouldn't it be possible to add token to UDP packets? So the client first connects to a TCP server and gets 4 bytes token unique per IP. Every UDP packed should start with that token. Since an attacker with spoofed IP can't get a valid token, their packets can be ignored with a cheap check. Might be even possible to use iptables/XDP for it?

We already have a token in the protocol and this is indeed a viable way to help drop invalid traffic. However, none of us are really experienced enough with the networking stack of the linux kernel (nor are these things very well documented) to craft a conntrack module or an XDP program to parse the header and keep track of the assigned tokens. Dropping these any later than the nftables stage is just not enough as we already do that.

We have a small team who work on this project during our free times and most of us just don’t currently have the time to dig into conntrack/nftables/xdp.

I found this tutorial [0] and example [1] how to read/update BPF hashtable data from python bcc module. If UDP packet structure is not too complex, it might be not too hard to implement I guess.

[0] https://duo.com/labs/tech-notes/writing-an-xdp-network-filte...

[1] https://lwn.net/Articles/747640/

It is a standard practice to have a "login server" and a separate "game server". The login server generates the token, and notifies the game server to accept it. The login server handles a queue, and can throttle login attempts. The login server can handle load balancing for multiple game servers, redirecting logged in players to different game servers. Its possible to run multiple login servers too, and restrict the access to specific login servers geographically and to run them with different providers.

"For the individual server infos the client currently has to communicate with each game server by UDP, thus revealing its own IP address without having connected to a server. Since one of the known attackers is running their own DDNet server, they can use this method to collect legitimate player IP addresses and spoof them in their attacks."

Interesting. I wonder if running an overlay network would help there. More choices today for userspace overlay networks. Rogue server owners would still see an IP, but they could only attack it from their connected server, not the internet at large. And you could do some kind of ingress/egress filtering.

Some sort of periodic coordinated switching from UDP port A to port B might help too, like a control message that tells the game client to switch ports. Or randomized initial port assignments combined with filters/firewalling or just in-band 'you're not supposed to send here, bye'.

Have you tried reaching out to the kid's parents?

Or their ISP?

The public prosecutor did. Not sure how much more intimidating we could be than a prosecutor showing up threatening legal action

A few suggestions:

1. Cloudflare offer TCP based DDoS protection too, see their Magic Transit or Spectrum product

2. This sucks, but put your servers behind WireGuard or Tailscale VPN so that in order to connect you need to have authenticated

As the blogpost mentions TCP is not exactly desirable for our project. Moreover, if you contact Cloudflare about those products you'll get a monthly quote that is far beyond what an open source project run by donations can sustain :)

That shouldn’t be the case. Email me if it is. I’ll make sure we help: matthewatcloudflaredotcom.

Thanks, reached out to you.

DoSing small games like this sucks.

Have you looked at ddos-guard's pricing? They seem to be a common budget option.

Thanks for the tip, I didn't know about them. From an initial look $240 for a 1 core vps is a bit too expensive for us.

Also checkout secured.gg [0], I recently discovered them while researching DDoS solutions used by other popular game servers (for Minecraft, and GTA5). I haven't used them but their pricing seemed reasonable to me.

[0] https://secured.gg/dedicated

The cheapest server seems to be about as much as we pay for 24 locations today.

I've no idea how complex this is to code or if it will work for you

but I'll throw the idea out to see if anyone else could improve on it etc..

initial strawman draft idea: have a front door service that just verifies your gamers (eg log on server) This will need to be protected by a Ddos but the throughput shouldn't be large. once authenticated your clients IP address is then passed to some sort of software based firewall protecting each of the main game servers

The firewall would need to be able to handle all the DDoS traffic as well, since your current idea would still pass the game server's IP back to a client. This is doable if you're hosting on a cloud provider and let their firewalls filter the traffic before hitting the game server.

Embark Studios recently open sourced (in alpha) a UDP proxy[1] designed for games that lets you implement a load balancing layer. This allows you to remove servers in the load balancing layer in the event that it comes under attack, allowing the game server to stay up and only having to disconnect a portion of players connected to the attacked loadbalancer. Having a proxy layer is also how Steam protects game servers using the Steam Datagram Relay[2].

[1]: https://github.com/googleforgames/quilkin [2]: https://partner.steamgames.com/doc/features/multiplayer/stea...

Why would someone DoS a small free game?

Testing. Proof that they can DDoS something bigger. For the Lulz. etc.

Because they are a teenage sociopath. (Quoting from the article: "... since the attacker was a minor ...")

That was a previous attacker, not the one now. The article does not give any identifying information about the new attacker except they know who he is, but don't have the data to prove it. Look at the top of the paragraph quoted to see this.

From the hint about knowing the likely identity of the attacker, its probably an old and very specific vendetta. Perhaps an early player who was spurned by the community or developer who fell out? As the attacker has run tweaked gaming servers and things, it’s puts it at the upper end of scriptkiddying.

Some people just have anti-social mental illnesses and this is how they manifest.

Some people are just not great. It sucks. This kind of thing saps the energy from the devs and makes it a job instead of a hobby.

I worked in games for a few years. We constantly got DoS'd. Probably twice a week. Luckily our content was extremely cachable, so once it happened a few times we just black holed stuff through Fastly. The attacks weren't super sophisticated, mostly just hitting a public endpoint a ton of times or trying to crawl API endpoints.

EDIT: Cloudflare already made a game demo on Workers... multiplayer Doom: https://blog.cloudflare.com/doom-multiplayer-workers/ that was written using WASM + WebSockets, porting over an open-source Doom and shimming in a UDP-over-Websockets networking layer. Despite all that, it's still fast enough for a first person shooter.

Original post: Have you looked into using a serverless pub/sub model, like Cloudflare's Workers KV? The example they give is a simple IRC-like distributed chatroom (https://github.com/cloudflare/workers-chat-demo), but theoretically it may work for games too.

Player state can be stored in a decentralized key-value store that Cloudflare manages (Cloudflare Durable Objects). They absorb all the DDoS and handle replication between edge nodes. You don't see any of that. https://developers.cloudflare.com/workers/learning/using-dur...

Then each game client uses a worker to access that KV on a subscription basis, and Cloudflare will route that worker to its nearest edge node and retrieve the state from there (which was previously replicated a moment ago, internal to Cloudflare's infrastructure). Changes to state are replicated across the edge network and pushed to client workers.

https://workers.cloudflare.com/

I don't know if this would result in acceptable latency, but it could help with DDOS at least. The main benefit is that it's incredibly affordable, especially when you're only talking about thousands of players.

Why is ddos still possible? It is possible for isp’s to stop this. There is a proposal for isp level blocking if spoofed source addresses. Also there should be something like an api where I can tell my isp that I don’t want to receive anymore packets from a given source and it should be propagated up the chain.

The key is in the first D: "distributed." A DDoS is designed to look just like legitimate traffic, but coming from many sources all at once. The goal of a successful attack is to both overwhelm the target network by sheer volume, and to make it difficult to stop the attacker without also blocking legitimate traffic. They persist in large part because they exploit the interconnectivity that makes the internet useful in the first place, without which it would cease to be.

It depends, some attacks rely on spoofing source address.. This should not be possible and is easily detectable by ISPs as illegitimate traffic. As far as my other suggestion, it would be a user accessible API. Whomever controls an IP should be able to instruct their ISP what is/isn't legit traffic, so the ISP does not have to know anything.

The firewall can't tell what is legitimate traffic and what looks like legitimate traffic, but the application often can. This is what DOTS us trying to cover: let the application server tell the firewall who to allow or block in real time.

https://www.rfc-editor.org/rfc/rfc8782.html

That exists: DOTS: https://www.rfc-editor.org/rfc/rfc8783.html

This is a nice read https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-...

It is indeed possible for ISPs to stop this, but my guess is that it's cheaper not to :) Large ISPs could require egress filtering for peering with them.

I could see this argument maybe 10+ years ago, but we are almost at a crises level with internet security with all the stuff happening. Legislation should be passed if needed to mandate this technology.

100% agree with you. We gave the ISPs more than enough time to get this under control, yet they don't seem to want to bear any cost in preventing what are essentially crimes (though as shown in the blog post cybercrimes are seen as a bit of a joke unless they cause monetary damages) taking place on their networks. If they are not willing to self-regulate it's the governments job to regulate them.

It's also telling that ISP's are fine with charging you per GB data you consume but apparently don't care about multi 10's GBS+ attacks that presumable consume tremendous amounts of bandwidth? I guess it says something about the true cost of data...

It is likely that they have plenty of spare outbound capacity, especially given how most home connections are asymmetric with respect to upstream.

As for their charging, that is called "value pricing" - what matters is the value to you, not the cost to them.

Cloudflare supports HTTP/3 apparently. Could you run the game on WebTransport over QUIC?

Is this game a really big thing in Germany? Seems like all the major teams are German.

A mention in a German gaming magazine very early on in the games lifetime brought a fair bit of attention

Historically Europe and South America were our main locations, but recently we have grown mostly in East Asia.

Player should be authenticated first to be able to to talk to the gameserver ( I'm talking about the accept() syscall ).

1h timeout is way too long, you should not have more than a couple of minutes max.

I worked on some popular online games and it was a combination of 1) + some IP tables rules ( to allow the traffic ).

Too many login would block the IP etc ...

With proper auth ( based on TCP ), IPtables, kernel tuning you can get a lot of good results without doing some complicated things like UDP proxy / relay / load balancing.

Have you found any good resources/write-ups on the best practices for this sort of thing?

Unfortunatly like a lot of things in video games it's a lot in-house r&d, public doc for those kind of problems is non existent.

The idea from my original post is that your gameserver should allow traffic only if the player is authenticated.

>> we are a free online game and thus have no measurable economic damage, the public prosecutor left it at a sternly worded warning.

The prosecutor is a #@*&%! : your time costs money. Working outside normal office hours is costly.

Maybe you need to setup a contract between the "organization" that runs the servers and yourself that states how much time (and money) does it cost to run the game.

Not to mention the damage to the service providers that have their other clients affected because of this. Besides, if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20

> if someone robs you, would the prosecutor handwave it away saying it's fine because they only got $20

Yes.

In fact, when my mother's grandfather was in conservatorship, his conservator (my mother's cousin) embezzled over $600,000 from his estate, and the prosecutor refused to prosecute.

We were able to get somewhere over half of the amount back through a civil suit.

Don't ever expect that the prosecutor is there to help you.

Monetary boundaries are quite common in law. The "grand" in the video game titled Grand Theft Auto refers to stealing of a car worth more than $5,000. Stealing of a car worth less than that is simply theft (of an) auto. There are a lot of factors that go into a decision not to prosecute, but there are many petty crimes that go unpunished and unprosecuted. (There are also some that are prosecuted, but that doesn't disprove that some aren't.)

i thought valve offered protection against DDoS attacks, or it's not available for free titles?

I actually received a nice email from someone at Valve about this following this post and we are currently evaluating if we can use it: https://partner.steamgames.com/doc/features/multiplayer/stea...

SDR works well and the latency is decent, but this would probably force you to drop the non-steam release or make some of the servers steam-only?

Documentation seems to say that they let you use the network from any platform as long as you have a Steam release. The bigger concern is that this requires linking with a closed source library, which means the open source version of the client could no longer connect to our servers without linking to a closed source blob.

That is sort of a non starter for us without some workaround like maybe hosting our own relays for the open source clients.

Ah! I was going to this actually. I discovered it when Bungie switched Destiny2 over to use it.

Their approach is quite simple and effective, it is what I suggest game developers do. For reference, it goes somewhat like this:

The clients connect to a relay server that just forwards the packets back and forth between the client and the real server. The client never gets to know the real server IP, preventing attackers from DDoSing the servers. If the connection to the relay server drops (which can easily happen if the attacker DDoSes the relay server instead), it can easily resume the connection with any other relay server, and the real server never notices it dropped.

This relies on the fact that there are too many relay servers to DDoS at once, and attackers never get to know the real server running the game code, so they can't make it unreachable.

In the US, Cogent offered $900 per month promotion for 10G commit on a 100G burstable circuit. If you rent several rack units and put a used Juniper hardware firewall from eBay in front, that may be one of the cheapest solutions to absorb 100Gbps attacks

Make sure you won't use the 100Gbps in the full month because it might DDoS your wallet ;)

Maybe try implementing a reverse proxy, it would mask your real IP address.

I wonder if using some sort of port-knocking scheme would raise the barrier for these attacks enough.

It would raise the barrier slightly, but in the end the attackers can try out all ports quickly and check which one causes lags.

Reply on news.ycombinator.com