OpenTTD's Infrastructure in 2023
openttd.org
openttd.org
Was waiting for them to get to the "why is it so complex" part, but after all the details of Cloudflare Pages, Cloudflare R2, Cloudflare Workers, Cloudflare Access, EC2 instances, multiple CDNs, hosted Redis, Nomad, Pulumi, web of proxies and APIs and front doors, a dozen microservices and an IaaS repo to make sense of all of this, it came down to:
> In total, we store over 150GiB of data, transfer over 6TiB of data monthly, have more than 10M requests a month, and serve thousands of unique visitors every week.
Basically my MacBook Pro from 2019 could host all their infra and data and serve the entire load (~3 RPS) with room to spare for my day-to-day work.
For anyone else who is reading the post looking to get inspired – ignore everything they did and start small. A single web service to handle all business logic, hosted on two rented VPS instances which split traffic. Data stored in mysql or postgres with some regular backup. Start scaling only when the load from this setup overwhelms you (and I can guarantee it won't for 99.9% of cases, including the one in this post).
Surely you can understand why people don't routinely host all their project's infrastructure on their laptops, even if the technical specs are enough.
So ... two macbooks then?
> Surely you can understand why people don't routinely host all their project's infrastructure on their laptops, even if the technical specs are enough.
I think the poster was simply highlighting that 150GiB of storage, 6TiB of transfer and around 4 requests per second to an average of maybe[1] 26 users per minute might not necessarily need a K8 cluster and 8 different cloud services.
[1] I'm taking "thousands of users" to mean "up to 19000", otherwise they would have said tens of thousands, dozens of thousands, etc.
But even then, you’re just pushing the cost onto the person maintaining the system.
Cloudflare R2 => Now you have to bootstrap and maintain your own high availability Minio cluster. On multiple servers in multiple data centers for redundancy, of course.
Cloudflare Workers/Pages => Now you have to maintain your own compute runner (granted, could be as simple as a docker container but that still both requires work to set up and transition over as well as maintain over time) and load balancer (once again, just Nginx or Apache but that requires setup and maintenance) to execute and serve this content.
Cloudflare Access => Now you have to maintain your own access control system like ory.
Cloudflare Tunnels => If you’re only running one node, this isn’t needed, so congrats I guess. You’d still need to provide internal access if you have multiple nodes in a HA environment, though.
AWS EC2 => Now you need to maintain your own VMs.
Etc.
This is a volunteer project. Having someone maintain all these things may not be even remotely practical.
It’s all the rage to hate on cloud services in 2023 but they abstract away a lot of operational work and that’s not something to be blindly discounted.
It's an open source game... if the site goes down for 10 minutes every now and then, would anyone even notice?
But also, the point was that a MacBook (or single server) can just replace the infrastructure defined in the article. That is clearly not the case if you’re suddenly losing uptime. That is a material deterioration, any way you put it.
And even if you didn’t choose to go the HA route, you still need to set up and maintain the server and all the things running on it. And fix issues when they come up. Choosing to have downtime does not magically make any of what I said go away; it at most eases the burden slightly.
> Maintaining the server
You make it sound like it's a problem. It is not.
I explained exactly what needs to be maintained and why that’s a lot of work. You are free to disagree, but just saying “trust me bro” is not a compelling argument.
Have you ever maintained your own server? Or even tried? It’s not nearly as easy as you clearly think it is.
> It’s not nearly as easy as you clearly think it is.
Dunno, I'm 20+ years in the industry. It's not easy, though it's just a skill. You can hire a person who would do that for you. It's not necessary to make it look like a wizardry. It's just a skill.
I offer counseling if you're struggling with the basics. For you personally it would be 200% extra though. I don't like unnecessary rude people.
That is in no way shape or form grounded in reality.
Waving credentials around and proposing solutions not remotely feasible for the problem at hand generally does not sway people nor provide for interesting discussion.
At least in the case of Steam, they print money and can still struggle to keep the lights running.
And that's completely fine. Nobody will die if gaming service has donwntime slightly (or considerably) longer than 5 minutes/year.
Edit: I would go so far as to say they could even have a weekly “patch Wednesday” where the expectation is set that there would be an hour of downtime.
HA is not necessary. Having 5m downtime once every 4 years while they switch between the primary and the backup droplet because something went wrong is okay for a game!
The majority of folks running solutions that result in effectively 0% downtime are vastly overpaying for what they get.
1. The web page you're reading
2. Mod downloading and server listings. Sure, less latency would be nice, but is it vital?
In particular, if you join a multiplayer game, that's the end of their server involvement, they're not hosting the multiplayer servers, so it won't have a gameplay effect. Add on to that that a lot of their players get the game from Steam, linux package repositories, or JGR's github releases page anyway.
Users don't care about tech choice.
> 2) Not great exemplars of quickly loading pages
And yet they're some of the most popular web sites on the internet, so does that not indicate that they do so despite the performance and therefore users find their performance at least acceptable?
> 3) Your bandwidth might be bad- I was able to load youtube.com in
Good for you? I'm on a 2GBit/s fiber connection with 3ms ping to youtube.com . I am however, likely located further away from their actual servers doing the processing (while ping measures the response from their edge servers).
You're right, they don't. The user experience is that they're front-loading their wait time loading the app, and aren't doing a whole request/response cycle for every action they want to take. The end result is the net wait for their active session being lower.
>so does that not indicate that they do so despite the performance
It indicates they're trying to maximize features, rather than minimize load time.
>I'm on a 2GBit/s fiber connection with 3ms ping to youtube.com
Cool maybe your computer is slow then. Sample size of 1, anecdata, etc.
And 10 of them would probably still be cheaper.
This turns my question into: does OpenTTD need more performance or reliability for its website?
I can kind of see how DDOS protection might be useful, but... I don't protect my stuff against DDOS: the loss of service is nullified by the effort and risk required to set it up and maintain. What would that calculation look like for a random forum?
I think you can unequivocally agree that a distributed service thats designed to be fault tolerant is going to more more reliable than your MacBook Pro sitting in you closet on your home Internet connection.
>I can kind of see how DDOS protection might be useful, but... I don't protect my stuff against DDOS
If you don't care about your stuff going down, then it doesn't matter. If you don't care about it, then comparing it to a setup where that is a feature, isn't an even comparison.
Even if you hosted this in your own closet on your MacBook Pro, OpenTTD's setup is still somewhat competitive. You might say "Oh I get that all for free" well a MacBook Pro costs money, your home internet costs money (although most consumer Internet is going to push back if you do more than 1Tb up a month- this is the cue for everybody who wants to rave about how great their Internet is to be contrarian below), you're paying electricity, and rent- even hosting your MacBook Pro is a marginal benefit from other expensive you already have, it's still not _free_.
But I've stopped fighting it when I have no stake in it. If it's good for their CV and they can afford the expense, then good for them I guess. It's not my problem. Maybe it's problematic from an environmental standard I suppose, but that could impeach many hobbies.
Same here... once it wouldn't fit in excel (65535 lines + ), it became "big data" :)
It is much more driven by personal career goals than real business or technical incentives.
I also don't usually care as everyone is free, unless they are on my payroll ;-)
> Basically my MacBook Pro from 2019 could host all their infra and data and serve the entire load (~3 RPS) with room to spare for my day-to-day work.
Yeah, their requirements remind me of somebody on a bittorrent forum describing their minimum acceptable seedbox.
> To keep the AWS infrastructure as cheap as possible, we wanted to avoid needing a NAT gateway: if you use IPv4, you need something that allows you to talk with the outside world. On AWS you do this by installing NAT gateways. Sadly, those are (relatively speaking) rather expensive. So instead, we run as much as we can IPv6-only.
If the OpenTTD folks are reading this: this is not true. You can assign public IP addresses to your t4g instances by using a public subnet in the VPC they are deployed to. They will correctly reach WAN addresses via their public interface and you will not need a NAT gateway. This incurs no additional cost. It sounds like your current instances were deployed to a private subnet.
You will not need to do any additional configuration on Amazon Linux 2 or the distribution of Ubuntu those instances deploy with using the Launch Wizard - both from the point of view of the Linux firewall and from the point of view of security group configuration.
This will greatly simplify your configuration.
> Nomad is similar to Kubernetes, AWS ECS, AWS EKS, Azure AKS, etc, but a bit simpler to work with in a day-to-day... For this we run nginx on all clients in the cluster. Nginx uses Nomad’s service discovery information to forward traffic to the right instance.
You are reinventing Kubernetes. This is okay. At the time that these decisions were made, ChatGPT 4 didn't exist - nowadays, if you want, you can "just" ask for Kubernetes manifests, and you will get correct ones, and you will see the light. The keywords for what you should ask for are `flux` and `eksctl`. You can create your Kubernetes cluster with one file & one command with `eksctl`, then `flux bootstrap` a Git repo that will contain all your YAML files describing your application.
Another perspective is, to ameliorate your NAT gateway mistake, you have to be pretty familiar with the AWS and Cloudflare networking details. If you used eksctl, you wouldn't.
Ah, but putting an EC2 in a public subnet is not "The Way"!
We've let AWS convince us to not put instances in public subnets so that they can make money hand-over-fist on NAT Gateways, which are WAY too expensive for what they are and do.
AWS makes money, it's a trade off; you can just as well put up an ec2 as a nat that is able to auto scale if you need to give your servers access. or only attach one during updates, etc.
edit: word
These are actually fully managed services that require very little maintenance and tend to reduce complexity versus non managed services.
> A single web service to handle all business logic
This probably isn't an ideal architecture for someone building a backend for a multiplayer video game.
> ignore everything they did
you seem to be ignoring the fact that their architecture evolved into this state over the course of a few years.
I would push back here. I think even this is ridiculous complexity in 2023 if you just need to serve a webapp or API. Managing VMs is a mistake now. Serverless is an extreme competitive advantage when you are small and trying to stay focused on the customers. I don't have time to babysit self-serving technological curiosities as we try to ramp. There definitely isn't any money in pet problems. There is also a benefit to noob developers - if you constrain yourself to shipping serverless function code, you can't possibly be tempted and fall into weird infra rabbit holes that rob you of your ability to deliver near-term value and learn about practical software work.
Our next-gen architecture consists of exactly 2 things. Azure SQL Database Hyperscale and Azure Functions. We deliver server-rendered HTML directly to the client from the HTTP trigger functions and they in turn connect directly to the DB. I almost suspect Microsoft doesn't like us doing it this way (e.g. mandatory URL route prefixes). But, too bad for them - we worked around it & our hosting model is effectively a prerequisite for the complexity circus everyone else employs. That's it. We are about to have a zero VM cloud infrastructure and someone with 2 days worth of YouTube training could become semi-effective at monitoring it all.
Become the cockroach of technology users. Use the barest subset of what is needed to get the job done, but do it in a clever way. Infest someone's cloud so you don't have to screw with boring things like compliance, audits, power supply replacements, deployments, etc. Use their products in ways that feel manipulative, but are still strongly within the lines of ToS and the barycenter of the overall crowd. Stay away from bleeding edge technology.
Imagine if you literally only had to push code (aka not declarative infra) to GitHub and pay one cloud bill. Certs, networking, patching, backups, monitoring, recovery, scaling up/out, etc all completely handled for you. Why wouldn't you want your life to be this easy? Is it because "fuck Microsoft[0]" that we continue to dig the technological equivalents of ditches by hand all day? Are we just poorly incentivized?
[0] To be clear, you could replace "Microsoft" with "Cloudflare" or a number of other hyperscalars. I am tempted to play around with the CF offerings again. I like the idea of an "emergency backup vendor" if things ever get spicy with Microsoft.
You are increasing the response time by 20-30ms just by using HTTP-Functions instead of plain ASP.NET (lots of abstractions and grpc channel to worker processes). Beyond that you will get long cold starts, no response-streaming and slow scaling. The cost of the consumption-based plan will skyrocket after small number of requests compared to serving from dedicated App Service.
I am not sure what you are referring to with GRPC channels and worker processes.
Because it is even simpler than screwing with ASP.NET Core if all you need is a simple SSR webforms experience (which I'd argue is most of us). I don't even have to worry about a startup.cs file or play games with the types of serializer or pipeline features I want to use. The only thing I have to do is take a http trigger method dep on ClaimsPrincipal and turn on AAD auth in the portal to get all my hard shit taken care of.
Why not return final text/html from a function that would otherwise return application/json to some cartoonish client-side complexity? Why add some API/web proxy middleware bullshit that serves no value? I am not buying the narrative anymore. This works, it is simple, it is fast. HTTP in, HTTP out. Microsoft would have to go so far out of their way to make this not work that it would look absolutely comical. "text/html is banned return content type because we are capricious assholes".
> You save lots of money
I agree. Time is money.
It takes me 5 minutes to spin up a C#/V4 function webapp that is mapped via GH actions to a multi-stage Dev/Qa/Test/Prod deployment pipeline with certs, monitoring, backups, DR, compliance, etc all handled by default.
Can you get your LAMP stack to pass a PCI-DSS audit after 1 hour of work? I can make contractual guarantees that my stack is going to pass those audits before I log into my PC for the day.
Complexity is a weird thing. It is really hard to understand the whole picture when you only look at the technology through a narrow perspective like "what brand is my database engine".
Look, I have been doing things the way you are describing in the industry for 10 years now. Most of the time, the shop really really really does not need it. It is not difficult to set up a regular backup and a cert.
Chris Sawyer's ability to create addictive building games that remain fun to play long after their contemporaries have ended up in the dustbin of history is superhuman, in my humble opinion. Add to the fact that he did it all in Assembly, and it's hard not to place his achievements on a bit of a pedestal.
This often meant a regression in graphics quality (e.g. Baldur's Gate 2 vs. Neverwinter Nights) and frequently a regression in gameplay too (usually due to horrible controls and/or camera, something like Monkey Island 4 is a good example).
That can easily handle TiB of traffic (Say 20+ TiB/month), have a few thousand requests a second (depends on what kind of request), and you can easily parallelize a lot of this by running your backend in a few docker containers on this dedi.
That's, say, $70 including the cost for a domain and such, and all it requires in maintenance is regular system updates.
Auction boxes on Hetzner often come with >1TiB of SSD storage, if you're so inclined.
Then again, I'm the kind of guy to play OpenTTD by making a few simple networks and watching the thing generate stable revenue, and then messing around with building something else, so maybe the mentality is just different.
It's paid for by donations
They go into this in part 2: https://www.openttd.org/news/2023/07/09/infra-migration
I wonder what's a reliable and cheapest way to host content or a website for long term such as 10-25 years.
But I would assume that two hetzner server for 100$/month an nginx properly configured would do the same thing.
Nice writeup cool to get insight:)
The stuff of nightmares. Our $200M/year revenue web app serving >2M daily users runs on a simpler setup than this.
Is it not the case that you just need to use their IP address and you bypass essentially 100% of what cloudflare offers?
I guess a targeted attack is hard (how do you find the "real" IP?), but there.may be speculative attacks just scanning through IP ranges
Then it doesn't matter if someone finds the IP of the actual server. Worst they can do is flood you with instantly-dropped connection attempts, but not probe services or run up your server hosting bill with large data transfers or anything like that. Scans won't find listening ports.
- accept traffic only from the published CloudFlare IP ranges
- connect to CloudFlare rather than accept traffic (Argo tunnel)
The first one unfortunately doesn't protect you from someone scanning from the CloudFlare ranges themselves. You can add a custom header in that case so that any traffic without the shared secret is not accepted.