I have web applications running that reliably serve 50k users per day and cost me $10/month, running on a single, cheap VPS.
It's great that your web app only costs $10/month but others may have web apps that are more computationally intensive e.g. video processing or ML inference etc or simply can't join everything they need at runtime.
And it's great that you're willing to deny those 50k users a day access to your service when that cheap VPS inevitably falls over. But others may be monetising that traffic and will want a HA solution so their revenue isn't impacted.
All of those add complexity and cost to an architecture.
In my experience, the complicated setups that are justified by the argument of "reliability" have more downtime then a single VPS. The reason is probably that there are more moving parts and more has to be maintained / can go wrong.
These days, a single VPS in the right datacenter has excellent uptime.
I've also had very high reliability rates with a single VPS. They've actually given me less downtime than AWS services at times.
I can't hit four nines reliably with single VPS platforms on my typical workloads, I need load balancers and redundant app servers. I could quite likely hit three nines using single VPSes. But if a client wants 99.9% SLAs, they'll be paying for HA and I'll deploy redundant ec2 instances, multi region RDS, and an ELB. And charge them 3 or 4 times what the OP is spending for it. (And I'll almost always deliver 99.99% availability.)
For my stuff or friends or people I'm doing cost saving favours for, I'll explain how much extra it costs to guarantee less then an hour of downtime a month, the realistic expectations and historical experience of how much downtime an non-HA platform might have in their use case, and often choose along with them a single VPS (or even dirt cheap cpanel hosting) while understanding and accepting the risks associated with saving upwards of a couple of hundred bucks per month.
It still has a redundant slave because I'm not going to bet my reputation on everything going right.
To be fair I can't be sure because a less than 5 minutes downtime would probably go unnoticed, but the fact is I never hear about this server.
> In my experience, the complicated setups that are justified by the argument of "reliability" have more downtime then a single VPS. The reason is probably that there are more moving parts and more has to be maintained / can go wrong.
> These days, a single VPS in the right datacenter has excellent uptime.
Again, maybe in your experience but that's not universal. There's literally no redundancy with running everything off a single VPS and if that datacenter has network or hardware problems, then your service is down.
Is redundancy necessary for the scale of OP's app considering it provides 0 income? Most likely not, but that's a decision they've decided on and there's nothing wrong with that.
What does excellent uptime mean in your book? With Digital Ocean's AM2 region I had regular downtime every few weeks and while I'm alright with it, if I had another VPS in another datacenter it would've had next to no effect on the customer experience. But an hour or more of downtime every two weeks isn't excellent.
What does excellent uptime
mean in your book?
Over the last years, across multiple datacenters, I have seen maybe one or two short downtimes per year. None ever lasting more then 5 minutes.
I know this, because I have gone through these issues with each of my projects. Just recently an infinite loop bug in a cron job ground my "single VPS" setup to a halt (and took the web server with it).
Yes some of the other tools could arguably be worth paying for, but if the author's concern is that he's short on money and $140 is a lot, why didn't they KISS and only use what they need? Then scale as and when needed in the future.
Y'know, do what HN regularly preaches?
And $140/month is pretty good value there probably... Even if that's just being able to point potential employers/recruiters at this blog post as evidence of experience building and running an HA website with more-advanced-than-free-Google-Analytics user behaviour tracking.
Article mentions in a few places that money could be saved - but lowering the cost doesn't seem to be the driver.
Makes sense to me that if you're building the site as a hobby/practice/demo it makes sense to "build it properly" - and it's fun.
On a site currently generating zero revenue, I hope the OP is happily enough paying most of that $145/month as a learning experience or for resume bullet points (which are perfectly valid was to spend your money). They've admitted elsewhere in the comments that the two $40/month droplets are way oversized (from an attempt to solve a problem that turned out not to be droplet size/resource related) - so without redundancy and without AWS hosted metabase, this would be about $100/month less expensive to run.
I still think that's over provisioned or under engineered. Like others have commented, I'd be surprised if the features you can see on the site require any more than the $15/month the FAQ claims it costs to run, plus perhaps the $10/month Discus expenses. That seems about where a hobby/side-gig project should sit for a lot of devs before you start thinking about how to make it pay for itself... YMMV, especially if you're not comfortable earning at least junior dev salary already in some reasonably well paying part of the world.
The fact that this person is spending nearly as much to support 50k users a day as I do to support more than 4 million cannot be hand-waived away by "people are so free to judge other's[sic] situations". The matter is worsened by the fact that the application is so simple that it doesn't even support user accounts. There is room for discussion here about efficiency in application architecture. More importantly, an article billing itself as "Costs of running a Python webapp for 55k monthly users" is silly because there is no way this is representative of anything. I'm afraid new hackers will be scared by the high costs listed here and be discouraged in their own efforts.
EDIT: Please disregard the above. I need an eye test, or maybe just to put my glasses on! Daily users are 3.4k (3400), not 34k. My apologies, I take it all back!
Your media files must be extremely small.
I'm guessing they're less than 20MB on average, because a) CF hasn't shown you the door yet b) they don't even cache anything bigger than half a gigabyte.
Another consultant that came in and try to do this, made our costs go up by 10x per month… so i'm not surprised when I see stuff like this here…
The knowledge of ones tools available at ones finger tips and the relative costs of such seem to make the difference for these things.
There’s statistically a 95-99%+ chance you’ll never get to 55k monthly users with your app so don’t worry!
Yeah, from your perspective, for others monthly savings of 10$ is allot, and not everyone earns 200$ for 4 hours of work.
And you're less likely to make mistakes like upgrading your server instance to try and solve a problem that can't be solved like that.
And the more you do it, the cheaper and faster it gets. That knowledge and skill has value.
And, as others have mentioned, there is enormous value in knowing how to operate on a fairly lean tech stack. It makes it so much simpler to scale effectively while keeping costs down.
There are other factors of course, but in general, many people come up with a rate for their own free time (which is often higher than the actual rate they charge clients)
Many people working on products, both on their own and within a company, aren’t turning down other profitable work to optimize existing solutions.
If I watch a two hour movie instead of spending two hours saving $100/month, it doesn’t matter how much I value my time, no one is paying me $100/hour to watch a movie.
But your argument is nonsensical because a one-time investment of time (much less than four hours for me, but I've been doing this for 20 years) can save you several hundred dollars a month. AWS AuroraDB, for instance, (which I also know how to configure, by the way) has much higher latency than a hand-rolled instance and will cause bottlenecks throughout your code in a DB-driven application. If I hadn't experienced the difference firsthand, or had failed to profile my app's performance adequately, I might assume I need to solve the problem by spinning up more ec2 instances to distribute the load. I've had the misfortune of working with a company that had exactly that problem and knowing how to spin up a new DB server saved the company thousands of dollars a month and took considerably less than four hours. Transferring a 3TB database to a new server without downtime did take considerably longer, however, but I was being paid hourly anyway, and it was still a worthwhile investment for the company which saved considerably more than my fee.
Any tradesperson should know their tools. A programmer is no different and if you don't know how to use your tools because your "value your own time higher" then thank you: you're the guy who ends up getting me called in to fix things at a much higher hourly fee.
* `apt-get install postgres-server` would have worked fine for my needs on my VPS. oh but i need roles, so let's start an ansible playbook. maybe let's tweak some settings. fine, still all short, easy to do.
* ahhh i should probably have backups. how am i going to manage that storage, where is that going to live?
* then i introduce a new feature & my database is running slow. explain query helps, but i also could use some metrics for these boxes, so probably need to start thinking about prometheus & node-exporter, &c.
i am radically in favor of a) personally facing these challenges and b) open-sourcing the operational knowledge & tools for setting up AND OPERATING systems.
yet at the same time i also think spending $171/mo for a year is an exceedingly wonderful option to have on the table. running my own servers is, to me, a lifelong project, something i want to deeply invest in. there's plenty of ways to go about it that aren't so arduous (k8s+postgres-operator+rook+tbd monitoring+tbd directory-services), but that willingness to keep engaging, supporting, maintaining, scaling things can be a very serious concern that extends well past the time it takes to set a database up: it's an ongoing "giving a shit" burden even when (seemingly) working fine.
being willing and able to hack through is great, and i am all for the coallition of the willing who elect to march through, hopefully not getting bogged down along the way. but wow if you are trying to start a business, it sure is nice being able to pay someone to spin up, back up, monitor, scale some services for you.
i hope some day "we" are better at such things, systematically, i hope open source ops helps give us better paths to doing these kind of things easily, safely, observably, resilliently. we're not there yet. but wow, this challenge to me- how we move open source from an older "software" model to an online service model, that empowers people to set up online systems as easily as opening an editor, that's the challenge at the heart of open source today. it's one that needs a lot more effort, a lot more work, such that we have good ways to stand up & keep up a database server.
If you understand the basics of algorithmic time complexity (that's your Big-O notation) and profiling your code then you're ahead of 98% of other developers in practice. I'm constantly amazed at how many developers think adding more libraries, newer frameworks, or more layers of tooling will magically speed up their code because "it's so fast". If you actually time things you'll find out doing it the "slow" way is frequently an order of magnitude faster.
And it explains there's currently zero revenue.
So it seems fair to judge the situation there based off those pieces of information we've been given.
As I posted elsewhere, the OP's choice to run dual redundant green/blue capable instances and cloud hosted metabase might have good reasons, but right now those reasons are not "wanting a HA solution so revenue isn't impacted"...
Green/Blue is all about saving resources and costs, not keeping them around. You misread the cause here. It has nothing to do with deployment strategies.
EDIT: not sure why I'm downvoted, I'm just presenting my use case. They are multigigabyte images encoded with custom wavelet compression that are cut into tiles (think google earth), each user needs 5-10 tiles every second
If you're just serving them (eg no image manipulation), that sounds like there's a problem somewhere.
https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Images/...
Your webserver still has to spin off a thread for each request if you want to do substantial CPU work for each request, but rest assured you'll get all the requests at once from the browser. Not 6 at a time like in the dark ages
The RFC recommends at least 100 streams. See SETTINGS_MAX_CONCURRENT_STREAMS https://tools.ietf.org/html/rfc7540#section-6.5.2
In my mind, choosing a dual redundant prod platform so you can do blue/green deployments is _totally_ unnecessary for a 55k MAU site generating no revenue. Same with cloud hosted metabase. You could run that on your own hardware - a spare laptop or probably even a raspberry pi for effectively nothing.
On the other hand, a hobby project/side gig where you can demonstrate real world experience in those two things could _easily_ pay off in the first week of a new job it helps you land.
If it's _just_ that extra $100/month they're spending there, it seems difficult to justify. If it's commercial experience doing that which is helping him try and land a job paying 20k or more extra per year? That's totally money well spent, in my opinion...
This app could easily be run on the AWS free tier, although even in the paid tier he could probably be managing a lot more than this workload for under $40 / month. (That's for two servers - which as has been pointed-out is wholly unnecessary for an app of this size.) The price he's paying is presently listed at $95.
Maybe I read his conclusion differently, but he said "would be peanuts" about the cost and "The bigger issue is that on the revenue side there’s a big fat zero."
Seems to me he's acknowledging he's built a thing that requires generating revenue to support itself, but that he's neglected the revenue generation part of his project, rather than complaining too much about the price of running it?
I'm mostly agreeing with you (and tristanperry and TekMol), but I'm probably being more sympathetic and ascribing un-supported motivations for why he's happy enough building it this way and spending this much money to run it. (Probably because I've been there before myself, and sometimes that expensive hobby project has paid off, sometimes it hasn't. I've never spend so much I've seriously regretted any of my failures though...)
Step 1:
Copy Peter Levels stack:
https://www.nocsdegree.com/pieter-levels-learn-coding/
If he serves hundreds of thousands of monthly users and makes a million a month with a single VPS, you certainly won't have scaling issues when you start out or just have tens of thousands of users.
Step 2:
If you really scale beyond that, resist the urge to bloat your stack. Think long and hard about every piece you add to it. Really understand each piece you add to it. Don't fall into the trap of paid services. Don't fall into the trap of "best practices".
I wouldn't want to read (or write) that code, but thanks for the article.
A major subtopic of this thread is "different situations call for different setups."
Saying "I won't work with PHP" is like saying "Gluten will make me shit my brains out." It is not virtue signalling.
Its an aside, but the mentality of "let them figure it out" is a major issue in education. Foundational knowledge should be easy to acquire so I can worry about higher level thinking issues. Literally spending hours trying to figure out how to set things up through hours of Googling doesn't really help that, nor does it promote the "figuring it out" people think it does - its just stumbling upon the right set of commands that let me move past this particular hurdle.
From the devops perspective, what about telling someone how to set up their own server to do/minimize X is so taxing?
- cloudflare free tier for caching, DNS, page rules, etc
- run everything on one VPS(digital ocean, linode, etc) pick cheapest that has specs you need
- any non-trivial storage (media, big files) move to Backblaze B2 it's cheap (you can use free tier cloudflare workers to redirect to B2 for free bandwidth due to Bandwitdth Alliance)
- free static page from Netifly (I can redirect to this with cloudflare in case my VPS falls over or something to provide info/links)
- If I want to look at logs or something I rsync it my local machine (if I cared I could set up a process to push logs/backup etc to private B2 bucket)
You may not need exact same setup, I am optimizing for caching and cheap storage because my site stores/serves lots of media files.
That's basically all of software development for your entire career. Never not had a day or a week not like that.
Here are examples of what I mean:
- An undergraduate Networking/Security course may not provide practice on appropriately salting passwords. It is merely discussed as part of some larger conceptual model. Students are browbeaten in earlier courses to not simply copy/paste code they find on the internet
- Debugging practice has to come from the student's own generated code, but if they made a mistake, they already are showing they do not fully grasp the material. There have been efforts to explicitly train debugging [1] but they are still in early stages of researching their benefits.
[1] The Code Mangler - https://dl.acm.org/doi/pdf/10.1145/3017680.3017704
Honestly, spending $200/mo is insignificant to me. And I'm pretty happy to answer the question of "Why can this guy build this thing on a single VPS and you can't?" with "Well, because he's better than me".
I can be up in 15 mins on Heroku with a Rails+React web app. And in an hour have a thing. Or for a static no-login thing, faster with Netlify.
But it doesn't matter. Because I never made a product as nice as the one he made. If the outcome is I'm -$200/mo that is irrelevant to me. If the outcome is I'm +$50k/mo that is very relevant. So I'm going to optimize for how I can do the latter.
Pre-compute and cache to reduce your need for beefy servers.
When you can run something on your machine or in the cloud, choose your machine.
EDIT: It's less about saving money and more about not spending it.
EDIT2: Forgot to mention: use SQLite.
Every app developer is aware about SQLite.
During a single request there might be 50+ cache-lookups, each taking a round-trip to a remote redis server to fetch a single key at a time. Batching those up to a set/hash would have been more efficient, but the codebase had evolved in such a way as to make that difficult.
Instead of making 50+ redis fetches it turned out that just fetching all the stuff from the database was faster.
(There will be refactoring to batch up the key fetches, but for the moment there was a measurable increase in performance under current loads just by removing redis.)
My postgres process doesn't come close to using up enough resources to push me out of even the cheap VPS tiers and I don't have to worry about locking if there's a heavy write load.
Plus setting up a nightly back up of any SQL database, regardless of creed, is like a 10 line cron job.
I think the point here is, SQLite setup would provide satisfactory results at this scale.
If the choice is betweeen a PAAS database offering and SQLite, you can pick SQLite. If you have skills / are prepared for managing dbserver yourself, then do that.
It is a common misunderstanding amongst software engineers that infrastructure serves "performance". Most of the complexity comes from redundancy and analytics (, realtime especially).
the servers are oversized for the load we're currently seeing. The reason for that is that we tried to solve a production issue by increasing the server specs. It didn't solve the problem, and now we can't down-size the servers without re-provisioning them ️.
You should really learn to add new servers by provisioning them and remove the old servers. Your app seems a perfect fit for it for moving between servers.
The bandwidth usage is just way too high and video compression requires GPU compute.
If my dev uses a slow stack or doesn't optimize v1, maybe I spent $100/mo instead of $10/mo.
Hell, let's say I'm spending $500/mo.
My mixed rate is ~$75/hr for dev time, so a week of dev time is equivalent to 6 months of hosting.
If optimizing (or using a difficult stack) takes one dev an extra week, then I'd better save $3,000 of hosting.