11 years of hosting a SaaS
ghiculescu.substack.com
ghiculescu.substack.com
Big agree here.
Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-dec...
This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".
I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.
Gitlab had a long downtime because the backup was huge and on the other side of the country. The backup server was on a low speed network.
https://www.arcserve.com/blog/lessons-learned-gitlabs-massiv...
How much money would you lose if you were down for one week? How many customer would you lose?
How much credibility would you lose?
For my peace of mind, I can't afford a spof when I know one lingering.
How long does it take to try it? A day?
Well then try it, either it'll work flawlessly on the first try, either you'll learn that the backup you have doesn't include logins, password and the security configuration that goes with it. Or that the dump you took lost some data because it wasn't in the right encoding.
Or the tape drive you're using need specific drivers that aren't available on the web anymore because the company website's closed.
... This is a work of fiction. Any similarity to actual events might be purely coincidental...
And more to the point, learning how to "handroll" Postgres could be beneficial. You could have learned about options for limiting the amount of memory, etc
Sure, managed is easier and use it when you can afford it easily. But before that, it's better to see how things are going (mem usage, disk usage, bottlenecks, etc)
- Install Debian 11 while booted in rescue mode.
- Setup the root file system encryption using cryptsetup and dropbear (to enter the key during the boot through SSH). Involves chroot and some fun commands.
- Setup ZFS encrypted mirror filesystem for the two additional SSDs.
- OpenSSH hardening and Teleport installation.
- Kubernetes installation (K3S)
- Connecting kubernetes to my Argo CD instance or an existing Kubernetes cluster.
And then through GitOps:
- Installation of openebs-zfspv
- Installation of kube-prometheus-stack helm chart
- Installation of (many) postgresql instances and other craps
I have been playing with Linux servers for 20 years and I find this fun and rewarding. But I do understand people saying that baremetal Hetzner is not for everyone. Especially if you start to have requirements such as "data must be encrypted at rest".
Not sure about Debian, but I believe Ubuntu Server will let you setup an mdadm mirror, LUKS (with LVM), and install and enable a Postgres server with a few buttons in the install wizard. It can even fetch SSH authorized keys from a Github account, covering by far the most important SSH hardening step (disabling passwords). Most hosting providers will also offer a one-click deploy that may similarly add your keys and do other common config
A better example of something that hosted databases makes a lot easier out of the box would be backup, replication, and monitoring
K3S is pretty lightweight and kubernetes is much more than cluster orchestration, so the pros win against the cons.
The fact that it's a supports multi-node means that you get all of the drawbacks of a multi-node system without any of the benefits. It's single node deployment but worse.
and you missing part about fault tolerance and fall back which is most complicated.
Kubernetes, "Argo CD", zero-trust, the sheer amount of "management" is off the chart.
"Installation of kube-prometheus-stack helm chart".. "Installation of openebs-zfspv"..
It's not postgres that's the problem here.
Many of the problems these tools solve are problems that wouldn't exist building things the old fashioned way. If you stick relatively close to the metal, operating this stuff is pretty easy.
However it's notable that a very valid reason to prefer managed services as a SaaS is to cover your ass if things go wrong. Your SLA violation is their SLA violation.
I can set up a new golang app on ECS with a load balancer and database, with a CI/CD pipeline, with 0 downtime updates in about 30 minutes. Most of that time is waiting for AWS to give me a load balancer. Our work applications have been running with this setup for over 2 years and the only thing Ivs done with infra in that time is adjusted instance sizes and bumped a MySql version.
I don't really need to set up a database or load balancer or anything like that because it already exists on the server. Just create a new database schema, new systemd service, new nginx rule.
RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB.
If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine.
But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.
Where possible go with simple but abstracted cloud storage, cloud tables and then a cloud db that is managed. We use Azure mostly right now but our storage system works across Azure storage, Amazon S3, Google Cloud and others. For tables, Azure Tables mainly. For database with filtering/paging better performant and ACID compliant cloud db, CosmosDB currently which is a dream with the differing apis (SQL, Mongo, Cassandra, Tables style). The more you can avoid vendor or dev-lockin the better so simple formats/messaging/routing and abstracted specifics/implementations.
When you store data in storage or a cloud db the scaling is "infinite" and you can also snapshot or backup to anther one, you never worry about data.
The front ends and APIs are mostly repos pushed to app/web services and everything else in data storage. Super simple and anywhere you need some special service that can be serverless or a dedicated setup, like maybe a RDBMS, chat server, network server or WebRTC/socket endpoint that interacts with the simple side. These managed as well if possible, though not always. Additionally, build cheap and horizontal scaling on web/real-time servers. Vertical scaling and sharding is for suckers.
Side note: CosmosDB is like a combination of NoSQL, document databases and GraphQL and it is ACID compliant and you can do REST or SQL, it can even wrap MongoDB and Cassandra and make them ACID compliant. It really feels like the best way. Not many have all that and ACID compliant. Not even Amazon Redshift has that, DynamoDB does if specified. Google Firestore does if specified. I used to be big on RDMBS Oracle then MSSQL then PostgreSQL and those are great for backing/reporting etc but CosmosDB combines all the power of RDMBS, NoSQL, document databases, and ACID compliant with little worry about scale. It is vendor lockin to Azure, which you can route around with platform abstraction, but currently can't be beat. As you got that clean API layer you could change later but best way is limited/clean and if possible, non breaking change API layers/signatures.
- Scaling is not infinite, it's up to 20gb per partition key (1), which can't be changed after document creation.
- One set of global indices, no equivalent to DynamoDB's secondary indices.
- Still can't run their docker container on mac (2) natively.
- Weird SQL-like dialect that's required for all but the simplest queries. JOINs are spectacularly awkward.
- Tooling is horrific. Based on the previous point, no existing tooling works for it (and nobody is building tooling for a DB with such minimal market share). For example, I needed to manually deleted 30 or so documents/rows yesterday - only way to achieve this is with 30 separate click-to-deletes in their UI.
- Minimises analytics options. There exist a plethora of business intelligence type tools that will happily sit on top of most common DBs. None of them like CosmosDB. So you're stuck with synapse link or whatever MS calls it now.
Overall it seems to combine the worst aspects of both RDS and document stores, with the worst aspects of both traditional and serverless infrastructure.
(1) https://learn.microsoft.com/en-us/azure/cosmos-db/partitioni... (2) https://learn.microsoft.com/en-us/azure/cosmos-db/local-emul...
Tooling could be improved and will be, it is fairly new still and the Azure Cosmos DB Emulator is not bad.
There is a CosmosDB Synapse setup that allows more analytics/intel on top like you said but same with other NoSQL, takes a bit to get worked in.
I actually like the flexibility of query types and that they include SQL as it makes it a bit more standard and somewhat less vendor lockin. You can use other types as well Mongo/Cassandra/Tables syntax. For filtering the SQL side isn't bad but most of what we do is flat/associative and not heavily normalized. For most of our data we are very cache heavy as well to reduce db hits and retries.
ACID compliance is huge and there are some design considerations.
What cloud DB do you use the most DynamoDB?
I work in pre-market fit startup though, so hyper-scaleability is not really a consideration right now, and would much prefer to be running postgres.
This is the total opposite of simplicity. Simple is PG and backup setup. DONE.
Maybe you mean "comfort"?
Other than that, it's not like queries are any simpler and the "simple" type system is, in my opinion, not a feature. I get that some might disagree with that.
Is there some other reason why you would prefer it?
Only one file to backup or deploy is the biggest advantage of sqlite IMO.
I use it in my production SaaS serving around 4 million requests per month on one of the lowest DigitalOcean tiers. The big ones for me were cost, operating simplicity and performance. I don’t need a separate process or server running which has saved me some money and time, and the app’s workload doesn’t need a ton of inserts so the speed is blazing fast.
There are 2,592,000 seconds in a month. So, 1.5 requests per second?
11 years ago you used Postgres because that’s what Heroku told you to use.
Those get quite expensive.
go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.
The most important lesson was about reducing the alerts and sleeping better. It's important to be able to leave your laptop at the office and not worry about things going wrong. This post gave me flashbacks of when I made websites for others and hosted them myself. Never again!
> We were doubling the size of our customer base every 9 months, and pretty soon this meant we needed more servers.
I've found that a moderately sized 4 core 'droplet' from Digital Ocean handles nearly anything I can throw at it (at least in terms of the web application world) - I see you product is doing employee management / scheduling - is there heavy compute / resources associated? Or is it more due to privacy requirements (i.e. you need a siloed instance / server per client?). Not trying to sound smug at all, just genuinely curious where the load comes from.
There is some heavy compute - most of the real value is in our pay calculations and compliance features which are all based on algorithms that calculate correct pay based on legal requirements.
But mostly it's just having lots of concurrent users. For a while a few droplets was plenty, and then one day we hit a tipping point where it was overwhelming.
It wouldn't surprise me if today's budget VPS does the work of several machines from 10/11 years ago, and costs less too!
We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two).
(1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.
Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
I began my career by scripting Asterisk using Python. I created a daemon using Twisted and that process ran for 3 years without fail (as in, the same Linux OS process was running for 3 years). This was on commodity hardware.
So, just make sure you take backups pretty regularly because you never know.
This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually.
They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.
I think that brings the ROI after 5 years compared with AWS to something like 3200%, less energy, if nothing goes wrong.
(I know that there are better options than running Windows Server on this box, but it seems to perform the same, it's far easier to administer by inexperienced devops, and in the long term it's actually around $1500/yr cheaper than running supported RHEL. Our other box is running supported RHEL but after testing Windows Server and having no issues we'll probably switch it for the cost savings).
plus more expensive network for cloud
Making decisions about infrastructure is as much about day-2 as it is about the initial deployment. I see fly.io and render in this camp. Shiny day-1 docs and demos and then wishing you good luck when you need a `rails c` to see something in the DB
Since that's generic enough to also see vercel as a heroku alternative, be it for node and not rails
We're considering using this https://bucardo.org/Bucardo/ based on this https://www.porter.run/blog/migrating-postgres-from-heroku-t...
Going to give it a shot when I have some free time.
Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop up.
- no wildcard subdomain ssl - poor metrics, nothing per instance - poor dyno granularity - (jumps from 2.5GB RAM to 14, no cpu/storage control) - no transparency on what each dyno actually is - external postgres replication disabled (deal breaker) - no first class postgres metrics and access logs/alerts (kibana recommended, but not great) - no external postgres backups (e.g. S3) - no deletion locks on dynos and add-ons, esp databases!!! - no warning when add-ons like databases are being deleted as a result of apps being deleted - deleting an add-on also irreversibly deletes all replicas and backups - painfully inconsistent naming for databases through their connection str env - dyno types for build processes use Perf-M? Not configurable - No lambda or github action style computing - No scheduled scaling - Unable to choose aws-us regions - Hard 30s timeout limit - Limited to 1 api key per user. No labels, configurable permissions, usage logging - no http/2 - frustrating enterprise offering. massive over-sell, near zero value
What are you looking for here? Larger build dynos? Heroku provides the build service for free so we use perf-m dynos to get fast builds with reasonable cost (for us).
This wasn’t *at all* their fault and there’s nothing dodgy about it. This was entirely your fault, and even mentioning this as a „partly“ negative for DO is imo very wrong. I imagine any other service provider, including Heroku, would have acted the same way. You can’t expect to mooch off a month of service without paying and without consequences. If you don’t pay your bills, service will be interrupted. It’s as simple as that.
DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.
DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for.
Email is the expected form of business communication in 2023. I don't think any of the hosts I use even have my physical address. Maybe they could SMS me. I doubt they have my phone number either though.
I would expect a billing warning notice when I SSH into the service though. Maybe that was in place for the 30 days before they shut the customer out. I certainly hope so.
I think that + allowing customers to add more than one credit card to an account are reasonable suggestions. Maybe they do that now - this was 7 years ago.
I'm not trying to shirk responsibility though. We didn't, for example, pass the buck by telling our customers our site was offline because of DO.
I can appreciate how terrible must have been to suddenly find all your servers shut down, and I understand how easy it is to set up billing and forget, especially with the day to day stress of running a startup, but what would you have done if you were DO?
> our site was offline because of DO.
Or maybe it's because of the bank that expired your credit card?
Not proactively updating expired cards is dodgy. Supplying an unmonitored email address as a contact address is dodgy. Cutting service after non-payment, email warnings, and grace period is normal and expected.
I would not expect Digital Ocean to alter the MOTD configuration of my Linux box because I forget to check my emails. That would be an invasion of privacy in my opinion.
What is this? Germany?
For larger businesses with thousands of dollars of revenue per month? Sure. For small $50 charges? No way.
Your example seems to be unrelated, although it sounds frustrating.
Fair enough, but that's why I was saying your experience seemed to not be relevant to the topic.
If you have an account manager or sales folk in the cloud org, IOW have a high spend, you get the twitter experience. If you don’t, and are a self-serve customer spending a pretty low amount like in this case and missed a payment and ignored the account emails reminding you to pay… what exactly is the provider supposed to do? I mean this in the nicest way possible but they aren’t so desperate for your business that they’re going to beat down your door so you can pay before you get shutdown. Important to keep in mind, if you don’t have a long payment history, you don’t look that different from a user that’s just trying to get something for free, and without paying for the communication channel, you also can resemble a company that just went bankrupt or something (which often creates a huge risk for the cloud provider if the cloud creds get sold by some disgruntled ex employee to crypto miners).
Source: I worked in this area at a major public cloud provider
Why not? If the phone companies here can extend that courtesy to normal consumers then I find it ridiculous to suggest that a cloud provider couldn't afford to do the same.
The using of an unmonitored email is the one that really gets to me.
The flipside of requiring a provider to submit paperwork to terminate your service, is the situation where in order to setup a new server with DO you'd have to file paperwork yourself. Can you imagine sending a paper letter every time you need to spin up a VM?
Which one do you prefer? paperwork in both directions or no paperwork?
Email is legal in all countries, as far as I'm aware.
I used to work for a SaaS firm. Every once in awhile the main office would receive a written and signed letters from (primarily) German users when they wanted to terminate their service. (We accepted that after confirmation using an authenticated email address, of course.)
But they aren't paying anymore!
Account management involves a lot of discussion and your account manager will keep the service running if they think the bill will eventually get paid.
It’s like saying car is a shitty transportation service because it stops working once the gas runs out.
Not paying attention the the reserve light, instrument msg and pings is solely your problem. More so because you are running a business…
Computers, on the other hand, are perfectly happy to use an expired timer to generate mail, a dashboard icon, or any other shout for attention that you please -- as long as you actually set it up. Paying attention to it is still a human problem.
I'd say you're dead right -- this is the only safe assumption. And any email client smart enough to check incoming email against a regexp could be extended to raise a not-easily-ignored alert if no pattern match (indicating success in processing a payment) had arrived by some chosen date of the month. Yet I've never come across such a feature in Gmail, Fastmail, nor Thunderbird. Maybe someone sometime has hacked such a thing into Gnus ...
Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire.
Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.
We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account.
That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.
Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration.
I mean: we're literally talking about credit cards often keeping the exact same owner name, the very same number and just having their MM/YY of expiration changed. It's not rocket science to update in a DB after an API call. And what's the catch on at least trying? If it works, you saved everybody time. If it doesn't work, it's not worse than your current "solution".
I understand politicians and lawyer-minded people saying: "technically it's not our responsibility" but they're wasting everybody's time, leaving money on the table and wasting money on support.
It's just poor judgment to react like that instead of thinking as to how life could be made better for everybody, starting with your paying customers (which, btw, are the reason you exist).
Are they at fault because their customer dropped the ball so badly about paying their bill? No.
To lend a bit of insight as someone who works in the payments industry: This functionality is generally not free.
If most of your customers come and update their expired cards already - or you offer a service which is so essential to your customers that they generally would freely expend the effort to do so if notified, it doesn't make any sense from a financial standpoint to pay the fees to subscribe to card issuer updates.
It's not as seamless as you're making it sound.
No need to use email at all.
Infrastructure is kind of a solved problem for common use cases today, just requires the expertise.
With our ducks in a row I'd next have look to a GRC function for the compliance bits whilst splitting the platform engineers time between embedding engagements and tooling investments.
You're on the right path man, I'd love to know what I know now back then but unfortunately time doesn't work like that.
This is the problem however for many (older) companies. They either don't care, or quite literally don't know the infrastructure solutions out there which can save literally thousands of hours per year of headache. Sure, for many companies with legacy systems they have a "dont fix what isn't broken" mindset, but from what I've seen, I always ask, if shipping and modifying new versions of a system takes hours or even days to complete, is the system really not 'broken'? I guess I never realized it, but having automated and clean infrastructure with tests and uptime metrics is a must-have for me on anything I build going forward. Take 2-3 weeks to save months of headache.
Couldn't be more accurate. Heroku and Rails is almost as much of a throwback as Node and Express. You just had to be there. And it was great. Web dev was always a hobby for me as a teen, but then I turned to the rails book as a means of learning a professionally designed system, when I wanted to get serious as a dev. It definitely served me well. Heroku, at the time, offered a very streamlined and accessible way to integrate rails. It was a great time to learn.
Here being Japan, there were a lot of opportunities to pick up ruby/rails along the way but I stuck on the JS trajectory partially because I didn't want to 'start over' with rails _now_. There were only a few times when not knowing ruby/rails meant I was limited to specific tasks so it was never really career limiting. It did mean that I actively avoided working in ruby shops, for better or worse.
If I had picked rails instead then, would I have transitioned into more of a backend engineer with some frontend duties instead of the reverse?
Not that it matters, but there definitely is a tendency in our industry to 'look down' on frontenders as not real engineers and thus not consider them for leadership positions.
I think that's funny because there some to be a ton of backenders that can't do frontend at all. And then they want to look down on FE when they can't do it themselves? It's not just basic HTML and CSS if you're building a complex app.
I do both (FE & BE) so... I've seen it all and enjoy it all. Not sure one is easier than the other.
And additionally to this during the time it became viable a lots of FE folks still had to continue battling IE6 in their daily lives so online documentation still lagged and had a clear smell of the cargo-culting. Heck even today you see people complaining about Javascript here on HN.
But being left behind today, you gotta blame yourself. Early realtime Google-suggestions using AJAX came already back around 2005 (?) and if you didn't take notice and still missed people were doing decent realtime games by 2010 you were doing your best to live under a rock.
This all seems silly.
I'm also disappointed that real frontend expertise is not valued as highly as a traditional backend engineer. Think about someone like an Architect. You could say that they are glorified artists (yes, I know they do more than that) and they are highly valued for their artistic input as well as their professional advice; but it's also not unheard of to have an architect solely provide artistic direction and a structural engineer provide support.
I agree and Node and Express are still great when going with just js. Node and Koa more now but still great. Socket.io for real-time. All are great for getting things up quickly, simple and shipping things.
> Eventually I realised that I wasn’t the problem.
I had the same experience. These free "Solution Architects" are a disgrace to the architecture profession. They are sales person and are paid/rated/motivated to sell you up. Not what is best for you. Ours went over the in-house architect with zero feedback and told our CTO/CIO how shitty the architecture is. Fun. Stay away from them.
I've always taken the advice of any prepaid 'experts' with a huge grain of salt, not just AWS
Did you want me to read it or click buttons for you? Too bad your UI gets in the way of the U.
I guess they do it because it works, and if the average person felt the same way, they wouldn't be doing it?
You might think this, but I tend to doubt it's true... people making these decisions end up being in their own little bubble, and not really have a good idea what people's actual response is.
That said, I find Substack's pop-up tolerable: I am after all getting someone's work for free, and I'd rather have a simple pop-up that I know I can get rid of than ads or other aggressive forms of pop-ups.
First the longer you stay in heroku ,the most complex is it to exit it the time you really need it and the less flexible you are in the time being.
Second, wish he had pay for a pit team sooner, but could this money better used investing in marketing or sales like he probably did ?
The guy has obviously succeeded as a business owner, would it still be the case if he had implemented these advice ? We will never know, but what we know for sure is not implementing these advice made him successful
"Made him successful" implies a causality that is a bit too strong.
Indeed, we will never know for sure.
Maybe implementing these advice would have impeded development of other critical areas of his business.
Maybe it would have would have helped make is business more successful as he would have had a more reliable product.
Or maybe the business impact would have been neutral, but would have resulted in better quality of life/less stress for him and his employees.
But in general, the way I read this article is: they made good decisions overall, but as everything in the world, it was not optimal (switching platform too early, making some big mistakes like the credit card one, etc).
It's a very interesting read nonetheless, with clear take away:
* chose boring tech you know and focus on your product, not the tech, specially in the early days
* grow your infrastructure and complexity with your product needs
* accept you will mess-up but properly learn from it, and grow your organizational knowledge, structure and processes accordingly.
Good luck on the next 11 years :)
As a counterpoint of sorts, here's my "hosting journey":
* run everything on a single physical server rented at Hetzner (DE)
* [... several years pass, business grows ...]
* switch to ansible, learn it, spend a week or so to write automation for a 3-server setup, also learn terraform and write terraform configs for setting up a Digital Ocean system from scratch
* run production on a 3-server setup at Hetzner DE, run a staging system at Hetzner FI, also serving as a possible quick manual failover, for a total of 6 physical servers, test re-initializing systems from scratch regularly, test setting up a Digital Ocean system from scratch regularly
* [... several years pass ...]
* that's it — I really can't see a need for more in the near future.
But then, my software is not in Ruby on Rails, and I have no experience with that platform. I use Clojure and ClojureScript and I was careful to design everything to be rather client-heavy at the start. I also never wanted to depend on PaaS systems, mostly to avoid lock-in, but also because I don't buy the "just use our magic database offering and forget about database problems forever" selling pitch. You can sweep possible problems and complexity under the rug and hide them, but you can't run away from them. I also do not use Postgres (collective gasp in the audience), because having a single centralized point of failure is not something I want in my setup.
I also never needed significant sysadmin/devops support. Granted, I do have some experience, but these things do not require dedicated teams, unless you are YouTube. A little ansible+terraform goes a long way, so does buying an hour of two of consulting from an experienced sysadmin.
Those physical servers that I use are significantly faster than the over-subscribed cloud VM instances that you usually get from AWS and the like. And they have 64GB of RAM, not some measly amounts. If I need more servers, it takes on the order of hours to get additional ones, but I'm not sure what I'd use them for.
My total hosting bill is on the order of 350€/month and is boringly predictable.
I'd agree Postgres is not the right answer to every problem, but the documentation of its failure modes and the mitigations thereof make it a "good" answer to "most" problems.
I'm curious what the "problem" (in access pattern terms, doesn't have to be business terms) and "solution" (i.e. persistence technology) is in your setup!
I am working on replacing it with FoundationDB. I want to have a fully distributed database with strict serializable semantics (see https://jepsen.io/consistency), and there is very little out there that gets the job done. FoundationDB is really impressive and works really well. I'm worried that it isn't "fashionable", though.
As for access patterns, I'm not sure if I understand the question, but I'll offer one thought: if you're writing an app, you don't need a "query language'. You'll quickly learn what your queries are, and the right approach is to restructure your data to fit your access patterns. Your "queries" will be written in your programming language of choice, not in the databases "query language".
I feel that the idea of a "query language" is stuck in our heads back from the days when the boss would come and tell you to produce a custom report from the database. It's just not how app databases are used these days.
Should have elaborated, by access patterns I was referring to proportion of readers to writers, the distribution of load over time, the distribution of transaction sizes, so on.
Although your discussion of query language is an interesting one – that the goal is essentially efficient (de-)serialization with "retrieval-from-other-process" costs that are minimal for your workload and subset of query space.
But honestly nothing these days is as good as Heroku was 10 years ago, if you just want to put something online and don't care how it gets there.
Possibly qovery.com as well.
I mean seriously, railway.app has just fifteen employees in total, and revenue in the single-digit millions.
Reminds me of when someone posted a "cloud storage vendors poster" with hundreds of vendors -- but not Azure -- some with revenues smaller than the cost of an Azure Storage Account one of my customers was using... to back up a single server. By accident.
Used it for a few clients but it's tough to keep using it with all the issues, so I'm looking at Render and other alternatives. But if Railway could smooth out the reliability / stability, I wouldn't have any reason to switch.
*I'm cofounder & CEO
Calling out D.O. for a fairly transparent process is very wrong. They did what they had to and OP was oblivious to a fairly standard process of "hey, your CC expired, let's do something about it before we turn off your servers" which OP seems to have ignored several times in a row.
All of the story felt like he was just stumbling from one hype to another, and from one easy to avoid mistake to another. Even when I had only 7 years of experience in total (now I have 21.5) I was paranoid enough to look for DB ID turn-over and absolutely would have made sure to upgrade the DB server when I start noticing it hits 80% load consistently.
But that's life for you. A technically excellent guy still lives paycheck to paycheck, meanwhile an absent-minded guy who is easily hyped has a successful business. [sighs deeply] ...Moving on.
Finally, choosing AWS but still only using EC2 is kinda non-intuitive for me; why do that at all? Maybe because they allow transparent upgrading that makes for less sysadmin work? If so then fair, but that's still like buying a Ferrari to drive on long empty roads but never go above 90 km/h.
> we’ve spent a bit of time learning more about moving off the cloud to a managed data center. But the nice thing is not feeling like we need to.
That SaaS's are moving/thinking/completed to running their own infustructure away from managed services like digital ocean and aws, like 37Signal(Basecamp & Hey).
This is a lot of switching around for a plain VPS.
What you want from day one is a completely managed hosting for tech stack you work with, but are not hard tied to forever.
Right now I am starting some new/modern meteor.js stuff again, there is meteor galaxy for hosting that "just works" for that stack, you bring your own mongodb (I prefer atlas itself, they even have serverless now), and everything is taken care of, including CI/CD/Monitoring/... . Its like a few minutes of initial configuration, and never think about it again, and if done correctly you shold have horizontal autoscale of some kind automatically nowadays.
Yes, this is significantly more expensive than directly using AWS (which they use under the hood), and even though I am personally highly trained in this stuff+terraform/cdk/..., I don't want to have all this work anymore when I also can shell out a few hundred bucks per month instead. Just in case Galaxy becomes a problem for some reason, I _still_ can deploy the app stack to some VPS provider, but I'd use some already existing automation (like meteor-up in this case) instead of really digging into typical devops topics for it.
There is a bad feeling in the mouth as an engineer to shell out "more than needed" for infrastructure, but my rule of thumb now is that I am happy to eat that frog as long as the potential cost saving is less than 2 infrastructure engineer FTEs, thats my trigger to _maybe_ discussing it.
At least one tradeoff is that you'd have to be physically in a location with adequate bandwidth / latency but for various use cases it might be very cost effective during some phase of the startup.
I wonder also how the AI induced focus on specialized hardware might change the calculus
Kind of hard to believe
Side remark, that's what I was trying to tell the dgraph guy. If you have an int64 ID for ALL your transactions and all your IDs it will eventually overflow. It's a single incrementing value for ALL actions. Inset a new record, new ID, retrieve this record, new ID etc.
Everything this guy writes about are newbie mistakes, stemming from a lack of "far sight", aka thinking 5 steps ahead. Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these.
What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not. I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part.
I know this article is about Ruby. But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer, I would NOT go AWS lambda and complicate my life and, most importantly, the development process or as I call it the solution delivery process.
This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations.
There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters. The split between backend and frontend happened around 2013. Having a classic website with low performing Ruby, that also has such a bad workflow, e.g. Blog, you can't start with Comment, you have to start with Blog, then Post, then Comment. This is archaic. With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client. If I need SEO I'm back to the scaling problem, but the Go backend won't be the one that needs scaling. And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle. Doing this for the SSR JS frontend is where it gets expensive. So either try to static render with Svelte, if possible, to solve this problem and load balance or get locked into complex auto scaling cloud nonsense. And this is where it gets REALLY expensive.
Your Ruby on Rails app might be good enough for a PoC, but that's about it. If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure. And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent. You can have your cake and eat it too. Just think it through from the start to the end.
>No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond.
Sane defaults are important. If the DB defaults to a particular value it's likely because it's fine for most use cases. It's an excellent thing to have enough customers and utilization to require something more than int for an ID. That said if there is little to no performance difference, I'd argue that this should be made the default upstream rather than the default recommendation or "best practice."
>Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these.
Not every one is able to do that kind of systems level thinking across an entire product stack, and that's OK. What's more important is not building things you will never use, or might use one day but don't need today.
>I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part.
I would spend more time thinking about how to market to folks who want to reduce their cloud investment. My current org is way over engineered both in infra and engineering in general. We have been working to reduce complexity and cost and it's had real big wins for us as an org. KIS is an excellent philosophy when applied pragmatically.
>I know this article is about Ruby.
The author could have replaced ruby with "Language-X" and the article would have been just as accurate. It's often not the language that's slow, more often it's the DB schema, data models, and business logic.
> But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer
If all it has to do is reply "Hello World" plenty of languages and frameworks could hit similar numbers. But if all those frameworks are blocked by a de-optimized external request, it doesn't matter. Here, the authors app was def having DB scaling issues, this could happen with any language, runtime, or framework. It will happen sooner with certain combinations than others yes, but utilization and customer feedback are more important than early optimization.
>This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations.
I think that's a harsh assessment. We all buy into some hype, and we all generally try to make the best decision with the information we have at the time. Instead work backwards, put yourself in the authors shoes, assume the idea at the time made sense, and then ask "What had to be true, or seem true at the time, to reach the conclusion they did?"
>There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters.
It matters, until it doesn't. https://www.tiobe.com/tiobe-index/ as an example, we see top spot held by a known "slow" language (certainly comparable to Ruby). JavaScript, PHP, and VB all beat out Go in utilization. Performance matters, is a statement that requires context. Sometime the "performance" of your developers matters more than your runtime.
>With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client.
And now the current trend is to start to reduce the amount of load on the client because turns out 2Mb of JS to render an SPA is not the best XP on a lot of devices and networks. It's important to understand who your users are, and the operating context they bring.
>And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle.
Horizontal scaling isn't dependent on language. You need to design your API and choose your persistent storage correctly to make this task easy. But in reality, most folks could just use caching and be fine.
>Your Ruby on Rails app might be good enough for a PoC, but that's about it.
Based on what, your personal opinion? It's a perfectly fine language for a back-end service and I say that as someone who has no interest in learning Ruby.
>If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure.
Maybe. We use scala at work, because "it can handle large volumes of streaming data" and yet it's some of the slowest parts of our stack because it's hard for folks to "do right." Sometimes you need a language who's feature is easy to use and hire for.
>And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent.
Yes and no. K8s is basically the next generation of LAMP for the cloud. It's fine, just understand what you are investing in and what it isn't. We use aws at work w/o k8s and it's a complex nightmare to manage in some respects. My k8s cluster at home however is fantastic and fun to tinker on, and causes me less stress with more uptime than my AWS clusters at work. I think it's more important to understand the problem space, and how a tool might fit in or not, than to hope it just solves problems magically.
The cloud isn't magic, it's just someone else's computers.
Perhaps its more to do with your attitude?
This comment comes across as very condescending and needlessly aggressive to someone admitting their mistakes made whilst growing their successful business and graciously sharing them so others can learn or commiserate on similar mistakes.
- scaling to zero for low use services
- reliably managing other infrastructure, like pools of ec2 or ephemeral ec2 spot
- getting 1000 cpu cores for 5 seconds to do latency sensitive data heavy lifting
the elephant in the room with cloud is always egress bandwidth billing. egress heavy apps just shouldn’t live on aws.
for me, the fun with aws is not figuring out how to make cloud lasagna and then write thought leader blogs about how tasty it is, but in understanding which parts of aws are actually better than alternatives, and how to compose them into good systems.
s3, dynamo, ec2, ec2 spot, lambda, r53. egress bandwidth aside, these things are for great good.
add in cf workers+r2 for egress bandwidth heavy components.