Hacker News Reader

Comment by hughrr | Hacker News Reader

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider.

The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.

If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.

View on HN

I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good".

Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).

Very good points there.

I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...

I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency?

I don't feel that's what you mean, and it's probably my fault for misreading your comment.

I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.

Yeah, I probably could have phrased that better. The sibling comment is correct, I meant that if your idea requires abandoning referential integrity or data consistency, it's probably not a good idea. Since it's the database that actually enforces those constraints, it may seem that the database causes the problem. But in most cases, the problem is the data model or the idea itself.

I see.

I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"

I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern environments is not realizing that in their particular cloud instance they are explicitly or implicitly using the cloud vendor's network storage and they are entirely hamstrung on that. The other is not configuring the database to use all the RAM- but those are usually covered by the existing tuning guides.

The next from a design perspective is to leave natural places to shard/split the data up so you could represent the data consistently on multiple systems. Sometimes it's pretty bonehead things like your database tables related to login are affecting your critical path transactions and/or also tied to the referential integrity of your application data in ways that are hard to split apart. It goes back to the IO thing where if you are reading and writing login or UI traffic you sure as heck are not writing some other critical time sensitive transaction especially if it's to the same disk. The other obvious thing is to be trying to run analytic queries (aka look at many records) on a database tuned for mass INSERT/write traffic and having no plan to do that with a different resource.

If you are still worried about it I would suggest just prototyping it using a currently modern API/RPC server in front of whatever datastore you come up with. As systems have gotten more distributed the straight line speed of connecting to the database directly makes less and less sense in my opinion and having an API server you control allows you to be clever later in ways that might not be obvious today (especially keeping application-aware counters so that you can see what your application is actually doing as query-style logs for databases are extremely expensive in performance). Either way if you sit down and consider the performance characteristics of the underlying system, think about each class of transactions are you considering from your application in a write/read context (hopefully from measured data), and using tools like EXPLAIN you'll quickly start finding what pieces of the puzzle might need to move to start scaling.

The answer also might just be some parts of your application would be better served something other than a database like a message queue, a more straightforward ISAM or key-value store, or depending on your situation just flat files written onto the fastest local storage you can get to then be later transformed and loaded into a database sometime later.

Thanks for giving me some thoughts to turn over in my head. I have to admit, having been able to support most of my projects on a simple LAMP stack, I may simply be overly worried about scaling. But my hope is always to have a sideproject skyrocket.

Did you read the article a little while back about Discord's challenges with storing tons of messages and how they upgraded their tech stack? Did it make sense to you?

The discord post made sense because they understood what their workload was and did the type of 'classifying' queries that I previously described. They also decided to move off Mongo when they were hitting the 'real limits' of the underlying system from not being able to fit critical things in RAM. So they had a solid understanding of what was going on and they moved to rectify it. I don't want to presume to know anything about your sideproject but there's always going to be an limit you hit in any system, and you just need to build monitoring to understand when you are approaching that cliff.

They also had some clear characteristics of their system for their critical query that they could take advantage of: a new message is written to generally once, the messages table scales linearly, and updates and deletes are relatively rare. Using those things they were able to seek out a datastore that helped maximize the thing that's happening the most which is the write, and they were able to engineer a system accordingly. They also know that the messages would grow linearly without bound at a variable rate, so they could plan for that capacity. To me, that makes a lot of sense.

Especially if it is a side project you can't over-engineer capacity but you can do at least the measured math of what actions are actually happening in your system and what are your expectations for those queries. I would suggest that often times you'd want to focus on your critical queries that if slowed down would have the biggest net negative, but you'd have to find those. Every user action in a system should be attributable to some combination of reads or writes that can be reasoned about to try and figure out where there might be problems of contention, linear growth, exponential growth or growth up to a limit for a table, or something else. Either way, once you start classifying some of the critical paths in your system often either an answer or better questions start to come to the top. Without some of this information you can't back of the envelope whether or not your system can take a perceived load.

I'd echo the sibling comment, but also add that thousands a month isn't that much if you have a profitable idea that is internet scale. Thousands a month is the cost of one software engineer.

> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.

This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because:

1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus)

2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc)

3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability.

Life would be a lot easier if you could say something like:

* I want a Postgres node.

* I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key.

* My application has some logic that easily knows which DB to read/write from depending on shard key.

* There's some small amount of "global application data" that might need to live on a single node.

For your "life would be easier" version, isn't that pretty much what Citus does? And Citus is open source so paying for the managed version is optional.

saying "Citus is open source" doesn't mean its as simple as running `yarn install citus; yarn run citus`. Its is much much more complicated than typical open source. You pay for support for a very good reason.

> The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus)

There’s YugabyteDB.

Sharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers.

It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B.

You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases).

There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.

On paper I’m not dealing with the same problem as the people in this thread, but I can’t help thinking that the root causes are the same.

I have a team that won’t even look at the fact that we measure performance by response time, but we are lumping together write traffic and two classes of read traffic into a single monolithic chunk of code. We are SaaS where our customers write and their customers read. And of course search queries are by far the slowest traffic.

But there’s some sort of mental block about splitting things up that is only slowly changing, and our app is so “flexible” that a load balancer would struggle to tell which urls involve search functionality.

I think people just want to “know where to look” but if your code is on a cluster there already is no “there”. You’re probably looking at some log aggregator anyway, so who gives a tinker’s damn if they come from separate log files on the same box or the same log file?

I just automatically silo by customer because I am so scarred by so much shitty SQL written by ??? that it just make sense for privacy reasons alone.

Row based permissions are not great imo, either performance is crap or you have some weird bug.

Totally agree there as well.

Reply on news.ycombinator.com