Hacker News Reader

Comment by onion2k | Hacker News Reader

When someone says, hey forget postgresql and just use SQLite, and someone else says good luck if you hit n rows, and someone says it can easily handle n+m rows with x memory …

This is a good example of how devs get things wrong. The argument between which database to use because you might hit the row limit has zero benefit to the user. It's devs wanting to avoid future work to migrate from one database to another; a problem most apps will never actually see.

There's even an aphorism about this exact thing in dev: YAGNI https://en.m.wikipedia.org/wiki/You_aren%27t_gonna_need_it

Solve the problem you have, not thr problem you want.

View on HN

Maybe most apps never see the problem of hitting the row limit because the devs spend time beforehand thinking about which limitations they're likely to hit.

It's definitely possible, but the limit is 18,446,744,073,709,551,616 rows, or 281 terabytes, whichever the database hits first, so you probably could put the discussion off until year 2 or 3 if you really had to.

Kind of like maybe the 'Send your friends money, divide the restaurant bill easily!' app founders first think will change the world could get really popular if it gets on the frontpage of HN..

I have a feeling you will long before experience other problems that makes likely you have to migrate to a different storage system before you hit that limit.

Given it's ~2 billion rows for every person the planet I reckon so.

Other than Facebook who has 2 billion rows for every person the planet (tongue in cheek).

Telemetrics and sensors data could (at least theoretically) produce billions of row per hour per single sensor.

I work on such a system and one of the things we do is figure out which of that sensor data to throw away. In most cases the change between measures is too small to be worth saving, and because we have less data saved we can analyse it faster and thus do something useful with it. The purposes of all that data isn't to fill storage, it is to allow making useful decisions based on it. If you can't make a better decision based on sensor input then don't save it.

Long before you hit any row limit you will have performance problems because you have far too many rows to process in a reasonable amount of time. If you think about this problem and ensure you don't have too much data to process the row limit is so absurdly high you will never hit it.

Long time ago we ran several large insurance websites, fairly standard SQL Server backend.

Once day I got into work and a slightly panicky dev said "We need to change the primary key on all the tables"

It was using an int and at the current rate of data entry, was going to blow the maximum record number in a couple of days.

Must have been in the Jerassic period of MS SQL Server primary keys...

If you choose a 32-bit integer for your PK it can only address about ~4 billion records. That was true 1000 years ago, and will be true forever until a black hole swallows everything and changes the laws of reality.

Forget how many angels can dance on the head of a pin, the real question is can God change the value of pi, or can God make 32 bit integers larger than our current max? In both cases what it means to math is more Interesting than any yes/no.

you could take this anecdote as a proof for how rarely these things happen

Yeah, that's true, the one time in 25 years.

Choosing to do nothing is still choosing to do something. In choosing to do something, one makes an informed choice. How informed that is is up to you. Row limit may be one, or performance or feature set may be another - say, choosing Postgres for some data types and easier indexing.

Making informed choices are of benefit to the user. How much benefit depends on one's opportunity cost doing X instead of Y. Choosing not to do X is still a choice, an informed one or an uninformed one.

Or to step away from choice of database, take privacy. I see an immeasurable number of things on Show HN that take user data but don't have a privacy policy, if they have one it's often copied and pasted for somewhere to placate Mailchimp or another 3rd party service, and very rarely is it actionable.

Not launching a product is of course problematic for a user. Launching one that doesn't confirm to norms and rights a user expects will also be problematic. Should we not launch? How informed should your choice be? California State law, GDPR? How about Malaysia, or South Korea?

There are rabbit holes one can dive in to. There are reasonable lengths to dive, of which zero should not be a default case. At least have a look inside. Solving a problem isn't rushing something out full of uninformed choices.

There are reasonable lengths to dive, of which zero should not be a default case.

The fact you're even aware that there's an alternative to choose means you're already slightly informed[1]. Being a bit more informed so that you know a solution will actually work is fine. I am not suggesting you shouldn't look at alternatives. I'm saying you should look at alternatives from the perspective of the solving the problem you have rather than arguing about pointless trivia that won't ever impact you or the app you're building (like row limits).

[1] Every web dev who used nothing but MySQL for 15 years should be crying now.

> from the perspective of the solving the problem you have

Completely. A small planning goes a long way, and helps define what problem is being solved. This is a particular peeve I have with lauding of pivoting. The plan didn't work out and you're now doing something else? That's great. At least something was gained and you're looking forward. But don't design for it. In other industries this goes by the perhaps less jargony terms salvage or wasteful production. But putting salvageable and waste in the description of a product solution doesn't sound as nice, and nor should it.

Reply on news.ycombinator.com