Startup Mistakes: Choice of Datastore
stavros.io
stavros.io
DataStore is 1 that is too big to pick wrong.
> Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me.
Does it? Even a prototype involves some iteration, which can be derailed by the kind of missing-schema problems that the OP mentions. Throw in a few operational problems, and your prototype isn't really any faster than something with better features and reliability. What's better for that first few lines of code on your laptop might not even get you to a prototype.
Like, MONUMENTAL.
That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later.
Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages?
And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.
You need to validate and test first. Get customers ASAP. If that means writing shitty code and using a shitty database or not even a database at all, than that's great. You can always convert to a better system afterwards.
I do agree that the longer you wait, the harder it gets to rewrite code and convert to different systems, but the most important thing for any startup is to get their idea validated as soon as possible.
Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres.
But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.
Wouldn't NoSQL be better suited for this scenario? Genuinely curious.
NoSQL db should be used if you are doing something that they are designed for (eg: storing data that can’t have a schema).
NoSQL often rely on handling difference in the data model in the application layer, which can become messy and cumbersome if you switch a lot of requirements.
And the Relational model is fairly simple. You could learn anything you need in minutes and just remember in add a few index here and there. For the small amount of things you need to do, is incredible how many features a RDBMS give you for free (like transactions, some basic axioms, and a query engine).
----
I don't remember the original quote, but is alike:
"Novices worry for code, Experts focus on data structures".
Properly chose your data-structures, schemas and data-layout will have a huge net impact in your code. That is why a well modeled database will perform well and allow to easily code against it.
This is the hard part, and most novices like to "defer" it.
In my times, we start designing the app first on the DB layer, including the queries, reports, etc. Now most start with the front-end or back-end... Imaging the are focusing in the abstract logic (because some can't imagine the datastore is part of it!) and ignore or reject the concept of learn what are the datastore capabilities.
That is like ignoring the documentation about arrays, building a layer on top of it, rejecting the idea of use arrays as full, and then wondering why his code performs bad and re-creating, badly, what it already have.
---
For fast prototypes, sqlite (stored in RAM) could be good. In the early stages I erase the DB in each run of the code (after the initial DB design, tweaking it). I continue to do that as far as possible, and only worry about migrations and all that when start shipping to customers.
Also, not be afraid of build several copies of the tables - like experiments- (customerA, customerB, etc), use views and peek on your DB documentation.
The origin is Linus Torvalds on the Git mailing list. Here is a copy from LWN: https://lwn.net/Articles/193245/
The quote is ironic considering that Git uses a bespoke and not particularly well-designed key/value database, which has resulted in notorious usability problems in Git.
But the point is 1) you're unlikely to ever get there. 2) if you're lucky enough to get there, if you start with your data in a structured database and gradually e.g. move things to JSON blobs in your Postgres DB, then move the things that really, seriously needs it off into a more easily shardable NoSQL datastore, it's a far easier direction to go in than the opposite.
Recovering structure when you've tossed it out the window is a massive pain.
And even a lot of the time when you end up needings things like e.g. ElasticSearch for search or something like that, it may still be better to keep the canonical data store in Postgres and stream changes to a secondary store to scale reads. Or add caching.
I'm not suggesting there are no cases where NoSQL datastores can't be the right choice from the beginning. But if you don't have a very compelling reason to, it's probably not.
Obviously, you want a sane legacy for the future but unless you already have hit that sweet product-market fit spot it is likely that you will spend a non-trivial amount of time optimizing (prematurely) for something that's in fact orthogonal to what you actually need.
The trick, as usual, is to balance engineering and business depending on which stage your at.
On the other hand, you want people who are prepared to accept that decisions that are insane in a stable business can be very much rational in a business where the cost of cash to fund large engineering changes is dropping at a crazy rate if you're successful, and doesn't matter if you're not (because you'll never get to it).
Of course it's good to avoid completely pointless mistakes. But it often matters far more to deliver.
A bad tech decision in a problem you can fix later, even if it's an expensive problem[1]. Having no customers isn't a problem you can fix later because your startup will fail and you'll have to close it. Consequently, at the start, you should work on getting paying customers over everything else, including prevaricating about what database to use. Anything is better than not making a decision.
All that said, "just use Postgres" as many people here are suggesting is actually pretty good advice.
[1] Unless it's so bad fixing it will sink your startup, in which case you have to live with it. That does happen but it has to be really bad.
You might be able to answer one of the questions without an MVP but very unlikely to answer both without. So you need to ship something.
It would be clearly stupid to not consider your technology choices at all in the early stages. But the best decisions are usually the ones that allow you to determine the answers to both questions as soon as possible. Because if you aren’t able to eventually answer yes to both questions you will not build a sustainable business.
If you are providing value, people are paying you and you can find customers, the technology sins from the past can be addressed. It might be really hard. But at least you’ll be working on something that matters.
Spending a massive amount of time on getting it “just right” will most likely teach you that you’ve developed the perfect solution to a problem no one cares about.
Speaking from experience of being involved with two acquisitions of companies I started (on the selling side) I can tell you that the “technology stack” discussion is a tiny footnote to the discussions on revenue, sales, customer lifetime value, etc.
And this is the reason a RDBMS is totally better. You NOT need to "spend a massive amount of time". With this:
1- You get a totally proven and mature tech 2- With massive tolling support and documentation 3- With the capabilities to model even most NoSql structures. 4- With a query engine that will outperform most developers. And more flexible that most NoSql give 5- With enough scalability that you will ever need for your MVP and beyond 6- And with the ability to plug specialized storage like column stores, time stores and more without complicating your code duplicating efforts.
---
P.D: I have been part of several projects that drink the NoSql koolaid, and be part of the weeks-long coding efforts that could be summarized as:
"This could have been a one line sql code/or index ..."
My last one, rebuilding core aspects of a RDBMS and still building features... dedicating large part of dev time is really "Spending a massive amount of time on getting it “just right”"
Speaking specifically about NoSQL: I’ve been able to launch entire MVPs using DynamoDB, some Lambda functions and HTML / JavaScript in about the same amount of time it would have taken me to plan out, build and refactor a proper schema. And I usually learned through the MVPs that my idea was stupid and no one cared about what I built, much less about my choice of database.
Let's say the valuation increases 10-fold between the MVP and the next round. Any $1 in investment I can defer from the first round to the second round then costs shareholders a tenth as much dilution. So even if I end up spending far more money, I may come out ahead, as long as the choices won't hold us back until then.
This doesn't take into account that I might not even be able to raise enough for a more expensive solution, so I might not have a choice.
So a lot of the time the right choice is what costs the least amount right now even if you know it will cause costly re-engineering down the road if you're successful.
For a mature business the tradeoffs are different. If you know your growth rates will be modest, then getting it right from the start matters a lot more.
Postgres is far from a boring relational database. Don’t make the mistake of building a mvp on rickety stilts and then swap them out later for proper columns while trying to run.
But I always wonder exactly "what" is easier with mongo or another NoSQL db. Just get a relation db like PostgreSQL, MySQL or SQLite for that matter. If your site starts to get performance problems go to the pub, celebrate a bit of success and then optimise!
The worst code I worked with was at travel agency startup and they are REALLY successful now. It was REALLY REALLY bad and hard to maintain. The owner of the business just said: well, bugs happen. But when a customer (not too many of course) encounters one, they probably call the helpdesk. Sounds stupid, worked great, business wise.
4 years later, I am sure they still have problems with the crappy code, but the business is fine!
There is something to be said for being able to rapidly prototype ideas, especially if your primary skillset is jockeying JSON in the context of web/app development. However, deciding whether or not NoSQL is the right fit for you or your project/business depends on how much of your time you will be spending getting down and dirty with the database.
Blindly picking a SQL DB (mysql/postgres etc.) is quite expensive from the get-go (a production ready mysql/postgres would cost ~30$/m).
Mongo costs ~$10/m (MongoDB Atlas), Google's Datastore is Pay as you Go (so your initial cost is close to $0 till you get paying customers), AWS's DynamoDB is similarly priced as well.
Sure, sadly all those noSQL solutions get really expensive as your usage goes up to normal non-webscale proportions, but at that point you have the $ to invest in a SQL solution.
The above was mentioned with bootstrapped startups/services in mind. Not your usual million funded valley companies.
For backing up postgres, all you have to do is setup a cron job that backs up the postgres' data directory to S3/Google Drive/Dropbox every hour/day.
If you want proper replication and failover then you can probably use 2 digital ocean droplets each for $5/month and another $5 VM for the application server itself.
Since it's free below 10K rows and only $9 for 10M. And, the dataclips feature always comes in handy.
I think one thing that's not mentioned enough in the SQL vs NoSQL debate is the benefit of powerful storage types. For example, when storing IP addresses in Postgres, you can use the inet datatype and easily query results if they fall within a given cidr range. Example:
SELECT * FROM audits WHERE ip_address << '10.0.0.0/20'
Why not just directly `COPY` your log files to something like Redshift?
The "important" events for data science are still stored on postgres (i.e. logins for checking if a login was malicious)
Yes, I agree that an key-value store that lets you do embarrassingly parallel reads and writes is useful for scaling, but has't it been said enough that You Are Not Google[1]?
[1] https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
They didn't chose NoSQL, they were forced to. I'm fairly convinced they started with relational stores. If a company or product grows to a point where relational data doesn't work, that's a problem you want to have.
The mistake is either thinking you need to design for facebook scale from the beginning OR thinknig that you can cut time in a startup by not having to bother with those pesky schemas that just slow you down.
If anyone remembers the most prominent and convincing articles on this topic, I would love to share that with my team
The application is locked into Google, but that hasn't proven a problem yet and can be designed around if need be.
I suspect the difference between us is that you spent the time working around these problems, whereas I expected it to just work.
I use both MySQL and MongoDB in my daily work on a classifieds site that does a few hundred million pageviews a month. Both are pretty solid performers. The article is correct that with Mongo you just move the schema into the code (new versions not withstanding). I think it's nicer to have the schema on the database side but it's really just user preference. We typically end up creating a schema class and defining it up-front anyway. There is also a small subset of cases where not having a schema at all is actually a benefit.
Starting with a popular SQL engine is a really good tried and tested method though.
When your schema is "in the code", even if you completely abstract it into a library, it means that the quick-one-off Ruby/Perl/C#/Python script will not have the integrity checks, and may corrupt your DB.
1) Use CouchDB and Postgres?
2) Somehow implement revisions and Postgres?
3) Use Postgres anyway and scrap the offline first?
Couch is a very good datastore for that use case, though, so I would definitely use it in some capacity for your purpose.
I don't see how that would work reliably. On the client would you source CouchDB or Postgres? Presumably you'd access CouchDB directly, but then why even use Postgres (for that subset of data, anyway).
If it's data you're going to want to run analytics on and sync to the client, you're probably going to have to store it in both places, I think.
If either of those are true (gotta be honest with yourself when deciding) then STOP. You have to think about the best tool for the job overall - sometimes that will be NoSQL, sometimes it will be an RDBMS. If you can't decide which of the two, then Postgres with its JSON support is IMO the best starting point.
—
Meta-comment: Feel like if the poster is self-identifying as the author when posting the link, it’s verified via say email/domain, an HN username has been ID’d in the past as the author, etc. — it should be automatically obvious in post, comments, etc.
Basically, "Postgres is a better default".
While I'm generally sympathetic to your post, there are some things that are red flags.
>If you have ten services accessing the same database and sharing data between themselves
Whoever access the database schema owns it. If you have ten systems accessing your database then ten teams own it. And if ten teams own it then nobody owns it. Nobody can change it. Seen this at a successful start-up that got big, but then couldn't rev order management, because every team had their finger in the pie, we couldn't do a schema update without breaking everyone, and of course one team was "under tremendous pressure to hit a major milestone and we just can't do that now" for over a year.
Now I would turn this around into a win for RDBMS by suggesting the use of functions or stored procedures: with an RDBMS we can construct an API, and then we can version those APIs. And then the team that owns the database can do what they like. That said, we can do the same with NoSQL databases by not allowing other teams to access them. The team that owns the NoSQL database is required to maintain an API for it.
I've only ever had nightmares with other teams coding against my schema. GraphQL worries me in that respect and I'd love to hear how people here have fared with long lasting GraphQL, in the real world.
>Django, for example, makes migrations trivial, as you just change your application-level classes and the database gets migrated automatically.
I've had automated migration systems grind to a halt and leave the DB fucked too.
>Priscilla used a graph database when her data was relational. Her husband, furious, filed for divorce.
There are no schema that are "relational" but not "a graph". However there are plenty of schema where a graph database is a natural fit but that require either one-table-per-node-type or building a graph model on top of your RDBMS (e.g. an Entity-Attribute schema). Oy.
There are also many schema where there is only one entity type, but every join is against itself, and we're looking to join all the way out to the clique. In this case would you suggest an RDBMS, and then put the iteration in the application? You suggested earlier that making up for the inadequacies of the datastore in the application is a bad idea.
I've got a graph application that uses NoSQL and it was the right call. An RDBMS would have allowed us to write something that worked for simple cases, but that would have brought the system to its knees based on some customer usage. The solution for the RDBMS would be the same as what we had to do for NoSQL. But up to that moment, the NoSQL allowed us to iterate far faster than an RDBMS.
>For example, if you later need to compile a list of all the brands of all the products on your store, an RDBMS can easily do that by reading the “brands” table
Only if you built a brands table. You can't argue that we can't predict the future, so use an RDBS because its easy to change, but then make arguments that require that the builder accurately predicted the future and built the schema with that foresight. Sure, we could go and pull a brands table out of the existing tables but thats work, and it might be work on a live database that brings it down.
A graph database would be just as likely to have 10 brand nodes since the overhead of creating the first such node is far lower than creating an entire table and updating the schema.
>Relational databases excel at easily providing answers to questions that weren’t predicted at the time when the data model was designed.
Or a NoSQL database with spark. "But spark is something new to learn"
And this is the biggest flaw in your argument. You're pro RDBMS because you know SQL and how to run RDBMS, create the schema, and write the queries. It is incredibly easy to get started with MongoDB. That right there is why it is popular. Not because its good. But when you say "Just get something started and use an RDBMS", you're actually saying "I know you know javascript, but I need you to learn Modula-2 for this part of the system". Fundamentally different syntax and strict types (or "schema").
>For all its unparallelizability, ACID is pretty damn nice.
Until someone holds transactions open across network calls and kills throughput. I've seen that issue lose a company a multi-million dollar contract because of contention on a single row. Or until someone chooses the wrong isolation level ("But I used a transaction!") and two transactions happily decrement non-atomically. This shit be hard, and part of the "hard" is not knowing you're doing it wrong.
>You’ll have plenty of time figuring out what to use when you know your exact usage patterns, if the business manages to not die until then.
So start with something quick and easy then. You've already managed to describe two scenarios where an RDBMS blew up in practice (multiple teams hitting a DB causing schema lock; its a network, not a flat hierarchy).
If you have an experienced SQL team, by all means go with RDBMS. But lets not pretend they are a panacea. Honestly, I'd use a graph database pretty much all the time - if I could only trust them. But its the quality, reliability and longevity that I have a problem with, not with the nature of how the database organizes data.
SQLite is also not too difficult to switch to a more advanced SQL in the future.
Let's say "don't use it for production for a multiuser server app" then I agree. But that isn't really a supported scenario at all.
The cost is that access through network e.g. NFS or SMB in wal mode is impossible (but you shouldn't have done that anyway), and that you can't just ship the sqlite file - ship a dump/backup instead, or you'll have to do recovery on the wal file you sent.
Of course, it's a good idea to use pgsql from the get-go; but SQLite deserves more credit than it gets, and is much more capable than it is usually assumed to be.
There are only 7 billion people, most of whom don't need your database updated faster than their ping time...
seems like a non-issue.
Can't you just idle in a loop waiting for the lock, perhaps waiting a random number of milliseconds (200-800)?
(So that rather than crash, the client just "hangs" waiting for the lock, in a busy loop.)
It just seems like this should not be an issue.
I'm just trying to understand your advice and learn from your experience, I am just confused by this followup.
In theory, SQLite works quite well with concurrent accesses, I have just never tried it because the libraries in the languages I've used didn't work well for that.
Though unless you did a truly comprehensive shootout it might be fairer to write: "Be careful using sqlite in production, the client libraries I tried in Python did not handle concurrent reads/writes, whereas clients for postgresql handle it just fine."
Anyway, thanks for the clarification!
> I hope I have helped you choose PostgreSQL for your next business
> .io domain
oh my.