Hacker News Reader

Comment by koliber | Hacker News Reader

I love this article as it shows how fast computers really are.

There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications."

One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that execute on a separate worker. These multiple processes often need to write at the same time. You could finagle it so that the main server is the only one writing, but you'd introduce architectural complexity.

So while from the pure scale perspective I agree with the author, if you take a wider perspective, it's best to go with a database. And sqlite is a very sane choice.

If you need scale, cache the most often accessed data in memory and you have the best of both worlds.

My winning combo is sqlite + in-memory cache.

View on HN

The one that gets me a lot, which is similar in practice to your point, is when I need server redundancy, even if one server is otherwise plenty for my task. As soon as I'm not running in one place, you need network data storage, and that kicks pretty hard in the direction of a network-accessible database. S3 works sometimes and the recent work on being able to atomically claim files has helped with some of the worst rough edges but it still doesn't take a lot to disqualify it, at least as the only store.

In short, once you need reliability, your complexity necessarily grows due to the redundancy and failover you need to introduce.

If your downtime does not cost much, you can host many things on a single tiny computer.

SQLite has become my new go-to when starting any project that needs a DB. The performance is very fast, and if anything is ever successful enough to outgrow SQLite, it wouldn't be that hard to switch it out for Postgres. Not having to maintain/backup/manage a separate database server is cheaper and easier.

Backups are super-simple as well.

I'm also a convert.

why sqlite over postgres?

You remove a bit of complexity. Sure Postgres is not hard to set up sn to connect to, but Sqlite is just opening a file. It being a file makes it also very easy to test or debug you application.

postgres is great and is also a good default choice. It needs a bit more setup than sqlite. Unless I need a capability that postgres provides, I go with sqlite. It just works.

It is simpler and removes failure points. You don’t need a separate database server process or network/socket connections. Everything happens in-process.

have you ever run automated tests on postgres? how long did they take?

If it's a server app, I almost always have to switch to Postgres eventually so now I start with Postgres.

If you look at those "When do you actually need a database?" constraints I think its missing consistency which prevents bugs and makes debugging easier.

When you combine all those a database is a better alternative for all but the simplest cases.

At some point you end up writing a lot of code to say something that is a single word on the db query.

Perhaps (besides simple things) if you have many millions of users and no money. Or if you need something a DB is truly bad at.

Seeing the Rust 1M benches were an amazing reminder as to how fast stuff really is.

The reality is that things will be blazing fast in any language if you save things by PK in HashMaps.

In the benchmark Rust is more than 50% faster than the runner up

Correct. Order-of-magnitude-wise, it's roughly the same as the alternatives.

In the context of writing a new service for a new company, you should not spend one second thinking about whether your technical choices will allow you to serve 100,000 requests per second, or 150,000 requests per second. If you are, you are focusing on the wrong thing. If you get to 1,000 requests per second with a real paying client base you already achieved more than most dream of.

On the other hand, if you are optimizing a mature distributed low-latency equity trading system that is consuming ten's of thousands of market data ticks per second, a 50% improvement in performance on a 20 machine cluster might turn into some real $$$ savings. But that's not what this article is about.

SQLite is famous for single writer how does that reconcile with you criticism of the article

Yeah, files as a database is fun, but I find you ultimately reinvent the wheel when sqlite is pretty battle tested, free and easier to get right or scale up.

I can't talk though because I actually find myself doing this a lot

Reply on news.ycombinator.com