Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by raverbashing | Hacker News Reader

Parent Full thread

raverbashing·

You would probably want some RAID + Backups, (and a very generous amount of RAM there) but this is ok.

In fact, I wouldn't doubt that for several "big data" users even SQLite would be enough.

View on HN

I migrated from MongoDB to SQLite for a project a few years ago, it was much more performant:

http://www.stavros.io/posts/using-sqlite-for-great-science-p...

I think the data size ended up being a GB or so.

I wouldn't doubt that either.

We do use RAID 6 (I think) + tape backups and the backup server itself is a hot standby. If you're curious, this is the chassis : http://www.supermicro.com/products/chassis/3U/836/SC836A-R12...

The motherboard is also Supermicro running dual AMD Opterons ( single processor on the backup server ) with 256Gb RAM.

From my experience SQLite is great when only one process with no parallelism needs to write to the DB and where no other process will frequently overlap reads with writes. However, for CPU-heavy computations that needs multiple cores to compute and write to the DB PostgreSQL has a much better fit for me even for fairly small data-sets. And the ease of use and setup of PostgreSQL is not that much more than SQLite.

As date guy that uses both, there is a higher level of expertise required to properly maintain PostgreSQL. SQLite is something that I can call instantly from Python without much forethought.

What is the price / performance tradeoff for using SSD in these applications nowadays?

Reply on news.ycombinator.com