Hacker News Reader

Yep. For things that aren't "big data" (nearly everything), I've started to use in-memory SQLite databases for analysis. Just loading your data into a table makes doing all the selections/aggregations pretty simple if you're familiar with SQL and it fits in RAM.

View on HN

Any particular reason you pick SQLite over more featured DB's such as MySQL or Postgres? I am assuming that they would be faster, with the only downside being more difficulty in the initial install/setup.

For a lot of "one off" tasks (like analyzing a big log file), just configuring a full blown DBMS like MySQL or Postgres would take longer than it would to write a script using an in memory SQLite DB and running the necessary queries.

I'm not sure MySQL and Postgres are faster enough to justify the extra setup work when everything fits in memory. My intuition is that they may even be slower, because they still have to write everything to disk, where SQLite can be memory only.

It's very easy to make a throwaway SQLite in-memory database for processing... not so much for a traditional RDBMS.

Reply on news.ycombinator.com