Hacker News Reader

Comment by pavlov | Hacker News Reader

I have a habit of not including a database at all when starting out a server app.

Instead I just use the language's native array and map types to store objects, and write simple functions to persist the data into a transaction log (usually a "newline-separated JSON objects" style file). On application startup, the objects are reconstructed by reading back the log file.

This approach can go surprisingly far. Memory is cheap, and not having a separate database behind a socket eliminates a whole class of potential issues and bottlenecks. The log file is easy to back up. If it grows too big, you can always compact it by dumping the in-memory structure.

(I guess I should add a disclaimer about "toy projects" and "serious production environments", because some people seem to get oddly upset about the notion of not having a "real" database for a web app...)

View on HN

You can get surprisingly far with that approach, but as a database developer, I should warn you that something that "works" is not the same as something that works under rare failure conditions. See SQLite's famous document[1] to get an idea of how things can go wrong. It's not nearly as simple as it might seem. [1] https://www.sqlite.org/howtocorrupt.html

Absolutely... But if I go with random database X because people are raving about it on HN, do I really have a better idea of how it works under rare failure conditions? Or how to fix those issues?

When it's my code and the persistence layer fits in a few screenfuls in a text editor, at least I know where to place the blame. Debugging database trouble is maybe the worst kind of work I can imagine.

I never recommend going with random database X, use something that's been around a while and has lots of users, then at least you know that you're statistically unlikely to experience a failure mode that nobody else has.

Rolling your own on disk persistence and recovery is like rolling your own crypto. What you don't know can definitely hurt you, you'd better be an expert, and even then you won't likely get it right the first time around. It's just a really hard problem involving complex interactions between your code, the OS, the filesystem, the disk drivers, and your disks all of which can cause data loss in very unexpected ways.

Your method can work, but it almost certainly has issues lurking that can cause data loss. You might be ok with that because it's a toy project, but it's obviously not a good idea for important data.

I also used this approach for many of my prototypes - persistence layer as a future feature. Then I found kdb+/q and realized that with the database built into the language, there was no need for separate infrastructure.

Here is an MMO in kdb+/q that fits on a single Vim screen: https://github.com/srpeck/kchess

I like approaches like this too. Largely because it tends to make you think about why you need consistency, more durability, better query abilities etc. for specific subsets of the data if/when it comes up, and what the specific requirements are.

When people throw everything into a database, in my experience the specific requirements for specific subsets of the data are often lost in the fog of history as people often don't document why they need to be in the database.

Then it turns out the database is used for everything from a short term cache which could just as well have been in memory, to logs which are of limited utility over long term and "just" need to be indexed for short term querying, to vital financial transactional data which needs to be kept for X number of years, to per-user data which is never, ever looked up other than when that specific user is logged in, and so would be ideal to shard, and so on.

Grabbing for the RDBMS right away is then the lazy way out where you can often get away with not seriously evaluating what is best, because it is "good enough". Then it's nice to start out with something that is sufficiently limited that you're forced to take it into consideration sooner rather than later.

I also tend to use cheap and easy solution to start personal projects and then migrate them to something else. I generally use sqlite as a database, so I can just copy the file directly, like there is only maybe two users on the server, it's not going to make a big difference.

I do this as well. This helps get the API right very easy on. When building services for my work, we've paired this with Swagger (using Compojure-API in Clojure) to get clickable API consoles. The end result is fast iteration on an api contract (between us and our consumers) and once we have bound that api contract, we can focus on rewriting the data persistence namespace however is best suited. Often this involves nothing more than adding a line to our DI system to get a database handle in our http request and then for any background worker threads (plus the obvious writing of db-specific routines).

I do this, but I just roll using initials so every time the app is reloaded its reset back to its original constraints.

This really helps with testing, because now you have all your initial dataset available without needing to write it after the fact, and the database connection wasn't always an assumed fact.

Reply on news.ycombinator.com