Comment by darkxanthos | Hacker News Reader

Comment by darkxanthos | Hacker News Reader

The subtext of the largely upvoted answer is that databases > text files. I may be preaching to the choir BUT:

"1. You can query data in a database (ask it questions)." If you use your file system as a "database" you can ask it questions too with file paths

"2. You can look up data from a database relatively rapidly." Given that lines of text can match some problems quite well, a text file can be MUCH faster.

"3. You can relate data from two different tables together using JOINs." Denormalization makes this unnecessary.

"4. You can create meaningful reports from data in a database." I work as a data analyst and make meaningful reports from event logs all the time. To be fair, I do prefer them being in a DB usually. ;)

"5. Your data has a built-in structure to it." Just because it has a structure doesn't mean an RDBMS is the right place for it to be codified. Also let's be clear the poster of this answer is clearly referring to an RDBMS.

"6. Information of a given type is always stored only once." It's important to note, this only matters if your data is not immutable. There are many applications where this is the case, but also many where it is not.

"7. Databases are ACID." Meh. This matters but after the advent of NoSQL era I now realize this matters much less.

"8. Databases are fault-tolerant." There are tradeoffs made in this name as well. Sometimes you can do better by letting your app be fault tolerant of your data instead.

"9. Databases can handle very large data sets." Files on disk can't? No I think here a valid point would be a database can span multiple disks and make them appear cohesive. That's a very nice bonus. For 95% (worst case) of the world though, that's not a concern.

"10. Databases are concurrent; multiple users can use them at the same time without corrupting the data." Again there are tradeoffs made in the name of this in an RDBMS (like locking). Reading immutable data scales across multiple users just fine.

"11. Databases scale well." CDNs have shown that files can scale even better. Again immutability is a constraint but I think it's an interesting constraint to explore for many systems.

Just to be clear I'm not saying databases are usually the wrong choice. Not at all. Instead they're just all too often the wrong first choice. Start with files, iterate from there.

You provide a nice counter-argument in general, but let me nitpick a single concern.

After the advent of NoSQL, ACID matters not much less, but in fact a lot more. Startups lose data all around the world because their shiny new database doesn't support the basic principles of databases.

Please stop peddling this rubbish.

Startups have been losing data all around the world with SQL databases as well e.g. Dribbble.

The implementation of ACID is just as important as the concept itself.

Okay, so ACID all the way, then! No concept or implementation can ever save your data from corruption or disk failure, which is why you need all the guarantees you can get in all layers.

Some thoughts:

"If you use your file system as a "database" you can ask it questions too with file paths"

You'll be much more limited in the kind of queries you can reasonably make. You'll have to decide on them ahead of time. This might be what you want, but it might not.

"Denormalization makes this unnecessary."

Denormalization means making your data match your application rather than the other way around. You can make stuff faster this way sometimes, but again, you're limiting the kinds of queries and access patterns you can make in the future.

"It's important to note, this only matters if your data is not immutable. There are many applications where this is the case, but also many where it is not."

Wouldn't a flat file which continually accrued immutable data technically be called a log? You can do analysis on flat log files, but it will get slow if your files are sufficiently large. Indexing can help here.

"CDNs have shown that files can scale even better. Again immutability is a constraint but I think it's an interesting constraint to explore for many systems."

What? CDNs improve performance by placing the point of origin for data physically closer to the client, thus reducing lag. What does this have to do with the difference between files and databases? Do you mean that static or cached webpages serve faster and use fewer resources than ones generated from sql queries at the time of request? Duh, but that doesn't really contribute to the discussion.

"Duh, but that doesn't really contribute to the discussion."

The idea that storing often requested data in something other than a database and that being a simpler solution than tuning the database was a revelation to me at the time when I learned this. The overarching point is a database is not a panacea for data storage.

> Start with files, iterate from there.

I agree so much with this. I run a small service that looks like it would be a perfect use-case for a database. Even though all the data is beautifully normalized and instantly searchable, using a database directly would be impossible. Instead it starts as a series of flat text files. From these flat text files, the database is automatically generated. Three big gains here:

Version control on the text files works perfectly as expected.

Creating ad-hoc and one-off columns (while figuring out how to best normalize unfamiliar data) is trivial and painless.

Comments! It is so easy to write in-line notes. This alone is the killer feature of text files. Normalizing and curating data requires making a lot of choices and comments let me keep track of these.

Of course there are downsides. You must edit the text file to add data. (With some exceptions, such as updating live prices/inventory.) If you have equal numbers of writes and reads, this system is not for you. But if you have infrequent (daily) writes then it is no problem.

On #7: ACID is a concept that matters just as much now as it used to, but I think a lot of people that thought ACID was important realize that it might not be so. I wouldn't say a RDBMS is uniformly better than text files, but in some applications, I wouldn't have it any other way.

When dealing with crucial data where robustness and correctness are paramount, an RDBMS may not be a bad fit. I cringe at the thought of an insurance claims system being done in NoSQL or JSON instead of a more robust system. Things are probably different in the startup world, but I'd rather take a huge hit to performance than a multi-million fine and audit for losing records from my DB.

Reply on news.ycombinator.com