Hacker News Reader

Comment by jsilence | Hacker News Reader

With the ubiquity and general awesomeness of sqlite I see no real reason usecase for these toolset over sqlite.

While I agree with the general awesomeness of sqlite, I worry about the blanket statement that you've claimed.

If you read the documentation 'Purpose' [1], it says, the issues with databases, like SQLite:

    - The stored data is not directly human readable.
    - The stored data is definitely not directly writable by humans.
    - They are program dependent.
    - They are not easily managed by version control systems.

I just want to say that I also love sqlite, but I cannot avoid worrying that when the asteroid strikes and the ocean men invade, I will not be able to program a sqlite driver on a homemade 8 bit computer.

[1] https://www.gnu.org/software/recutils/manual/Purpose.html#Pu...

Sure, text files are awesome, I get it and generally I agree. But sqlite is in every nook and cranny of our computer and mobile systems, it is in Android, it is in the browsers. I find it hard to imagine a postapocalyptic world where one would be able to read a file somehow, yet not be able to get access to some sqlite processing software.

I like the rectools for educational purposes, but beyond this for any however small purpose project I'd rather switch to sqlite pretty soon.

Human readability. I don't need any other software but a text editor in order to easily view/edit the database.

My ambivalence exactly. The user in likes plain text for its accessibility and flexibility, but the programmer in me dislike it for its gross inefficiency.

Does some sort of general data compiler/decompiler exist? Please don't say Binary XML.

There are so many use-cases where the compute inefficiency of this just doesn't matter but the human efficiency of being able to read and edit the files is a huge win.

Consider the gross inefficiency in the other direction. To simply edit done simple data, you have to have how much code and infrastructure? And how much of that is plagued by bitrot?

It is a "scaling" problem. At small scales, yes, human time can be more valuable than computer time. At large scales, humans will sacrifice their own time to make computers work faster. The problem is with intermediate scale, where the computer takes a bit too long to do simple things.

Case: You really, really enjoy your text editor and have even built special tools that work within it, but it doesn't support SQLite and you generally enjoy using text tools anyway.

One key differentiator is that recutils are for use cases where you want your data files to be in a plain text format.

sqlite databases are binary blobs which don't have meaningful diffs if you check them into source control.

I wrote a SQLite <-> Json converter for this reason. Ended up coming up with a weird convention in the Json to represent relations. It might be cool to instead write a recfiles <-> SQLite converter. I want to use SQLite from within my app, but want to check in readable data to a source control system.

https://sqlite.org/cli.html#csv_export

The CSV format is really ugly - i barely consider it human readable.

Why would you check a database into source control?

Not trying to be funny, genuinely wondering.

Sometimes you want data under version-control. You want to track changes and maybe you want to be able to reverse them. An example is human-supervised data-imports.

Example: we operate a pipline that pulls datasets from multiple government agencies and merges them. The sources are updated periodically and updates frequently introduce new inconsistences. To reconciliate, we use auxiliary datasets that record tweaks to each source, such as entries to add, mutate or delete.

In our implementation of the pipeline the source datasets, the tweak tables, and the resulting merged dataset are kept in Sqlite for ease of processing. The pipeline writes out dumps of each table during processing, even for intermediate stages. When running the pipeline, I can scan the diff and decide whether the changes are reasonable or tweaks must be introduced or removed. Once I'm satisifed with the result, the dumps are then committed to record the current known-good state.

When somebody wants to know why one record in the result is the way it is, I can determine how it changed in the source data, how it was tweaked, and what the result was before and after. It's really easy to produce diffs between revisions. Code-reviews are conducted over changes in pipeline code, validation logic, source and result all in one step.

In GNU Emacs SqlMode is not as nice to work in, whereas recfiles integrate out of the box with org mode. Doing the same with sqlite works, but is not nearly as nice.

Tell me when you can grep or sed an sqlite database, or cleanly check it into a git repo.

I've never heard of Recutils before, but I've been using grep, sed, awk, cut, etc. to work with data for years. sqlite is just too much of a hassle for version control and sharing. If I use sqlite, I have to export to text, commit and push, pull on the new machine, then import into sqlite. I've never run into performance issues even with thousands of records.

And where you need a reasonably performant relational data store, all those cumbersome gymnastics might be worth it.

But if you're working with a modest dataset where volume and performance aren't a primary concern, working straight in text is infinitely more convenient. There's a reason people do so much bulk data processing in CSV, JSON, or other easily manipulated formats.

SQLite is hugely more complex and therefore brittle and difficult to debug that this.

Especially if you absolutely need 100% reliable compatibility between many different devices.

Even more so if the format need to be part of a standard and/or a contract.

- SQLite is damn easy.

- It's almost a standard due to licensing.

- Tiny, really tiny.

Ah the difference between easy and simple!

It would be simple to implement an append-only writer in Recutils. I don't know where I'd start if I wanted that for Sqlite. I don't think the format allows it.

You would write a program that opens the database and perform insert statements.

I meant append-only on the filesystem layer which allows stream-reading for example.

I understand. I think should be avoided.

Go ahead and compare the SLOC count with a parser for the "rec" format.

Also, why are you ignoring the other points?

Reply on news.ycombinator.com