Hacker News Reader

Supporting this, I don't think protobuf is a good format for storing history files. Backward compatibility is tricky with it - I don't think the history format is going to change all that much, but I think there were changes in history of fish shell like adding time field. If the choice would be on a binary format, I would much, much, much more prefer SQLite for that. Also, B-trees are perfectly fine choice for searching history, as they make prefix searches trivial.

TOML is definitely an interesting choice if a text format is deemed necessary however. Array of tables syntax should be a fine syntax choice for storing history.

I was about to say SQLite, it seems perfect - binary, fast, easy to update cheaply in place, immune to escaping et al problems (if you use prepared statements, not glue SQL with + of course), reliable, available in most languages (even for C and C++ the integration is trivial - it's a single pair of files drop in) and on the CLI, stable, the history data is very row/item oriented and not totally heterogeneous or free form, etc.

I totally drank the SQLite koolaid a long time ago.

History file may be read & updated simultaneously my multiple processes. Is that sufficiently safe from corruption with SQLite? (honest question, no FUD implied. same risks apply to alternatives too.)

Yes, SQLite has robust multiprocess locking and transaction support. https://sqlite.org/faq.html#q5

In addition protobuf is designed for communications, not incremental updates in a local store. As such making things robust to store things on a development machine is non-trivial. SQLite is so much better for that. After Firefox switched to use it for its own history, most problems with history corruption disappeared.

You can ad extra fields to protobufs just fine, they just need to be added to the end of the definition.

Out of the mainstream formats, JSON is among the easiest to parse - thanks to the CLI tool jq. Does UCL have something equivalent?

Protobuf (or any other binary format) has the benefit of being faster to load, but I agree - it should not be be used to store things like history. It should be easy to parse history without having to write code. And with Protobuf, you need the schema.

If a binary format is going to be used, it should be optional - or perhaps there should be tools to convert the history file to/from a plaintext format.

There is uclcmd [https://github.com/allanjude/uclcmd] which could do with some more work but was created with the intent of being a UCL equivalent of JQ.

Also since he’s the kind of person that a HN reader will probably Ike to know more about. The original author of the aforementioned uclcmd, Allan Jude [http://twitter.com/allanjude], is the former host of the TechSNAP podcast [http://www.jupiterbroadcasting.com/show/techsnap/], current host of the BSD Now podcast [http://www.jupiterbroadcasting.com/show/bsdnow/], a current member of the FreeBSD Core team, and co-authored not one but two books on the ZFS file system FreeBSD Mastery: ZFS [https://www.amazon.com/FreeBSD-Mastery-ZFS-7/dp/0692452354/] and FreeBSD Mastery: Advanced ZFS [https://www.amazon.com/FreeBSD-Mastery-Advanced-ZFS/dp/06926...]

Would like to see comments in configuration, Json doesn't support comments.

UCL can be trivially transformed to JSON or YAML; ucl-tool supports this from CLI.

Thanks for writing that up, I linked to it from the issue. FWIW I agree with you that shell history should be a simple format, and even ad-hoc parseable via Unix tools. zsh uses simple : and ; delimiters. JSON is more complex, and binary formats like protobuf are way out.

Long-time fish user here, I'll second this comment and implore that you choose TOML for the configuration format. TOML format is easy for users to understand, and well-written parsers for it exist in every programming language.

For configuration I would very much prefer to have something that can be properly grepped/patched/generated with shell scripts if necessary. What openssh uses for its config mostly fulfill this, but ini file also are not that bad. Anything hierarchical like JSON/TOML/UCL etc. becomes very unattractive the moment one has to patch them from automated setup scripts.

There pretty mighty tools around to grep/patch/generate JSON data with shell scripts and they are made to deal with hierarchical data.

https://stedolan.github.io/jq/manual/

http://jsonnet.org/

Both are single binaries, so easy to run them where you need it. I don't know comparable stuff for the other mentioned formats, whenever I need to deal with yaml in scripts i run

https://github.com/bronze1man/yaml2json

I would be totally fine with any format that can be converted to JSON on the fly. Of course I would love to have something like these tools more integrated within a shell, but the representing text format on the disk is not that important for me.

Templates in configuration management can help with this a bit.

Thanks for bringing up UCL! Hadn't seen that before but I definitely need to try this in a few places.