Hacker News Reader

Yes, text configuration files are dumb because they require N parser/script editors times M configurable programs. Furthermore, what people really want is universal progmatic and CLI access to configuration.

Microsoft Microsoft'ed configuration management with the way they implemented the registry. The Apple PList way sort-of goes there but doesn't quite master it.

A better way would've been is configuration via code and command-line that's easy to interface for all purposes. It's important to:

- be able to transactionally backup and restore all settings

- have multiple instances of the same program with different settings

- wipe out settings to default

- enumerate all settings within a program or within the whole system

- subscribe and be notified of setting changes

- ACLs or privileges to separate users and processes from each other (similar to the Windows Registry)

- audit and undo setting changes

- hide secrets that are provided from dynamic providers

- allow dynamic providers for values of settings

> Yes, text configuration files are dumb because they require N parser/script editors times M configurable programs.

No, they don't. It's N editors + M parsers (where M=programs) I suppose if you want really smart editors that are also developed completely independently, such that each require it's own parser, its N+M parsers plus N editors.

> Furthermore, what people really want is universal progmatic and CLI access to configuration

I'm fairly certain most people don't want anything CLI anything.

> I'm fairly certain most people don't want anything CLI anything.

I'm fairly certain they all want CLI everything, it's just they don't know it yet.

It's more convenient to call and export needed function directly via C API than force everybody to make and use another "universal" layer.

I want better CLIs. The author's criticism is spot on when it comes to those. The CLI flag format should be specific to the shell not specific to the program.

The low hanging fruit is a common parser/serializer, which is usually handled in library code in Unix - handwritten parsers are usually the enemy.

The other bits describe a whole configuration management system, which gets into the realm of questions like "For version X of software Y, value Z can be between A and B, but this changes when using version X+1", which is extremely difficult logic to encode into an external data store. Even worse are relational consistency issues that come up across structures.

Having a "fail if invalid" results in people setting values and then being frustrated at the system.

BTW, this all existed in a language neutral way in the early 2000's with XML parsing and validation leveraging RelaxNG and Schematron. Unfortunately those were deemed ugly and hard to use, and thrown out in exchange for the half baked JSON/YAML solutions.

I know it’s a bit hacky, but you can satisfy most of your requirements with environment variables; and most of your other requirements with a mini daemon that sits on top of `env`.

When I think about it, this is basically what the various “keychain” daemons provide, which IMO are underused despite their terrible ergonomics.

Switch and router OSes have a unique CLI configuration system that is both simple and quite powerful. Compared to Linux I especially like the ability to enumerate the entire configuration with one command ("show running-config" in Cisco's IOS, for example).

I would love to be able to see and manage a Linux system's configuration in an analogous way. Obviously there's a lot more to a modern general purpose OS than a simple network device, so I'm not sure, maybe it wouldn't work very well?

This sounds like your basic configuration management tool plus a service discovery tool.

> So now you turn one problem into two problems: manage the schemas and manage the data

This seems to completely ignore the fact that we are already managing schemas right now in the form of ad-hoc parsers and serializers which are arguably much worse than a more formally specified alternative.

The HTML vs XHTML situation: would you prefer a hard failure on any error, or a graceful degradation system that is therefore always slightly degraded?

We go back and forth on this because tightening the schema only works when you can adequately define the requirements of both ends up front, and there isn't a vendor battleground happening in your standard. Developers end up escaping into an unstructured-but-working zone. Classics like "it's such a hassle to get the DBA to add columns, so we'll add a single text column and keep all the data in there as JSON".

> tightening the schema only works when you can adequately define the requirements of both ends up front

There are degrees of tightening though, aren't there? It would be a matter of necessity (to me, at least) that an interface like he describes be extensible. To me, it's about better OS primitives with more guarantees, etc. Versioned APIs with options to fallback to older versions (to at least some extent) seems like an obvious outcome. The last thing anyone would want would be the equivalent of JSON in a database column, which would completely defeat the purpose.

> The HTML vs XHTML situation: would you prefer a hard failure on any error, or a graceful degradation system that is therefore always slightly degraded?

Definitely would prefer the hard failure. The community will literally fix all of that in a week -- all template engines and DOM generators will adapt to avoid the hard errors and everything has a good shot at becoming better as a result.

"The database for a filesystem is an classic. WinFS, the filesystem that should have been a key feature of Longhorn/Windows Vista is based on a relational database."

I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

> I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

You can implement record-based files (RAX, ISAM, etc) on top of stream-based files. The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

There are plenty of more advanced storage systems available for Unix. You don't need the OS vendor to supply The One.

The problem is triviality to implement, a similar problem occurs with TCP and UDP.

Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.

Implementing packets on top of a stream is not as trivial as implementing a stream over packets.

Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.

Record-based files would be the proper lego building blocks.

The disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.

Yeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.

Now you want a different block structure.

No? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.

> Implementing streaming files over record-based files is easier

What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.

Not really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.

What to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.

Allow clients to choose which they receive?

May be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.

Why would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.

The problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements.

TCP offers a single reliable globally ordered stream.

UDP offers unreliable packets in arbitrary order.

What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability.

TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.

"You can implement record-based files (RAX, ISAM, etc) on top of stream-based files"

Sure. And we have, many times over. A base implementation that supported fixed and variable records, shared process mmap() access, etc...would have value. And perhaps the kernel being aware of it would have some benefit.

>If you can build it from the basic blocks, it's not a basic block.

The idea behind the author's post is that a directed acyclic graph is a basic block that is just one layer of abstraction above raw bytes. JSON, YAML, TOML, INI, .properties, environment variables are all just different ways to represent a graph and if it were possible to store the graph directly in the filesystem then one would not have to manually [0] serialize the graph to bytes and deserialize it back later.

[0] graph -> byte conversion is the job of the graph filesystem

Thank you for pointing that out, thats exactly one of my points.

> The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

And the normal programmer's logic is: "if it's not already integrated in the OS then I am not going to use it". So stream files it is, forever.

I think history has proven this. People are very risk-averse even if the risk is quite minor (example: using sqlite3 instead of the file system).

Exactly. And with more opinion behind it: the fact that most of those items were shipped in a extraordinarily well-supported, mass market OS literally decades ago and still didn't catch on maybe says something about the value of the design ideas.

I think the only thing in the parent's list that didn't catch on was WinFS, which I believe is because it never even shipped if I'm not mistaken.

Registry and Powershell are integral parts of being a Windows administrator.

I might be leading a little, but just because nix didn't choose to use these things doesn't imply they are bad. Just nix is "different".

Right, I meant "catch on" in the sense of being clearly superior to the point that everything feels they need to have an answer for it and move to it. In fact the Registry[1] and Powershell are uniformly considered controversial even within their problem domain.

And I'm not arguing these things are "bad", exactly. I'm saying they aren't the panacea claimed in the article, that frankly they provide limited real benefit in a practical sense, and that the proof of this is that the one system that tried to implement them generically is aging glacially into legacy status without these ideas having propagated.

Basically: if these things were that great, at some point other systems would have picked them up while they moved on past windows. But they didn't. So... I guess my point is "why should we care about this stuff specifically?".

[1] Obviously the registry was emulated in the Linux world with the gconf/dconf stuff, which likewise was quite controversial and hasn't had anything close to universal adoption.

> the fact that most of those items were shipped in a extraordinarily well-supported, mass market OS literally decades ago and still didn't catch on maybe says something about the value of the design ideas.

that statement seems heavily draw on the idea that success is the result of a meritocracy. But 'how good something is' often plays a small part in the selection of winners and losers.

In software architecture, it's really the only good evidence we have, though. I'm all for better metrics, but there aren't any. The point is just that none of these ideas are remotely "new", they've been heavily pushed by one of the biggest players in the industry for decades, and if anything was really beneficial in a profound way we'd surely know about it by now.

But we don't. So, no, that doesn't prove they're bad ideas. It does show that whatever benefit they have is underneath the noise floor of the experiment, though.

File system allocation tables effectively are highly optimised databases already. I think the issue isn't the file system itself but rather the OS syscalls.

> File system allocation tables effectively are highly optimised databases already. I think the issue isn't the file system itself but rather the OS syscalls.

It really is the file systems as well. If allocation tables are highly optimized databases, they're not the kind of safe and robust database we're used to:

https://danluu.com/file-consistency/

> If allocation tables are highly optimized databases, they're not the kind of safe and robust database we're used to

Those two points aren't mutually exclusive (that link, by the way, is discussing syscalls like I'd mentioned).

When I talk about "optimised" I mean in terms of other limits placed on the table that makes sense with regards to storing file metadata but wouldn't be desirable for a RDBMS.